For the March edition of imaging and Informatics community webinar, uh, we are excited about, uh, the presentation today the, the, the speaker, as well as the topic that, uh, he will present.
And this will be the majority of this decision and would be follow up.
Followed up by some updates about and further announcements, but I do have an announcement that I put early on in this presentation and that's to call your attention to an upcoming ai workshop on April.
3rd and 4th. So, April 3rd will take the place of this. Uh, this community webinar, the regular, our time that this webinar is held the 1st, Monday of the month. So we are in the process of potentially, uh.
Rescheduling the April imaging and Informatics community webinar to a week earlier or later, and that's a pending confirmation.
So we'll be in touch about that but a little bit more about this workshop this is to address, uh, imperfection or limitation of data sets in AI for cancer.
Research registration is free and open to public through out this month and, um, 1 of the features speakers is known to many in the airport is usual.
You from University of Toronto I believe he will be speaker on the 2nd session there about 6 sessions in this workshop. And the details are on the website that I believe will put a link in the chat. In case you are interested to.
To sign up for it. So, um, today's presenter is Dr Georgie. Uh, he's a professor and chair of advisor for Informatics at the department radiology at Cornell, uh, medicine. Uh, he's also.
The leader of several working groups and task groups within, uh, imaging.
Societies including and SIM as well as the CO, founder of platform for code base image, annotation and analysis.
And by now, I think everyone has heard about and other large language models, and the potential that they have to essentially.
Revolutionized a lot of aspects of our, uh, daily lives as well as, uh, access to information and healthcare delivery systems and Dr. she is the senior author of the current radiology editorial on this topic.
And so we excited to have him present and really appreciate his availability. So, without further ado, I turn it over to George.
Thank you thanks Dave on and thanks to the NCI for.
Inviting me, it's really nice to be part of the.
Intro with your ship and Joe, you guys should probably go hear him talk.
Instead of me, he's he's kind of obviously the.
Super expert and you've already got for him? Yeah, I'm working on the insurance.
Um, can you guys see my screen.
Yes, yes. Okay. Yes. Great. So, um, let's see, I'm going to get started here.
Um, let's see.
I'm going to turn on my captioning, so.
Feel like people and see better.
I do that. Um, okay.
So just give me 3rd.
Great. Uh, so yeah, thanks again, Caroline for the intro I'll be talking about.
To, and the potential healthcare application of large learning models.
Large language model excuse me? My main experience though is with chatty PT, but I think most of what I'll be discussing war.
Apply to other applications, using large language models. Um, I also realized that if you talk to other people about this kind of.
Generative ai evolving text they won't really understand you, unless you say since.
L. m's large language model is not a very common term yet.
Uh, but again, these can really be seen as interchangeable for today's talk with, with the exception of a few slides. Um.
So here are my disclosures already mentioned.
I'm kind of an informal advisor to open a I.
I'm not paid and I don't have any, you know, stock or unfortunately with open a. I, but.
Um, I have been using some of their Pre release versions.
Just a little bit about me, um.
I studied at Duke University with a major computer science.
I spent some time in Paris for my master's degree.
Went back to Duke for grad school, and then I've been at in New York City.
Ever since it did at residency at Columbia, and I've been at Cornell since residency and fellowship.
So, I saw this in my news feed this morning yet another article about Chad doing something cool.
Chad invented this game and this guy put it online and wrote an article about it.
I showed it to my 5th grader, and he was like, cool.
So, not only did it come up with a game idea. It also provided the code complete with pretty reasonable CSS, which is kind of the, the.
The code for with how the website looks.
And to me, it's really amazing, but just 3 months after the launch of chat.
Which is again, 1 of the large based off of a large language model of which there are many.
It's already kind of routine and just cool from my 11 year old.
Right. He's been using the chat bot to help him code roadblocks games.
So, I think that goes to show how quickly humans, and maybe especially kids adapt to new technology.
Um, by the way, I think Chad is better at Python than Lua, which is the language is for roadblocks. And so maybe it's an opportunity for other.
Large language models to capture the market for this age group.
So, you know, the topic for today is but AI, right? And the is everywhere.
And I think even just a few years ago, I don't think you could say the same thing.
And so we've really come a long way in terms of developments.
Um, and we're starting to see.
I help make important decisions in many areas.
Including education, employment, advertising, increasing.
And even health care, and I had this slide in kind of right after RCA I've given a couple of talks on this already.
And right after our happened, someone asked me what was the biggest thing that happened during our today.
And really for me, the most impactful developments around in a time.
Happen outside the meeting, um.
Someone wants to annotate the document. I don't know if that's something I should say. Yes. To you.
All right, that was confusing. Um, and and really, you know, I've listed a few.
Here, but stable diffusion.
Whisper, which is an open source model for.
Speech recognition and chat.
I've been really impactful in the last few months in our whole field that maybe most fields.
We'll have to recalibrate after these developments.
How will they recalibrate? Well, maybe find some of the answers that said I'm sorry I had to.
Sam annual meeting, which.
They'll probably have lots of AI sessions, uh, back to.
Chat this is a New York Times headline 1 of many, right? I'm sure like, you. Most of my news feed is filled up with articles.
Experiments, um, and.
To give you a sense of how quickly it was, it adopted. Um.
You know, chat was the, you know, by far the most rapidly adopted.
Uh, consumer application.
In in the history in our history, taking just 2 months to reach a 100Million users.
Um, interestingly, I think it's really the only kind of.
Real API app out there. Maybe Google translate.
Is now considered kind of AI But back, then it probably wasn't using to say that our machine learning.
And it's been really impressive. It's been described as.
The world's best chat bot, which I agree with. So let's try it out.
So, back in December, I work a lot with radiology. I. T and so.
Uh, I sent them the following email just before the holidays here.
I T, team, I couldn't find the words to which you guys have the holidays. So I asked Chad to to do it for me.
And, of course, you know, excels at writing, probably.
Many or most of you guys have tried it by now and.
It really provides, you know, nice. Um.
Material or copy that you can use asked it to write a poem about medical imaging and it did that without any issues.
Um, I was also asked to give a talk.
For the arsenate ai certificate.
Um, I, I was 1 of the course directors for part 1 of their RCA certificate. And this is this is a part 2 or the advance certificate.
And so my top topic was the normalization of images.
For deep learning, and so I thought I'd asked Chad to to provide an outline.
And where we're relevant, provide some Python code.
To illustrate now, this is a fairly technical.
And special specialized area.
Um, and as you can see here.
It actually did fairly well for kind of an overview.
Um, it knew a lot about com.
It could write some Python code.
And kind of load images in, and it went through all the steps.
For tactile normalization, um.
And, in fact, I did base my talk off this outline.
And provided a sample Jupiter notebook.
That people could use, and as far as I can tell the code, the Python code was fine.
So, showing some friends in radiology, this is 1 of my former.
Residents and he hadn't tried it yet. Um.
But he texted me back and saying, just waiting for the day. What I can just say, generate a abdomen focus report with is uncomplicated, Super diverticulitis. So, which I responded, why don't you think it would work?
Um, and of course.
You know, does a pretty reasonable.
Uh, report generation for a, and call this afternoon.
So, it's pretty impressive. There's been a lot of written about it already.
I'm going to kind of run through the section very quickly because.
Uh, I don't think we have a lot of time. I've given a few talks on this where I've shown.
Kind of other physicians, um.
Some of the capability for kind of answering the medical questions.
So, what are the requirements for being a doctor? Well.
We all know, you take the sat in high school and then and college, and then you do medical school where they're step 1, 2 and 3.
And then residency, it may be some fellowships, subspecialty boards. But what if.
I can answer any question that you can answer as a physician.
So, I think this audience and I, I'm having a hard time because I usually use zoom.
I can't monitor the chat very well. I'm not sure where it is to be honest.
But so I'm not going to kind of do what I typically do is kind of.
Have the audience try to participate in this? I'm going to show you some of the questions.
And hopefully you guys will be oppressed with.
The answers if you, so, um, you know.
I'm going to quickly read through this question 60 year old woman.
With chronic hep C and cirrhosis presents with the initial onset of download extension, it can be Donald pain.
Exam reveals 10 societies, which are the following is indicated now, remember.
You know, unlike kind of other.
Qa sites where you're kind of maybe like a Google forms where you're putting in a question and then.
You have to kind of put in the different options for a B. C and D. I'm just pasting this in right? And asking chat it to you with the answer is.
You can see here and this has been vetted by.
Some of the doctors in my other tasks that it picked option. D, right. Um.
Therapeutic and diagnostic crystal, Jesus, which is what we think is the right answer, but it also provides a nice discussion.
Uh, for some of the other options here, here's another question.
Uh, this is, I think, a trauma related question.
About what the next steps are, right? And again, it picks.
What we think is the right answer option. E, right and then has a nice discussion.
About securing airway, et cetera.
Here's an interesting question. This is.
61 year old with.
Diabetes hyperlipidemia, hypertension presents with Timmy. Uh.
Without speech problems, the symptoms resolved while you evaluate the patient.
Which of the following is correct. Um, and so you guys can read through the.
The options here, but.
When I kind of asked this question to other groups of physicians.
Most people didn't think that the right answer was in 1 of the options here.
And, in fact, either the chat.
And so, instead of picking an answer, it went on to say what a thought was the right.
Option would be right to perform further testing. So I thought that was really interesting because even if you ask a question, it's not limited to kind of picking.
1 of the options from from the choices you give it.
This 1 is very specific to radiology and it's so specific that I'm shocked that.
He would even understand it so this is during a multidisciplinary tumor conference the radiologist is asked to provide.
Interpretation of a complex post op scan was performed in an outside institution.
Right. The outside record isn't available the clinical team plans to use your interpretation for their management. What is the most appropriate action?
It's something that really, really I'll just deal with quite a bit, but really no, 1 else would would.
Deal with this, and the answer it gave, I thought was very reasonable as option B, is to document your interpretation formula in the medical record. Although there's some variation, depending on your institutions and policy.
Some people might repeat the scan.
Um, so I was by this point Super impressed, right?
I think you guys have seen enough examples, right? I'll kind of run through these.
Quickly, but by, by the way, you guys should feel free to interrupt me and ask questions and.
Again, I can't see the chat in Webex for some reason.
But you can feel free to interrupt me if there's any questions in the audience. Um.
You know, this question, I gave a nice answer, which.
Is probably the right answer, but it talks.
About kind of other potential diagnoses.
In this case, another kind of radiology, very radiology centric question.
I put density in the right cerebral hemisphere.
Show some Mr. findings, whereas the differential.
It gives them very nice differential discussion.
Including stroke infections, inflammatory to mass.
And hemorrhage, um.
Yeah, and then, you know, if you keep experimenting, you realize that, you know, certain examples like this 1 generate a normal test extra report.
It has guardrails in place, right? I'm sorry. I'm not able to generate a normal just such a report as an alert language monitoring, et cetera.
Similarly, what is the survival rate of stage for lymphoma? Right? I'm sorry. I'm not able to provide specifics about our rates. So.
It's interesting that they do provide some guardrails in certain situations.
For kind of non is a non radiology example. Um.
You know, as a radiologist, um, I, I don't.
Think I, I probably know what they are now, but when i1st saw this.
Um, I wasn't sure if what all the abbreviations were right and so if you.
Put this in chat, you ask it to describe and complete sentences.
Uh, it understands all these abbreviations, which is really impressive. Um.
And if you want to get further assistant, you can ask it to.
Present an HP format and it'll right, right. A nice kind of.
Note for you so you can hopefully, you know, maybe.
Make some edits and put that into the.
And, you know, on and on, they're kind of nice examples in this case.
We ask you to cite the evidence and.
At the time I thought it was really impressive because.
You can see for this patient, the 3 conditions.
That we asked it to.
Provide a plan for, uh, included what seems to be references.
In a lot of cases, maybe in most cases people have realized now that the references that PT gives are fake and are not real.
So, I would probably not trust it with signing the literature.
At least not this version of.
Uh, but the discussion for each condition is actually pretty reasonable. So, George, there's a yeah sorry to interrupt.
There is a question in the chat about under what conditions does fail in the diagnosis and they said an article.
From my page today. Okay great.
I don't think I've seen it again. Is there a way? I can look at least for me, it's at the.
Bottom right corner, you know.
I realized when I'm presenting the looks completely different. Yeah. So not to interrupt your your presentation. We can, we can get back to this at the end of the presentation. Yeah. Okay.
Um, all right, I'm actually gonna, I'll send you a quick video, so, um.
Okay, so can you guys see this.
Back now, right? Yes. Um.
Okay, so this is maybe, um.
1 of my favorite examples, build me.
And basic HTML table with medical imaging modalities, and a number of exams performed each year in the U. S. I like this because it's kind of 3 separate.
Web searches, right you have to kind of know HTML.
Or ask about HTML, you need to.
Know, what the medical imaging modalities are, and then you have to find kind of the number of exams that are performed each year. Right?
And if you ask that question, it actually comes up with a really interesting and and.
You know, pretty useful example of.
How to build this HDML table right?
And nice little disclaimer at the bottom that these are not actual numbers, but maybe estimates.
So, I really like that example it shows you how.
Not only can you kind of interact with it, but it can really combine multiple things to 1 step.
Right. Um, I'm not going to really go through this much, but obviously it's very good at generating.
Kind of creative content, right? I think it's pretty good. I don't know. Maybe poets wouldn't agree with this. Um.
Great. And so now I'm going to dive into.
How does chat work? Um.
And and really, this section is pretty specific for, um.
And and it's really kind of the, um, focused on that. Um.
As a disclosure, um, I am not a computer scientist, right? Um, but I'll do my best based on kind of.
Dive into how works.
Um, and this illustration describes how was created, and there essentially 3 components.
Uh, to make this chat bot work, there's 3.
At the top left there, there's instruct at the bottom left.
Uh, which uses reinforcement learning from human feedback.
And then the 3rd layer really on top is a safety layer. So we'll start with 3.
So, what is it stands for? Generative? Pre, trained Transformer.
And it's an large language model based on their own networks.
And some of the common tasks that we see for NLP is.
Are things like text classification sentiment analysis is a nice application of that.
Question answering text generation, text, summarization.
Language translation et cetera, right? So.
And actually can do kind of probably most, if not all of these things very well.
And really to to kind of.
Hyper simplify how a chat works, or how works is essentially.
Predicts the next word, right? In whatever application that you're trying to do.
Right text generation summarization and it's very similar to how your phone smartphone works. Right? So if you type in.
Bounce shouts. Right it most likely is going to predict.
Kind of in the middle here, I don't know if you can see it not is, is what it predicts us. The next word is probably a lot better than what your phone predict currently.
And here's what we mean, when we say that was trained on the Internet.
You can see kind of the sources that.
They used the train and a lot of people think oh, it's Tryna. Wikipedia. Wikipedia is actually a very.
Small part, um, portion in terms of number of tokens.
That was used for training, um, most of it was.
Really the Internet, um.
Right so so the 2nd, part of.
Chad was something called instruction and and this is really the basis for.
The real improvement for over 3.
Um, and they use something called reinforcement, learning from human feedback.
Which is summarized on their website and you can get the details there.
But essentially there were 3 steps, retraining chat from 3. so the 1st, 2 steps.
Involve humans in the loop so step 1 was having humans right? Outputs from sample prompts.
Human outputs are used to fine tune.
And train a new model, which was really kind of what they called point 5.
All right, so that's step. 1 step 2 provides.
Outputs and then the humans assess these ai generated outputs.
And then the humans rank them from best to worse.
And in the process, they train is so called reward model. So this is a separate ai model.
Reward model, um.
But then they use in step 3, because this reward model then ranks the outputs of.
Other outputs generate from to keep improving the outputs.
Chatbot output, so it's step. 3 is unsupervised and there's no.
There's no humans involved. It's probably a very.
Simplified version of what they actually did, but there's an interesting paper.
Around how instruct you works and if you guys.
I have a chance. You should look at it. I think I provide the reference later.
So, finally the, uh.
Added a safety layer on top, um.
I don't know if you guys remember, but when was released admitted pretty big splash.
But it was also shown to provide some unwanted.
Biased and in some case she is very bigoted output so they realized.
That had they had they had to change their approach.
Uh, oh, yeah so the article I was talking about is is kind of listed on the slide here. Um, so in the paper they published last spring.
So, I described the following it is, is from ice.
Excuse me this is the future.
Yeah, so garbage go with that. So, um, what would they realized, though, was something more clever than that? Because language models bigger doesn't inherently make them better at following user's intent.
Right. So, for example, large language models can generate outputs that are.
Not truthful or toxic or simply not helpful to the user.
Um, so, in other words, these models are not aligned with their users.
Um, and so the really the goals for were to make it helpful, honest and harmless.
And they want it to be very useful and truthful.
For the patient, so they implemented various guard rails and safety rules.
You know, like these to make it more user friendly so they were still able to leverage but then with the safety layer on top, all of a sudden.
It became more user friendly right? And so now we're going to pivot to.
Potential health care implications for chat and other.
Large language models, uh.
And this is really mostly about large language models, but I think.
That other API techniques involving images will ultimately be part of.
Some of the health care solutions and I'll show kind of.
Maybe an example of that later in my slides.
This is a nice quote.
And I think a lot of us would agree with, to some extent.
And this is from the former CEO of a Kaiser.
I am certain that 5 or 10 years from now.
Every physician will be using this technology if doctors use chatbots to empower their patients, we can improve the health of this nation.
Right, I mean, it could be sooner or it could take a lot more time than expected, but this kind of technology will probably be adopted by.
A lot of people, whether you're a provider, um, or a patient.
Here's Here's a graphic showing a bunch of the large language models out there. Um, and their relative sizes typically.
The size of a language model is, you know.
Um, it's shown to be kind of.
You know, relevance or proportional to.
Maybe the, you know, potential quality.
Of its output, right? And so they've kind of generally shown this with.
Um, these ai Neuro network techniques in terms of model sizes.
Um, most of these models shown on this slide are probably.
Still kind of researchy and may not have a nice user interface for the mainstream user, but it's, it's useful to see this. And you can actually see that here on the left. There's a red.
Um, circle here, um, is not.
The largest model anymore I think it was at the time.
And then, I think we have for coming out later this year.
But even not being the largest model is that.
Quite an impact I do want to point out some of the takeaways this was written up about.
When it 1st came out 1st of all this shows that, you know what I was mentioning before that.
Language model performance skills.
Uh, as a parallel of model size, basically, the bigger the, the model size typically, the performance goes out.
Um, but the point I really wanted to make here.
Was that at the time it it took about, um.
You know, 4.6Million dollars worth of GP compute to train.
And I think the equivalent of like, a 100 years of GPU compute time.
Um, which makes it really, really difficult.
To do in a research setting.
Right. So, only very large companies with large.
You know, research budgets, um.
Like, some of our largest tech companies can afford to do this, which is a bit.
Concerning for me as somewhat in academic.
Medicine, I'm sure maybe you guys have some thoughts thoughts on this, too.
But that's something that we have to pay attention to, you.
Uh, because if we want to be able to compete with the state of the art.
We need to do something about compute right? It's it's really expensive.
Right now, so, in terms of more consumer oriented products, that leverage large language models here so list of 29 other, you know.
Uh, products that leverage these, um.
Right, and you can see here on the left kind of.
You know, they categorize them kind of a little bit randomly. Some of them are.
By company, but some of them are used for things like coding and translation.
Research, but here's a whole list list of them. I have to admit, I don't think I really used any of them.
Uh, besides copilot, which.
Is the 1 from hub that's now owned by Microsoft?
Um, and so it would be interesting to see if anyone.
And this audience had experience with other.
Large language models as well.
And I've listed kind of some of the ones that I've seen more in the media. So, the 1 from Google, Google, bard a, I.
Microsoft, even though it's a big investor in, has its own version being I think it was shut down recently because.
Inappropriate responses that it gave certain.
Beta users, um, uh, Facebook released.
Very recently another large language model. They had 1 kind of last summer for research. I think it was called Galactic. Uh.
And they, they had to shut that down because.
I don't know that the outputs we're, we're streaming out some.
Something inappropriate or not useful um.
In terms of biomedical, uh.
There was 1 that was used to be called pub.
It's now been renamed to BioMed language model.
And I think that one's open sourced. Microsoft has its own bio that was released recently as well.
There are probably others out there that, but that's kind of the list that I found.
And and of interests.
It I think everyone saw, or, you know, if you haven't, you know.
Chat passed the medical board exams right? Um, and what's kind of interesting um.
From that is that it paradoxically.
Outperforms or BioMed.
Language model is now called, um.
Even though, you know, the was supposed to be specialized around.
You know, kind of biomedical types of information.
And no, 1 really knows why, but people are speculating that, um.
You know, training on domain, specific domain, specific training may have called greater ambivalence.
In the model, as it absorbs real world text from ongoing academic discourse, that tends to be inconclusive.
Contradictory or non committal right? And so and maybe Chad has access to.
Additional information such as patient facing.
Primers or, you know.
Medication information that sounds more definitive and congruent so no, 1 really knows why, but it's.
It's not completely obvious that if you.
Train a version of a large language model.
Specifically for biomedical purposes.
That it would perform better than general.
Uh, generalized, large language models so I thought that was very interesting.
Um, uh.
Back in January, we had a New York state Rankin society. Webinar.
Where the topic was, um.
Chat potential impact on patients.
Physicians and practices and notice here at the bottom, you know, I highlighted that.
We ask people to register and then we kind of ask them a few questions about.
How they are using, um, and I asked him on the same thing, cause I didn't know if there was any registration for this particular session. Um, yeah, but.
What I'm going to discuss kind of what we talked about, but before for that, I want to show that.
I was kind of interested in how people were using the so we had the Excel spreadsheet.
Downloaded with, kind of all the answers from the questionnaire and so.
Decided to plug it into.
Essentially, I copied and pasted the contents of the Excel file directly into the chat.
Right and I asked them to do some analysis.
Um, uh, the people who registered.
And it actually did a pretty reasonable job.
Right. So it took, um, just direct information from a spreadsheet.
He was able to provide kind of a nice summary as to.
You know, the people who are registering right? What do they use it for? It says most registrants currently don't use.
But it also kind of highlighted some other use for.
And then I asked it to generate some Python code.
Right. So that we can maybe plot some of this information out and.
As you can see, it's really impressive. Right? Like, you can just ask it to it.
Be your personal data scientist.
So, I thought I thought that was pretty impressive. Um.
And you guys, you can either take my word for it, or watch this example, but if you copy the code, right? And you paste it into I still use as my editor.
You paste it into your preferred editor.
Right. Um, and then you run the code. It will actually.
Work, so this might be a new way to.
Do analysis on data that maybe is not super important, but it'd be nice to have analyzed.
And so out of that webinar, we actually published.
An editorial that Kevin had mentioned.
About, and how they're potentially double edged sword and I'll kind of summarize that. I mean, you guys can read the article.
But we really focused on, you know, the possibilities of something like this to help with.
Imaging appropriateness and clinical decision support we thought it would.
Be potentially useful for generating kind of letters for preauthorization, maybe to insurance companies, you know, as my friend put it, you know, using to help.
By PS, right? Um, and.
You know, generating content that we could include in.
You know, the medical record, right? Um, we could use it to summarize the medical records. Some patients have very lengthy medical records that.
We take a long time to summarize if we wanted to know something about the patient before we interpreted.
Some medical imaging, right? And so would probably be the.
You know, are there large language models would be the option to.
To kind of do something like that. Um.
There was talk about kind of interactive computer, aided diagnoses and then my colleague Keith, until he was very active, kind of.
And then New York state society thought it would be a good.
Way to leverage this technology for advocacy of important issues in our field.
So, I think it was a very thoughtful editorial. If you guys have a chance to.
You should have a look.
This paper came out recently, I guess about a month ago now.
Where they kind of.
Imagine using.
Large language models, um.
In kind of the clinical workflow, and the figure that you can see here shows how patients may directly.
Interact with large language models. Right? So.
In this figure, it shows that you feed the outputs of.
You know, potentially multiple AI, clinical ai models into a prompt.
That the large language model can then consumed.
Right. And then at the right it shows an interactive chat session with a patient or provider.
Um, or these results, um.
I think it's an interesting scenario that seems within the realm of possibilities now in 2023.
Probably unimaginable even just 6 months ago.
Yeah, George, Mary, uh, stop and ask a couple of questions that are personal chat. Uh, 1, is that how could, uh, the gardens.
Patient harm, or harming the patient with using.
To, and the other is, uh.
I think related to this case, under your example, about summarizing a patient record, will the medical record be exposed or are the HIPPA compliance considerations? Yeah. Um.
So, I think that, um.
You know what's interesting is that.
Open a, I reached out to my friend and myself.
Specifically, because they were interested in.
Health care uses right use cases.
So, they they are obviously very concerned.
About, you know, potential harm.
Were inappropriate content for health care.
I don't see this as any different as from using current search engines where you can find.
You know, fake information about how what you should do for certain.
You know, conditions that you might have right? I think it's, you know, there's tons of fake, fake treatments and bad information online as it is.
Yeah, but for for summarizing the case, you need to give Chad to other tools, access to the patient record. So so, for the question number 2, because I was answering a question number 1 question number 2, I assume that.
You know, hospitals and will eventually interact with large language models.
With companies that have beers associated with them, right? And so.
Sure, you kind of have some kind of legal agreement in place and that you probably wouldn't be using the current public.
Chat for these types of applications you would.
Enter enter, enter into agreements with maybe specialized versions of the.
Of this kind of application, right? And so it's not really so much about.
You know what we can do today or how good is it today? It's about I think, in my mind, you know, we see how good it is already today that I can at least, you know, probably extrapolate how good it will be in a few years. Right?
And that's that's kind of the excitement and anxiety that I have is not not about today's version of chat.
So, I have an anxiety about future, so so these models are trained based on current.
Records or language content, and I believe was chained up to.
Information about 2021, so if we rely more and more heavily on these tools and less and less on humans, um.
Are we going to have to I mean, there's not going to be enough content generated for them to train on the future.
Yeah, I, I think that's very interesting. I think, I mean, in 1 of my slides, I, I kind of talk about, you know, how do how do we.
Figure out the training data, you know, issue. I think the real time issue is something that I'm sure that they're working on right?
Just like, you guys may not remember this, but Google had a very hard time, doing real time searches.
For a while, and I think Larry Page even admitted that, you know what real time is really hard.
So, I think with large language models, it would be the same.
It's going to be a very hard problem, but I bet you that someone's working on it.
Right. So I think that these are all very good questions.
But I would assume for a lot of them that these will be resolved or improved dramatically.
Over the next few years, given the amount of effort and investment that's going into these language models.
Right, but those are great questions. Uh.
Yeah, so so there there's a question about how do you, you know, this.
All this misinformation right? How do you keep it accurate? I think that's really pointing to the safety layer.
Right of chat at least for.
Um, and they do talk about kind of roughly how to do, but it's, it's a little bit of their secret sauce.
Right. And so they have a way to they're doing their best to try to filter out misinformation or.
Uh, let, you know, when they're not sure about something and so I think we'll get better at, you know, these large language models again. If you, if you just want to predict the most likely next word.
It's very different than if you want to predict the most likely next word that's not harmful or that's not fake. Right? And I think that's the innovation.
That open I did with chat jbt is that they, they have a safety layer, right?
Um, so it took something that was basically unusable.
And they made it infinitely more usable. It's not perfect.
They made it a lot more useful. Um.
How many of us in this meeting will be working.
At the 7 to 711 0, what.
In 5 years, then you wouldn't know what that means.
That that's the oh, it's 711.
They will have a chat box. I would they need any workers at 711 yeah.
Yeah, good question. Yeah I don't even know I, I don't know what to think about the future.
Job marketing. Well, here's a better. This. This is Mark. Um.
W, W, what and you've done this George? How does how do knowledge workers like us and maybe in biomedical research.
Stay ahead of the curve or incorporate large language model.
Thinking into how we think about our professional development.
Yeah, yeah, um, I do have a slide on education um, and I, I think maybe.
You know, it's potentially 1 of the main benefit, if not like.
The biggest benefit of something like this. Um, but I think you're right, we have to always adapt to that. I can tell you that.
You know, companies around the world are scrambling to figure out how to incorporate this technology into their products. I, I really don't know. I'd like to say that.
It's already pretty easy to incorporate right? Like, open.
And some of the other companies already have APIs where you can easily access and.
You know, incorporate outputs from.
You know, these models into whatever you're doing, whether it's research education.
Or, if you know, if you're, you're kind of building a commercial product, um.
When they 1st launched, they didn't have the actual chat.
Version available as an API, but I think they just released that. I want to say about a week ago.
And so I think.
Um, I think it's going to be easy to incorporate. I don't know.
What it means for, you know, jobs later on, I think we're still trying to figure that out. I think there's a ton of, you know.
Regulation that's going to come this way, right? And the Europeans are talking about kind of.
Heavily regulating AI, right? So, hopefully we have some guardrails in the U. S as well in place by then.
Should I go on so, George, I want to do a quick, classic, um, where we have about 8 minutes left of the hour. You have broken the record for number of participants to this webinar it picked up, I think 333 to 5.
so so, I think because of that, we can give you the rest of the time, and we postpone the announcements if if my colleagues don't mind okay. About 56 minutes left. Yeah. Sorry, I added something, so I only have a few slides that, um, no problem and, um. um no problem and um
And, yeah, it's probably not me, but you should've been to it, right? Like, they came to it.
Here about that, so so I'll just quickly say that Stanford human centered ai group.
Published a nice document. I think it was just a few days ago on their perspectives on generative, which includes both the.
Large language models and also generative for imaging. Sure. Some of the diffusion based models it's worth looking at. I won't kind of spend a lot of time on it. They do talk about healthcare.
Uh, they talk about it, like synthetic patients and using it as a physician assistant for some of the things I've already talked about decision support and clinically adjacent task like.
Help with complex scheduling and medical code billing and things like that.
Ultimately, hopefully relieving the physicians of.
Some of the tasks that they currently have to do manually and so there's more time for.
Physician patient interaction, which is, I think what everyone wants, there's a section on biomedical research.
So, it's really it's really worth having a look at that, but I do want to show this because.
I'm not sure. I mean, maybe some of you guys have, but this is this is fairly new to me. So.
What have you had? Multimodal chat.
So, if we can combine chat with images.
So have a look at this example um.
There's a picture here and obviously this is Einstein, right? And.
You know, his famous equation, right? And if you ask.
If you ask this ai who is in the, who's the person.
In the in the picture, it answers Albert Einstein.
What is the equation? Equals? M. C. squared? What does it mean? It means.
That mass and energy are equal. Isn't that crazy? You can now.
Start to interrogate images, so.
You know, of course, as a radiologist, I'm very curious about not just.
Einstein pictures, but about X rays right? And so.
This 1 is a random, probably publicly available X Ray.
Right and then ask what is this? It's an X Ray what body part is it test?
What objectives are relying on the left side of the chest.
Peacemaker, even this very general model that.
You know, it can can already kind of interpret some easy things about X rays. Right?
So look at this, so this is a mammogram. This is a normal mammogram. What is the category.
And are you sure and it says, yes, this is a normal mammogram so.
So, I think we're really at the tip of the iceberg, in terms of what we'll be able to do with some of these.
You know, chat, plus, you know, other technologies, um.
And we're kind of out of time, but, you know, I think there's a lot of.
You know, innovation that we can do in education, and maybe education could be potentially the most important contribution of large language models. It's actually a very interesting discussion.
Um, as long as we can figure out the best way to leverage them. I'm I'm talking to the school leadership of my kids schools to figure out how we can.
Incorporate this into students.
Okay, so in summary in summary is everywhere right? Um.
And other large language models are transformative.
What's kind of interesting is that they're widely available and to me, they're near.
Right and this really democratizes ai for many fields, including health care.
I think we'll see rapidly rapid improvements over the next few years and.
You know, multimodal possibilities um, it is a double edged.
Opportunity right I think we're going to be overwhelmed with API generated content.
And 1 of the questions that we talked about earlier, how do we ensure.
You know, training, data, quality for future LMS.
If everything is all the contents a, I generated, like, how do we, how do we know what's good and bad and real right?
Um, and I've seen some, you know.
Opinions on, maybe we're headed for a dystopian future. If everything is ai generated. If.
Including everything that you see and read, and even your zoom meetings are kind of.
You know, generated with fake versions of yourself.
Right. Do we have to think about a reverse Turing test to prove that you're human.
Right and I saw this quote that I really like, um.
And it's kind of a random person on, you know, Hacker News, right?
And I just, I thought it was very clever. Maybe we just go slower.
Something like, Hippocratic oath could be a good principle for.
To follow for high impact technologies.
Uh, to do some good may not justify the harm.
Right so so, maybe all AI, researchers and developers need to take this out, right?
And not just doctors, uh, it won't fix all our ai issues, but maybe it's a.
Um, and I just figured that in my last slide, um, I would use to do something relevant for this group.
So, I asked it to summarize the previous seminars from the NCI.
I gave it the link right with a list of.
You know, previous webinars.
And acids provide a list of speakers and 1 line summaries and recommend new topics.
And speaker suggestions for future sessions, um.
As you can see here, it's a nice summary. It's not everyone's in there. For some reason. I know Elliot and Andre fed often, make it there. Maybe they're blacklisted or something.
The future sessions, you can see.
Is a nice list of topics that you guys might want to think about.
I don't know if they're real people. Maybe you guys would know them. Um.
I wasn't on the list, so, you know, wasn't very happy with that. So I asked.
You know, if I should be giving a talk about right? And the future and potential impact of.
Healthcare large language models and so we'll see what chat rights.
Well, this is great. I think we are already being replaced and sorry, I'm typing in real time.
I read that real time, but I was kind of typing live here and as you can see does not recommend me for giving my talk. Um.
So, if was using chat for speaker suggestions, this talk, would it never happened.
All right, so thanks, Kevin. Thanks everyone. Thank you. Thank you. Oh, it was very interesting. And engaging presentation. I appreciate the extra time and you did have.
The largest number of attendance to this webinar. So go to chat with PT that and and I appreciate everyone's participation and attendance. Thank you so much.
We'll check back with you in a few years. If you're still around.
Thank you so much. Thanks, Kevin. Thanks everyone. And the recording will be available in a few days for others.
Okay, great Thank you. Ali. Can we close up? Please.
Yes, we can close out Thank you so much everybody for attending and thank you George for the great presentation.
Thanks everyone bye.