Okay, thank you. Um.
I'm going to begin by introducing our speaker today if a coupon or others come on board with, with the other parts of the agenda. Please, let me know afterwards, but it's my distinct pleasure.
Really, really distinct pleasure to introduce today's featured speaker. Dr Elliott. Seagull.
Yeah, he holds several leadership positions at the University of Maryland. He's professor, and also vice chair of the Department of diagnostic radiology as well as the director of the Maryland imaging research technology's laboratory.
He holds adjunct appointments as professor in bio engineering and computer sciences at the Maryland, Baltimore campus.
He's also the chief radio, chief of Radiology and nuclear medicine for the Veterans Affairs, Maryland health care system.
And it was under his leadership that this became the very 1st film lists, health care enterprise in the world.
elliot's research interests include picture archiving and communication systems telemedicine, imaging, Informatics, artificial intelligence and so very, very much more.
He's authored several 100 articles, editorial and review papers and has several book chapters to his credit.
On a personal note, though, when I became when I began my career at NC, I.
Ah, Elliott was a regular member of our staff meetings.
She contributed many ideas and directions to the cancer imaging program in those early days.
It was always very useful for me to listen to his remarks and his strategies for directions in cancer imaging.
She's going to speak to us today on a topic that I'm sure is a great interest to each and every 1 of us bringing ai hyped reality for routine, clinical practice, defining, and addressing the DAT gaps. Elliot Thank you. So very much for being here.
And I eagerly turn the microphone over to, you.
Thanks, Robert for the great introduction. And thank you for.
Hosting this yeah, even as you're covering and and and getting better. So I really appreciate it.
And it's also great to have the opportunity to kind of be back and talking with NCI again. And.
Maybe, uh, strategizing and I'm really looking forward to.
Questions and discussions, um, at the end of my presentation and.
That was a perfect introduction because.
You know, what I'd like to do is sort of maybe about 15 years later.
Kind of give some of my observations and, and maybe recommendations for consideration.
For opportunities for the cancer imaging program and NCI. So it's really a pleasure to be back here and pleasure to be talking with.
So many people who haven't passed and are currently good colleagues and mine also and so really appreciate it.
This was a image that was taken at the Baltimore via medical center and this turned out to be in 2004 the kickoff.
For the National electronic medical record initiative, which turned out to be.
The kickoff for meaningful use, and many of you remember that that ended up creating this incredible transition that happened really quickly because of government incentives.
To the electronic medical record, and I had the opportunity to actually talk with them, uh, President Bush personally, when he came to Baltimore, we spent a few minutes, a few minutes talking with each other.
But during his speech, uh, he mentioned that the Baltimore was the world's 1st finless radiology department.
And then reading off of the prompter, the teleprompter he read that that meant that images would not get lost.
And we'd always know where to find the images.
Um, using the pack system, and then he said in the 2nd reason why it's really good.
That they went digital was and then he paused.
And there was nothing on the teleprompter and as you all may know, you know, George bush's was not necessarily.
Coming up with, uh, comments and so he paused for a while and then he said the other reason that it's great that they're digital, is that the films don't get wrinkled anymore.
And there was kind of a pause and then everybody sort of stood up and gave him a standing ovation cause I guess they didn't know what else to.
To say, and I found it interesting that, you know, despite the fact that he may not have been an expert.
You know, he was also struggling with, you know, what is the other advantage of being digital rather than, than film less? And that was kind of.
You know, my 1st job was to figure out, how can we take advantage of the transition from film?
To a digital imaging, and so this coming June actually marks the 30th anniversary of the operation of our films radiology department at the Baltimore vha.
And the 1st call, of course, was any image anytime in anywhere as President Bush had suggested. suggested
But the 2nd thing that I really was hoping to do was to.
Um, hope that that move to digital and advances in computer aided detection and diagnosis.
In the rapidly emerging field in the 990 s was something that we would be able to take advantage of.
When we were digital, and so, since I give a lot of lectures and talks about predictions for the future of.
Health care the future of diagnostic imaging I occasionally get asked what was the prediction that I made that where I was the farthest part of the staff.
And, you know, the 1 that I come up with is I would have been really surprised.
You know, almost 30 years ago to find out that 30 years in the future after we went digital, these algorithms algorithms or cat.
You know, are still not anywhere close to being in routine use and even 30 years later, there's almost no impact.
Of these algorithms yeah, on patient cancer care and I'm hoping to see that.
Changed pretty dramatically in the future as Dr. Nordstrom was mentioning.
When I was 1st, asked by Dr Dan solid them of the cancer imaging program to serve as lead of the ca, big imaging workspace.
1 of the pushback that I got from the ca, big leadership.
Was why should radiology.
Or imaging, even be part of the workspace, I mean, unlike genomics they told me, you know, radiology is really just an art form.
And radiologist just kind of comment subjectively on images and there really is not a real quantitative component.
And so, that was kind of that laid down the challenge for me, and laid down the gauntlet at which we worked with the cancer imaging program and many others, and created all sorts of really cool and innovative projects, including.
E pad and the annotation image markup. Annotation.
And many others that have endured today, but particularly the cancer imaging archive or the.
Uh, has it has been called, um, in the past also and so, you know, this archive was originally something that, um, we were asked to set up by the cancer imaging program.
To How's the image dataset from a, from a writer.
And and so we were essentially asked.
To create that, and, you know, my vision was to create 1 that not only, you know, would hold the and the writer data collections, but could hold multiple different collections and, you know,
in working with the NCI that resource as you all know, has evolved through all the great work of the cancer imaging program and.
And colleagues and others to be, you know, probably have the greatest influence.
On algorithm development of any effort that has existed so far and it's continuing to make major contributions.
And so, you know, my major goal for the original cancer imaging archive with it was that it could serve as a training set.
For a computer aided detection and what's now being referred to as a I.
And, and I was just at an meeting on Thursday that they referred to as Sir looking at.
How to advance ai applications? They've both both of those organizations have done great work, identifying, use cases in promoting algorithm development with competitions, such as.
And, but, you know, as was the case for or, and the many advances in imaging, and I believe the NCI now.
Needs to step in and may be in the best position to step in and play a major role in moving things forward again. And I really see that. That is a possibility, um, to go beyond the work.
That other society organizations have done and vendors have done and so this presentation is really intended to serve as a call for action to NCI.
To consider providing even greater leadership in AI, translation and implementation.
In routine cancer care this is actually 1 of Dr nordstrom's slides from and, uh, as he points out has done a great job for data collection protocols.
Clinical workflow and adoption into adoption into clinical trials. Um, some of the work on that's been done with some great algorithms.
Associated with that program with algorithms, for clinical decision, making have been really impressive.
But, um, the 1 thing that I feel as a practicing radiologist, who specializes in oncology, is what we need to do is take that next step toward clinical acceptance and regulatory and commercial hurdles.
Uh, conquered that Dr Nordstrom included as kind of what would take place after quote unquote, more years.
There are so many amazing tools that have been developed from and other initiated and NCI initiated, um, programs. There are dozens and dozens of algorithms to choose from.
The problem is, I don't have access in my routine clinical care.
To any of these algorithms, and I would love to and I'm operating, you know, for the most part really? Without these types of algorithms, as has been pointed out. Um, you know.
Um, the algorithms may offer valuable, anatomic, physiologic, metabolic and molecular information and provide insights into disease location and extent reducing the need for invasive procedures. And that's what I want.
I don't have it from my everyday routine, clinical practice in oncology and I'd like to share a little bit, you know, of why?
And also share that the fact that even though some of these are available for patients involved in clinical trials, as you all know.
You know, only even 19% of the patients at NCI designated cancer centers.
Essentially participate in clinical trials where it's only about 4% in community care programs.
Maybe 5%, integrated network, cancer programs and academic, comprehensive cancer programs. So 95%.
Of cancer patients don't have access.
To the type of tools that are available for.
Um, AI for clinical trials and how do we make those available for clinical care?
Well, this is a slide from J eerie from 2018.
But I think really, in a compelling way, demonstrates the council.
Between the imaging community that kind of has had the attitude. Well, if we build it, they will come if we fund it and we develop the algorithms, they'll figure out how to get it.
In the clinical community that says, you know, we're doing okay. And, you know, NCI, you know, sort of.
In its perspective saying, well, you know, perhaps you're, you're both wrong. Um, this is a recent.
Um, uh, um.
Media a story that talks about, you know, in my opinion, why it's so important to bring these types of tools.
To a routine cancer care. 54 year love story we were doing some tasks through ups and downs, including pads, breast cancer more than a decade ago, which is why she was so diligent about screening.
And concern when she notice changes in her other breasts.
She had imaging and a nearby hospital part of N. H, Northern Arizona health care and they told her what.
They couldn't find anything eventually paxol breast surgeon bet to Pre. I literally couldn't believe.
What I was saying, it was screaming cancer, advanced, breast cancer. 2 years later Pat was gone. Dr decree had just moved to Sedona to develop a state of the art breast center at an age.
That was just the 1st patient, she says, who came to her with a cancer mist on previous scans. These were misses that.
Or not subtle. The hospital has an exclusive contract with Northern Arizona radiology.
Degree who says she continued to flag misread mammograms.
Push them to hire dedicated breast images.
Including Mike, who couldn't believe some of what he says was missed I said, my God, that's the size of a tangerine.
I don't understand how they can call it. Normal. The hospital requested the radiology group, have those new breast images review more than 6,000 patients who scans had initially been read by a particular general. Radiologist you.
Lizzie said they found cancer in an additional 25 women. The recommendation was returned on among them. Christie Gallo. christie gallo
And so this is just a, an article and just something that's come out.
Literally, in the last few days where.
Um, this underscores a phenomenon that the largest radiology groups in the U. S. are noticing and that is.
A major discrepancy between the quality, the quality of the interpretations and accuracy between sub specialist mammography in general radiologists, and you might wonder well, what percentage of mammograms in the U.
S.
Are interpreted by fellowship training, specialist mammography in comparison with general radiologists who do this.
In addition to the regular work, and the answers, um, somewhere around 80% or so.
And so the largest radiology groups in the country are now looking at ways to be able to significantly decrease the discrepancy rate that they get with their sub specialist. specialist
Radiologists in their groups and the general radiologists.
This is an article that was published in was a network open and essentially.
What they found was that, um.
An ensemble of AI algorithms combined with radiologist assessment, significantly improved overall accuracy and so radiology groups across the country, whether it's.
Mammography, pulmonary embolism, um, stroke, detection, lung nodules.
Are increasingly becoming convinced that 1 can improve cancer care by having radiologists,
particularly non specialty specialty radiologist but radiologists in general work in combination with AI.
The challenge, as I mentioned is that ai really is in very, very few places, and very few practices.
And that's changing now, but relatively slowly. And so what are the challenges.
To bring a I, to cancer patients. Well, there's a number of them 1, is that it's difficult to know.
Which algorithms to actually pick, and also, you know, how did these commercial algorithms work and how do they compare with algorithms that have been developed with NCI funding?
Um, and then clinical acquisition protocols are much more variable in.
Clinical practice than they are in clinical trial, which makes many of these algorithms somewhat brittle when they're applied to so many different acquisition protocols.
The other question is, does the algorithm.
Have the same patient population that are using the algorithm in comparison to the.
Population that was used to train the algorithm in the 1st place.
The other question challenges, API algorithms, unlike my radiology residents, and fellows at the University of Maryland.
Don't learn or improve over time they stay static.
Until a whole new version of the algorithm has been.
Released and, you know, when we talked about machine learning, the implication is that it should learn and improve over time. How do we do that? Incorporation of personalized patient data?
And history is not being done currently in the algorithm as a radiologist. I use that information.
In the history, it's critical. Our algorithms don't do that currently lack of integration of AI into the work flow is a major issue.
My PAC systems and other systems don't actually have a workflow mechanism for me to consume most algorithms from vendors or from NCI.
Medical legal issues, including, you know, do our store, the results delivery of.
Ai algorithm should it be to just radiologists or should it be the oncologist? If I deliver them to oncologist and other non radiologists how do I do that?
The other issue is lack of a standard platform to consume these algorithms.
Security and privacy and reimbursement challenges and then the fact that there isn't really an index.
That I'm aware of, if I'm looking for a publication, I can go to pub med or Google search. There's not a mechanism for me to discover algorithms from NCI or H, or algorithms in general that may be out there.
Um, and data sets that are out there.
Also, we have relatively crude methods for screening patients. For example, our screening for lung cancer is done according to the criteria.
Smokers over the age of 50 or 55, depending on when you look at the criteria and that was great for us study.
But what I want to do is personalize or make better precision.
Determination of who should actually be screened. Fdi FDA clearance is more focused.
On the claims of the algorithms, rather than whether the algorithms work or not and it NCI algorithms are much more rigorously tested in multiple facilities than actually the majority of vendor algorithms.
That get FDA approved and so let's talk about some of the possible challenges and solutions.
Many of you are aware of the effort called quantitative imaging biomarkers alliance and what that does is creates a set of additional standards, a higher bar.
To achieve consistent, quantitative imaging results, for example, in pet the values that we use to measure metabolic activity very tremendously from 1 scanner to another, or even the same scanner from day today.
And so this represents a challenge. I've been doing some work with the and a accreditation organization to look at.
Creating incentives.
Or clinical practices to be able to have a more rigorous.
Protocol to ensure that their SUV values.
Would be more accurate and more likely to be similar to those obtained in other institutions.
And on other machines, another challenge is defining image, quality in clinical practice as many of, you know, who may have been involved with looking at images from many practices.
There's an incredible amount of variability in image quality and in dose.
I was involved in a CMS funded project with, um, the University of California, San Francisco, where we actually asked the question. Can we look at image quality? It's easy to reduce dose.
But what happens with image quality as we reduce those and was.
Willing to consider paying additional dollars.
For departments that were reducing, um, radiology dose.
For, but yet they didn't want to compromise image quality.
So I worked with, uh, Duke to look at physics characteristics and then we created some machine learning algorithms to predict how 120 readers would rate the quality of those 800 scans. And so the question was.
so the question was
You know, is quality really in the eye of the radiologist or is quality some physics metric that someone can.
Um, you know, measure, where is it a combination of the 2 and it was a really interesting seeing how a machine learning algorithm or an API algorithm could predict.
The behavior of the readers and subjective ratings better than any 1 reader could do in general.
The other challenge challenges include with regard to clinical, um, image quality.
There's really no central reading corps and so quality really differs from 1 clinical practice to the other.
There are commercial algorithms from mammography, for example, that measure patient positioning and other quality metrics, but those really have not gone into widespread use and then acquisition protocols,
like iterative reconstruction and to reduce noise have an unintended consequence of destroying texture information, which can be really important.
In, uh, AI image performance and so we need to look at.
Not only quality, but even how acquisition protocols.
Can impact the AI performance.
Another challenge is the whole idea of having an API algorithm adapt itself.
To a particular practice or learn as time goes on and so we get these algorithms from vendors that have been developed on a particular patient population.
And we want to essentially look at having it learn from our radiologists or.
Our oncologists, or from the results, and FDA has a white paper that they issued a couple years ago, looking at the intriguing possibility of actually having FDA clearance on software that learns with feedback, um,
to be able to adapt itself. I was involved in a research study that we did at University of Maryland, where we got an API algorithm from.
And we found that it worked well at University of Maryland in certain populations, but in other sub populations.
Older African American women.
Uh, or, uh, or, uh, younger male patients, we found that it was off, but that we could apply a correction factor.
That we derive using machine learning and then we could create this sort of.
Fix to the lens where we could say we're going to create a compensation factor for certain populations that our institution we take the result of the algorithm,
and we're able to increase it or decrease it, depending on on our training set. So, if we found that, um.
Uh, older African American women were consistently.
Judged as having bone mineral density.
That was lower than the actual value we had a formula for being able to essentially correct for that.
And so, that sort of thing, being able to personalize the performance of an algorithm to a particular institution or to have it learned from feedback is fortunately something that FDA is looking at.
But it's not a model that to my knowledge and.
Has really encouraged, or seen algorithms, uh, to do. Um, another, um, issue is.
Um, the FDA's interest in post markets or balance as, you know, the FDA, um, looks at postmark kits, surveillance for drugs. Yeah.
You know, we, the FDA approved this drug, but we're going to watch to see how patients on the drug do after it's been FDA cleared.
To see, and to constantly check to make sure it's safe and to make sure that it works. And so the FDA is really interested in doing the same thing for AI algorithms.
And I've been working with them on a project to create a standardized method for being able to identify data elements to report feedback on algorithm performance.
This is the American College of radiologist ai central from their data science Institute.
The categorizes FDA cleared algorithms and there's somewhere between 1,010,120 algorithms that are specifically for.
Um, AI for diagnosis or segmentation many of them are for image processing enhancement but even now there's only a relatively small number of algorithms that have been cleared another really important.
Point that ai algorithms don't do at this point.
Is incorporate clinical history I would never want to diagnose a patient with suspected cancer and as I'm looking at their pet or.
Tp scan or MRI without knowing their clinical history and there's a Beijing a priority probability of disease. That's really important to know. But these algorithms act, like, the patient has not had prior studies.
Like, the patient doesn't have any history at all.
But the patients do, of course, and so, in order to look at the impact of.
Patient information, um, I also downloaded.
Um, the, uh, screening trial, which is many of, you probably know.
Was an NCI sponsored trial for about a quarter of a 1Billion dollars.
That followed 155,000.
Of subjects over many years to determine which ones develop cancer and what cancers they developed. And as part of that study, they asked about 800 screening questions.
For example, patients who.
Um, are Hispanic have only a relative odds ratio of point.
475 of developing cancer in the next 6 years, whereas the native Hawaiian or Pacific Islander.
Has a relative risk in comparison to a Caucasian population of about 2.8 and so just knowing.
For example, that there's a higher risk of developing cancer in the next 6 years for a black patient, Hispanic patient, uh, for a patient with COPD or a family history of lung cancer,
knowing those factors 1 can and should be able to factor that in to an algorithms determination of whether or not a lesion.
Is, uh, is cancer it's something that we do in clinical care all the time.
So I took the data and created a tool that allows 1 to be able to fill in information such as smoking history gender.
Whether they're on aspirin or ibuprofen or any 1 of 800 other.
Factors and what we found in our tool was that we could, for example.
Look at the relative chance of developing about 40 different cancers based on those 155,000 patients from um, and so when we apply the.
In criteria to this population smokers above.
Uh, the age of 50 or 55.
What we found was that 7.6% increased risk of developing lung cancer.
A 2 fold increase in mortality and so I guess what I'm saying is.
For any individual patient, if you tell me about that patient.
I can essentially give you a precision based.
Likelihood that that patient's going to develop cancer.
Within the next few years, that should be something that is a major factor in determining the likelihood of detecting cancer by a particular algorithm. But we don't use that.
This is from a study that's just being published now in.
Where I am a number of a CO authors.
Looked at, um, taking the pixel data from a scan looking at lung nodules and creating a score.
Um, that essentially was a malignancy similarity index.
To look at the pixel data to determine what's the likelihood that's something that looks like this, um, is cancer. So if you have a new.
Lesion being able to take that lesion and compare it to patients whose lesions turned out to be malignant in the National long screening trial is what this paper is all about, but then combine that.
With a priori likelihood of developing.
Lung cancer, for example, based on ethnicity, smoking, history, et cetera, and you can come up with a much more refined likelihood.
And by using these tools, we can screen patients more specifically.
The patient doesn't have to be a smoker over the age of 50 but might be a Pacific island, or with a family history of lung cancer that should qualify that patient regardless.
Of whether they meet the CMS criteria, um, that patient should be screened on a regular basis. And so, as we've talked about personalized or precision medicine.
We can use these types of data for, uh, also for screening and for, um, determining which patient should be biopsy, which patient should have follow ups. And when the problem is, is.
I don't have access except for research purposes or publication.
The a vendor can't at this point to my knowledge.
Download with the idea that it would be used for the vendor's API algorithm. There isn't a model at this point the model for me to do this research is to say I'm a researcher, which I am.
I'm going to publish which I am, but I want to be able to use this in clinical practice.
And there's not that permission mechanism that exists for downloading and utilizing these data sets in those particular types of applications.
The other thing of course, is that most ai algorithms that have been developed, look at only 1 study, but don't look at trajectory of change over time. So, for mammography, for chassis scans.
How allegiant changes over time is clearly critical to me as a radiologist but yet these algorithms really, for the most part don't look at that trajectory of change over time.
In their determinations another challenge.
Is, um, essentially to integrate ai into our clinical practices and most of that is happening requires platforms at this meeting that I was at recently in general.
The consensus is that we should have platforms.
To consume these algorithms, whether they're packs.
Whether they're with our speech recognition system or advance visualization.
There's not been a standard for these types of platforms, medical legally. This has been a challenge to some extent. For example, it's still not clear whether we should be saving the AI markings.
Most people with mammography CAD systems.
We're not saving the markings, but we're deleting them.
Supposedly, for medical, legal reasons, as we move forward with a, I think it's really important that we have a mechanism to be able to save the AI.
Interpretation and recommendations rather than just deleting those, but that's been a controversial area also. And then the, the next question is.
If I have a single, um, scan of the chest, for example, and I have.
20 algorithms that are looking at, uh, coronary artery calcification, interstitial, lung disease flow dynamics, lung volumes.
Pulmonary hypertension chamber sizes, et cetera.
The, er, Doc doesn't want to see all of that information that the algorithms have created. I need a place to store those data.
So, they can become consumable by other algorithms in the future, but there's no place to put it. Do I put it inside the packs?
But that's not really discoverable outside of Radiology. Do I put it into the? There's no place in the for those data. So how, when we capture quantitative data.
Can we make it available for other algorithms subsequently without creating these unnecessary long radiology reports? Kind of like an appendix.
Essentially to a clinical paper and so there's really no effort that I know of to be able to create a standard mechanism to be able to do that. Another question is.
A lot of the AI algorithms.
I think are optimized to be consumed by oncologists or pulmonologist.
In addition to radiologists, but at this point, most of the AI algorithms are available just to the radiologist just at radiologist workstations. But I think we need to think about.
How can we get those algorithms out to being able to be used by our oncology and other colleagues? So they can do quantitative analysis and assessment in addition to that done by the radiologists.
The other question is, if I love an algorithm that's been developed by NCI or with NCI funding then how do I take that? And run it? There's no prohibition.
From a regulatory perspective against my using it for a clinical purposes.
It's just that someone can't sell it and it makes certain claims about it, but I don't have a platform.
Nor does NCI require that their API algorithms, run on a single platform and so it makes it difficult for me to be able to consume these algorithms. Even if I was given.
Um, access to them, and there's really no consensus about a platform for AI.
Security issues are a challenge for me in all places, but particularly within the and.
Dod, where there has been difficulty in consuming.
Api algorithms and software in the cloud and databases in the cloud, as, you know, there's been some groundbreaking development and reimbursement,
and that's helping us to be able to pay for AI but it's still probably 1% or a fewer of applications where there is really any model for reimbursement. reimbursement
I think we need to consider that and then.
You know, I think it's really important that we create a mechanism to index AI, algorithms and databases the natural that, um, National library of medicine does a great job of indexing publications.
I want them to index also, um, databases and algorithms that I might be able to find and use. Also this data sets.
Solicitation is really encouraging so that people contribute large datasets. But now how do I consume those and index those also?
Excuse me and then finally, you know, I'd like to extend the dot com standard and optimize it.
To, um, be able to support machine learning research for example, I'd like to have a repository and I'm working on a project where we could create research databases that would essentially represent down sampled images.
Most of the algorithms utilize small matrix sizes, like 344 by 344, you know, rather than a whole by.
Chest X, Ray, um, Matrix or a huge mammography matrix.
Most machine learning or convolutional neural network algorithms, only need small versions of the, uh, image dataset. So it would be great to create automatically a research.
You know, parallel archive, that stores images, just in that format.
And so I've kinda been hinting at opportunities for AI for NCI.
To facilitate implementation and routine, clinical practice, including creating a well defined mechanism for researchers and clinicians to find.
Data sets and algorithms, um, could NCI organize a forum or structured process?
For user rating of NCI funded algorithms and experience utilizing those algorithms would it be possible to create a repository of image acquisition protocols.
For clinical trials to be used for clinical practice can I learn from cancer trials?
To help me with my routine clinical practice. Um, could NCI sponsor grant?
That might require an algorithm that finds lymph nodes.
On a demo or pelvic scanning to learn and improve over time. All the algorithms that have been created essentially are static.
Which has some application, but how about calling for an algorithm that actually has the capability of learning with feedback from radiologists or feedback.
From pathologist, could NCI help to facilitate creating a platform that could be required for any ai algorithms that are funded by NCI?
And could we essentially revise current guidelines for screening for NCI to be personalized, based on own databases, such as CIO, or the National lung screening trial?
Can we revise guidelines for follow up with lung nodules based on these new personalized screening?
And can we create software to take com to the next level to be optimized for better encryption and safety and security and machine learning and, uh, can we create a standard mechanism for representing.
Quantitative data that that ai algorithms come up with, that wouldn't be appropriate.
A store in a patient's report and so I'm going to go ahead and.
And stop there. I know that was a lot of information in a short period of time, but I really do believe.
The NCI can once again or continue.
You know, it's amazing efforts, and I can take a leadership role in addressing some of the gaps.
For essentially taking ai from hype to reality for routine, clinical practice.
So, thank you all very much. And thank you Elliot, thank you very much. Um, that was, uh, inspirational, uh, number of questions, and suggestions for us.
Um, before I asked some questions, let me open it up and find out if anybody else has questions.
Too much overwhelming.
Your, your comments Elliot are quite apropos considering the efforts we are now going through with the, uh,
data management and sharing policy that we're going to be unrolling here in January. And a lot of your questions I think are are.
Central to how we're going to eventually want to want to be unrolling a lot of these data sharing plans.
You talk about archives, you talk about sharing and managing algorithms that sort of thing all that's got to be discussed and figured out here as we move forward.
And I'm really happy that NCI is doing that. Just as you say, Robert, because I think that will have a major positive.
Impact.
On ai for, you know, week clinical, uh.
Uh, practitioners, so I'm, I'm really happy to hear that.
I know it will take some time, but I think that could be a major force and moving things forward.
Yes, if we do it correctly, and correctly is the key word there, there's a lot of unknowns at the present time and you your presentation here can unravel quite a few of those.
If we take your slides the heart and start working on on those. I appreciate that.
Uh, 1 of the 1 of the problems with, uh, that I know with, uh, um, just general community practice is the fact that we're talking about imaging with.
With instruments that are not cheap, you know, these things are are fairly expensive and and it's a lot of community practices are lucky to have access to even 1 thing. Let alone more than 1. um, and yet.
Developers are constantly changing their, their, the way things work. They're changing the kernels that they use the changing the data acquisition rates. They're doing all sorts of things in competition with the other guys doing the same thing.
And so we end up not only with a, with a bevy of, of.
New products on the market, but we also end up with a long litany of of of things that are there in the in the community resources that are chugging along in the old fashioned way.
And this makes algorithm development exceedingly difficult to, to, uh, to verify.
You can make something work on 12 maybe even a few of these things but when you start to talk about making algorithms work, universally you really up into a problem any comments on that. that
Yeah, no, I, I think you've identified a a major issue and.
Really, of course, what you're articulating is the challenge that you've been dealing with for so many years with clinical trials. Right? I mean, as we do clinical trials.
And as we try to look at newer imaging techniques, or use of contrast, or, you know, dynamic protocols.
With clinical trials, you know, the question has been sort of have a lowest common denominator. Or do you just allow, you know, contribution from high end systems?
And, you know, even I know some clinical trials have even gone so far as to specify a particular vendor, for example, and, and I, I, I would resist doing that. I mean, we radiologists.
Have to read studies, you know, from a variety of different machines and I have a number of colleagues who do Tele, radiology where they have to read studies from maybe, you know, 50 different states.
In all sorts of different types of conditions and so, you know,
1 of the things that I think you're underscoring is the question of generalizability of these algorithms and there's been a lot of really interesting work on trying to create ways that algorithms can learn and generalize.
So that they work with many different types of scanners with different types of protocols.
And also minimize the differences and learn more rapidly the same way that we humans tend to learn a little bit more rapidly. Um, the other thing though, is that, I think that some of these algorithms.
Also should raise the bar for what we're doing with clinical practice. I mean, I've been an advisor for 1 of the accreditation groups.
In radiology and does a tremendous amount of accreditation and.
You know, really, the question is in 2022, what should be the standard.
For imaging patients and, you know, should we be using some of these.
You know, really slower, older MRI scanners or pet scanners and.
You know, at what point should we essentially, you know, draw the line and I think ai may help us. So I think it may go in both directions.
I agree that it's a major challenge for AI, but I think as we move toward the use of a, I see, I should maybe help us.
Qualify what types of protocols and scanners are optimal for that ai. And as time goes on to make those changes I realize that the money may not be there everywhere to be able to buy new equipment.
But I think as we move forward with cancer care.
I think we, a, as you suggest, have to take into account the reality of today's situation.
And economic issues, but also, I think it's incumbent on the NCI to sort of set a bar for, you know, where some of these systems should be. If you're practicing, you know, community.
At least community, standard of care in 2022 and that with a relatively significantly higher bar for what? That standard today.
Good point. Very good point. Okay. Data quality here. Great. Talk. Thank you. Very much. I think 2 comments 1 is on the Pre down sample images.
You mentioned the other on standardization dot com and so on on the Pre down sampled images in the imaging data comments project.
We, we've chosen not to provide those, but rather to assume that everybody will take our images, which may, which come from and do the down sampling themselves as appropriate for their project.
And, you know, as a, as a radiologist to spend all this time, trying to optimize spatial and contrast resolution, and, you know, expensive monitors to be able to see subtle things. Do you think that in future.
A, I is actually going to make use of that information that radiologists have claimed to use for, you know, 50 or 100 years or or do we really not will not need all that information and if the machine can get away without it, what kind of the person on the dotcom side,
we do have efforts to standardize platform interfaces, output, results, communicating with the via fire and so on. But to a large extent, those have kind of stalled out because there doesn't seem to be a lot of commercial.
commercial
To implement those in products on either either the producer or the consumer side.
So it sort of remains to be seen when we'll reach the tipping point when there's enough use of AI clinically to to pressure the vendors to implement standards.
Yeah, so 1st of all David, it's great to hear your voice and it's great to connect with you and thanks so much for tuning in and sharing your your feelings.
I'd love the 1st question that you asked, which is really, I mean, both of us have worked really hard looking at monitor standards and looking at resolution standards.
I helped to create the standard requirements for mammography for, and secondarily for, you know, displays and matrix sizes.
And so, you know, it's always been a challenge for me to see every, virtually every machine learning or every, you know, convolutional, neural network.
Paper looking at subtle things, such as interstitial, lung disease and down sampling 2 or 344 by 384 or lower matrix so that it fits within the ability of the computational, um, algorithms.
And so, um, number 1 yes, I, I do believe. um number one yes i i do believe
That, um, there is an increasing amount of interest and work, and I've been involved in a project at University of Maryland where we're looking at much larger data sets and efficient ways to be able to take full advantage of the dataset.
You know, 99.9% of studies that are out there are still using those tremendously down sample data sets. And my proposal was just, you know, given that virtually everybody is still using those as we develop new techniques.
For larger data sets that wouldn't it be great to have repositories? Maybe such as.
You know, the cancer imaging archive that also would allow downloading at that matrix size since.
That's what everyone's using to, uh, to, to learn on rather than requiring downloading a very large other data sets and and the same for maybe at University of Maryland. I could take a downloaded sample. Every time I create a new image.
And put that into a research file, I'd love to see us using larger data sets and I've again been doing research in that area.
1, other thing is, I'm really encouraged to hear what you all are up to, with regard to com, and, you know, maybe looking at higher levels of security, privacy, more efficient ways to store and inter, operate, um,
images and I strongly am a big believer that that should all stay within the dotcom standard?
I'm hoping that some of these API algorithms as they come out will encourage or re, encourage people to Relook at some of the great projects that you.
That initiated and maybe make some of those more important also. So I think the challenge is, to what extent can ai algorithms.
Help to kick start and help to accelerate some of those processes with.
You know, with com and but for right now, since everyone's using those down sample data sets.
I think it would be really interesting to create more explicit mechanisms.
To create repositories at that pixel size too. So thanks for the question and the comments. And I really agree with what you said.
Great, well, in the, uh, looking at our washes here at the time. Um, Kevin, do you want to take over and finish this up for us?
Sure, thank you. Bob, thank you Elliot for a great presentation. It was thought provoking and like to follow up on some of the ideas shared. So, um, I think we probably have time for a quick update from, by Justin.
Kirby IDC update was given recently and I'll just say that the next speaker is actually David Cooney who was just asking a question um, for next month? Uh.
He's been leading the imaging, uh, medical imaging, the identification task group effort, and they've prepared a nice, um.
Report and he will be giving a presentation on the report from the task.
Thank you I'll turn it over to Justin. Kill me.
My internet's been a little spotty, so hopefully, this comes through. Okay. Um.
So just a quick set of updates, uh.
The major thing this month that we wanted to highlight is that we've published our 1st annotation data set.
From drive from clinical trial network.
Um, this is part of the, an NCI funded project where we're going to be, um.
Basically, putting a priority queue of some of the various MCI clinical trials that have been published on on already and generating tumor segmentations in and also seed points,
the hope is to expedite research and automated tumor segmentation and detection. So the very 1st data set this, or 382 has been published, and there should be a number of new ones coming down the line in the next. line in the next
3 months.
Then we also had a couple of other data set updates the, there's 1 with pancreatic Neuro and to consumers imaging. This is a collection of 38 patients with demographics and survival details.
details
Which can be used to study imaging characteristics of these rare tumors.
Then we had 2 glioma related data sets that had, uh, the 1st 1 is from preoperative MRIs.
This is a 501 subjects with preoperative brain tumor imaging, advanced fusion and profusion techniques.
And then the other 1 is an update to an existing collection, you.
That 1 had been published previously with 6, I think 600 subjects the histology imaging has been added for, for a number of those so that you can use those as well. Both of these data sets.
I believe we're heavily utilized in the brain tumor segmentation challenge. That was conducted last year and so we're actually working with the organizers to bring the rest of the data that we're part of that together. So that. that
In the coming months, you'll be able to grab all of the different components of the data from that challenge.
Last thing is just a quick infrastructure note. We've been posting the National log screening trial dataset for quite some time due to the size of the data set.
It's been running on its own server separate from all of the other data sets that we host and up until recently, there was no API access available to that. We finally be able to get that sorted out. And so now, if you're.
Data scientist or programmer looking to get at the large data set through.
A rest API that's now possible using the documentation here. All of these things are available on our Twitter, Facebook and LinkedIn pages.
So, if you're not following us already, please check those out. That's all I have for this month.
Great Thank you Justin. Well, that's all. I have.
Okay, well, that's all we have for today. Um, I think our speaker wants more. Thank everybody for being a part of this. I think it was very interesting and productive and we'll talk to David cleaning next month and we'll see what he has to say.
So, thank you for joining us.
Thank you thank you. Looking forward to David start.