Okay, uh, I hand it over to you um.
It is just 1 minute past 1 so please decide if we wait.
For not a minute or 2 before you before we get started.
I see people are still joining, we can made another minute.
How long is your presentation?
Um, I'll try to keep it around 40, 45 minutes so we have plenty of time for discussions.
Okay, well we do have some items, um, update.
Items after the presentation after Q and a, of course. Um.
Why don't we get started when I hit 50 participants?
Huh.
Okay, let's get started. Um, well, thanks everyone for joining. Uh, this is the last.
Webinar for, uh, spring of 2020, and before we go on a summer high, it is till next fall where we pick up the 2 webinars again. So today's my pleasure to welcome Dr from Stanford. Dr.
Russo has received their PhD in computational by, by medicine from University of Texas, following post structural training at Case Western on multimodal image analysis. She spent some time at G, Google research as a.
Scientist before joining the faculty of radiology at Stanford in 2018.
Dr Russo is the recipient of 7 patents in medical image analysis and an NCI or 37 Marriott award at Stanford leads the personalized integrative medicine laboratory,
focusing on developing analytic methods to especially align radiology and pathology images in patients that undergo surgery for cancer treatment.
She was very kind to accept our invitation to this webinar to share her insights and research results in bringing bridging the gap.
Between radiology and pathology through Informatics and machine learning without further ado. It's my pleasure to welcome Dr. vulnerability Russo to give her presentation. Dr Russo please.
Um, I hope everybody can hear me, uh, and see my screen at the same time. Perfect. Uh, Kevin, thank you so much for the introduction.
And, uh, thank you so much for the invitation and it's my greatest pleasure today to present some of the research that, uh, my team has been performing in the last, uh, 344 years.
And today's stock is focused on the bridging the gap between, uh, radiology, pathology of the prostate through machine learning.
So, as a disclosure, um, I'm a consultant for Rush and, um, I do have some research grant from g, healthcare and Phillips healthcare. None of that is relevant for the presentation. Today, as Kevin mentioned. I lead the personalized integrity medicine laboratory.
Um, and in my team, the focus is to develop computational methods to improve the interpretation of radiology images. Um, specifically, we focused on localizing cancer.
Um,
not only that,
but also distinguishing subtypes and all the steps that are required to get to that ultimate goal,
which involves,
um,
doing registration between radiology and pathology data as well as developing other methods that are focused on,
um,
Oregon segmentation like here shown in the case of the prostate of course.
Um, for the strategies that we're using my lab, um, they are mainly, um, machine learning, based, um, and with a special, um, focus, their multi model data integration. Um, and we'll see all of that today.
Um, we work on a variety of applications, prostate, breast brain um, we're looking also, like, in the spine. Um, today I'm only gonna talk about the process because the story there is a lot more complete in terms of what we have contributed to the field.
So, I, I think this is more appropriate for today's presentation.
With that, um, I would like to also tank the folks in my team that have been instrumental for developing this method and achieve business accomplishments. So you can see them here.
And this work wouldn't have happened without the contribution of Dr Jefferson and return funds from the neurology department.
And then all the other folks that work with us in collaboration, Christian candor, thank you for all the pathology annotation edgy on the radiology side and, uh, Jim Brooks. Um, kind of as an advisor for how we drive this so.
Why are we looking at prostate cancer? Well, known probably by now, that is the most diagnosed cancer in American men about 20% of full cancer. Um, and it is the 2nd, leading cause of that, in American men 10%.
So if we look at, um, clinical care Mr. Smith will undergo some sort of screening, either having a test with collaboration of a digital rectal exam. And if there is, there are any concerns.
Um,
the,
the person Mr Smith will,
we'll go and see if the diagnosis of prostate cancer can be established,
which usually that is done through a,
um,
ultrasound guided biopsy and Ryan,
this context comes as a way of driving that ultrasound when it's used.
Um.
As a way of identifying the lesions.
If a biopsy comes back as being positive, a certain form of treatment is chosen for that patient. And here, I'm showing the patients that we'll get.
Radical perspective, and this is the pathology data that is being collected as part of that pathology processing. So we can see there is a variety of images that are acquired through the process of diagnosis and treating the prostate cancer.
So, in this context, prostate is being increasingly used to either got that biopsy and then guide some form of local treatment while prostate them. Right?
It's a very good modality to see the contrast of soft tissue contrast. It's still not perfect. Cancer can be missed and depending on which cohort 1 looks at this number can range between 12 to 34% of clinically significant cancers.
There could be a lot of false positives.
Um, this are, um, um, lesions that require biopsy confirmation. So the patient gets an extra biopsy or a extra needles being targeted during the biopsy procedure.
And at the same time, um, there could be large variation in, um, the interpretation of, uh, the.
So, in this context, the goal of my team is and the promise of is to address these shortcomings to be able to help clinicians, find those cancers that I've missed traditionally, by radiologist to reduce the number of false positive that required biopsy confirmation.
And then decrease the injury, their viability. And the way we go about this is that we are looking to do 1 more thing in addition to all of this, which is to guide the needle towards the aggressive cancer.
Um, specifically, when this lesion coexist, when you have aggressive and nimble and cancer within the same within the same vision. So we developed automated methods to distinguish aggressive from England, prostate cancer and I'm right.
And then, our novelty is that we bring the pathology information, and it's not only the labels, but also the images into the space. And I'll show you how we go about that.
Um, so, because we are looking at doing that, we start our methods with a certain group of patients those that receive radical perspective.
These patients will have the perioperative and then we'll have the pathology of the whole gland as it's shown here. Our very, very patient. Um, um.
Pathologist will outline where cancer is present, and maybe will tell us where England cancer is and where aggressive lesions are.
And through the registration between radiology and pathology, we can map those labels of aggressive and England cancer on to the right.
However, 1 of the things that, um, we have, um, kind of looked at is that 1, the pathology information provides a wealth of information is not only that we can tell where there is clinically significant and in the LinkedIn.
So we can also know whether we have extra static extensions, but not only that there is a large body of work focused on automatically analyzing the pathology images.
There was a recent Panda challenge, for instance, that is focused on decent grading on the pathology data. So, we took this data and this concept of of.
Extracting information,
um,
uh,
through an automatic fashion,
and we wanted to now also get in addition to all the things where we get labels from our pathologist,
uh,
to also get labels from,
um,
automatic approaches to do Gleason grading and specifically,
they can distinguish the different Gleason patterns that are indicative whether that cancer is indolent of,
or or aggressive.
So, um, and by using automated approaches, we also.
Go around 1 of the issues related to how teen years it is to actually annotate, um, here's the pathology data for the purpose of bringing it that bridging the gap. Um, towards the.
So, for instance, here, I'm showing you examples of how the human annotations, and the, um, digital pathology standardization called that. Digital pathologist is actually the model develop to detect lease and rates on the pathology data.
So we compared the human annotation with the.
The digital pathologist, uh, annotation, I'm gonna zoom in this area so we can look a little bit more carefully and I'm going to go to the different colors that we are seeing here. So, what we've seen in in blue.
And, yeah, green, those are the 2 labels coming from the automated API detection of listen rates. Blue is decent grade, uh, 3 and then, uh, green is listened rate for.
And,
um,
on top of that we have the annotation of our pathologist so we have in black what the pathologist has outlined is the same person that outlines all the things but in black,
he has outlined what,
um,
it is cancer and then in red,
um,
his note said,
um.
Decent score 3+4 with a lot of 4 in orange visa score 3+4 with some 4 and then everything that is yellow is releases core 3+3.
So, what we can see here is that, um, the results of the deep learning applied on the pathology to develop or extrapolation rates match very well, the ones of our pathologist. But we are starting to see a number of things here.
1, but the labels are not always consistent when the human is annotating them while that is happening, because this really is really take a lot of time to annotate this. Um, and this are very people very busy people that are doing this annotation.
So, um.
It's kind of important, um, um, that they do annotate everything, but at the same time, they do their best with the annotations. Um, and we can see another problem or another difference.
If you are not a problem, but another difference there is really a difference in resolution between the annotation in that the, um, the.
Api for detecting listened grades really outlines.
Uh, it really outlines glance and has a lot more cavities. Um, whereas the human goes in and create these more smooth presentations, but, uh, in outlines, but generally they match very well.
Um, and and then we have, um, decided that, um, this would be a good way to label our data. And we have a variety of, um.
We have a variety of, um, information that is being projected. Now from the pathology onto the radiology. There is a, uh, chat, uh, question, whether this is, uh, training a supervised question. Um, this is not work.
That comes from my team by the way. The 1, that is related to doing listed grades of pathology. There are a ton of approaches. They generally do have supervised, uh, labels. So it's supervised learning. So they do have labels for their training. But you can look at the pundit challenge.
Um, they have a lot of data in in that, um, in that challenge.
So with that, um, now we can go back once we do the registration, we can go back and and, um, uh, compare the labels that we get from pathology and the ones that we get from radiology.
And, um, we are, uh, uh, I hope I convince you by now that it makes sense to actually, uh, take the data from pathology, do the registration with the to get the labels. Um, but I want to point a couple of other issues related, um, to.
Um, essentially using radiologist labels and that is that, um, 1 radiologist as I said, they might miss solutions. And here we can see, they didn't outline the, the England region that we see here in blue.
Um, but, in addition, um, we do know that the, um, extent of cancer or memorize sometimes is, um, less than we see on pathology. So, in other words, the, the size of the lesion is under, uh, estimated by.
Um, and then, um, some of the features of the future of cancer make the outline a little challenging.
Um, so then there are differences in outlines and also there is always the question of emery visible and invisible lesions, especially the invisible lesions are very hard for radiologist to outline. But of course, if we, um.
Um, if we use pathology, we can actually address some of those shortcomings, so with that our contribution to this space has been to develop a variety of methods to distinguish indolent and aggressive cancer.
So them, right and to do, so we have developed intermediate Pre processing steps, focused on segmenting the prostate on labeling the data and then essentially training and the best way possible. This, this models.
Um, so now I'll go into the details of what that actually practically means, and how we go about doing, uh, this, uh, different steps, uh, required to achieve that, that, uh, outcome.
So, 1 of our 1st studies, um, it's called Rhapsody and it stands for radiology pathology, open source, multi, dimensional integration platform.
So,
Rhapsody,
um,
is a work that has,
um,
registered radiology and pathology data assuming there is a slight correspondence between the all the pathology samples,
or the pathology slices coming from the whole month.
And then the, um, the radical perspective. Me and the, we actually achieve that by using 3 D printed modes, which is standard of care at our hospital. Thanks to Dr sun with that.
Um, our approach reps are the, uh, 1st, uh, focuses of reconstructing the histology um.
Sample, um, and then we do a paradise registration between all this, uh, slices and with that information we use it as a way of, uh, refining the alignment between the whole month, um, pathology with the.
And then, with that, we get a 1, the extent of cancer onto the. Right? But we also get a pixel 2 Pixel, um, matching between the radiology pathology, which we'll we'll see in a 2nd way. That's relevant.
Um, here, I'm showing you a 3 D, reconstruction of what Rhapsody does, um, achieve, and then kind of how it looks when you look at it in a slide by slide, and I can go back and forth between them. But you can see the level of of details and these are our human annotations.
So, as I said, our pathology is very, very patient and we, we appreciate so much the time. He spends to annotate this.
But through the user Rhapsody,
now we can actually see really how the pathology information matches,
the radiology,
um,
image,
and we can see the different features aligning and cancers and and,
and,
and kind of where,
which parts we see.
With Mike and I,
um,
as well as all the other features,
that are are relevant for this,
um,
we didn't want to,
um,
only look at this with traditional registration approaches,
which is what Rhapsody does we also wanted to speed up this work and address other challenges.
But, um, in the 1st step, we wanted to speed up the work and then we did that by using deep learning and the deep learning approach. Um.
Essentially works very similar as the traditional Rhapsody approach we still as human slice to slice correspondence.
Um,
uh,
but then instead of doing the security registration that we,
um,
do for traditional approaches,
we actually train a deep learning approach that will learn and predict,
uh,
the different parameters that are needed in order to align these 2 data sets,
both defined and deformable and ultimately,
we'll get a similar approach as with Rhapsody.
Um, the good thing about, um.
Machine learning approaches and in this context, um, the way this approach works is that we can train with medical data, but at the same time, we also can frame with natural data.
So we can simulate this kind of information that allows us to learn how to reduce their data sets.
And in here, um, we use the PASCO data set, um, where we randomly apply transformation that our network is learning, in addition to re, using natural images. We also use medical images, um, from our cohort. Um.
Um, um, and then instead of applied, because we have a lot less data, we applied 50 random transformation. Um.
There is a question whether this is a 2 D or a 3. D. this what I'm talking about here it's a 2 D uh, to to the registration at least, at this moment I'll talk about in a 2nd.
Um,
so,
um,
what we have shown with our deep learning approach is that,
uh,
while we train with data coming from Stanford,
which is the 1st row,
um,
the approach generalizes very well in,
um,
other cohorts cohort,
2 and 3,
and then shout out to the this both cohorts come from the cancer imaging archive.
Um, and, um, the approach generalizes very well, um, in the 2 other cohorts that we are able to register even though the pathology has a very different nature compared to ours. And I'm giving you a little bit more information here.
This is so the whole month of the quadrants that had been stitched together. Um, and then, um.
This is kind of a lower resolution version of, uh, the histology images. Um, so the different cohorts are very different. And, um, they're also the comes from different vendors compared to the data that we use from Stanford.
So, this is very good because, um, we see that our approach is generalize very well. Now, when we compare the traditional registration and the deep learning registration, in way of, um, comparison, um.
And we use a variety of metrics to know how well, they work. They actually are kind of competing with each other. They get very similar dice coefficient. They get very similar. How's their defense with the, um, that means the distance between the prostate boundaries stays the same.
And and that also means that usually our errors are within 2 millimeters on the border of the prospect. And when we look within the inside of the prostate, we can see that our errors are getting, um.
Maybe a little larger 2.2.32.6, um.
Um, but generally, we are below 3 millimeters of errors within within the prospect. But the advantage of using deep learning is that it works a lot faster compared to the traditional approach.
Um, um, we, we do have 264, uh, seconds, um, with the traditional approach in average. But then, um, with the deep learning, this is a lot faster. So, um, that gives us that, um.
Um, uh, gives us a little bit an advantage there. I think there's a question about using, uh, uh, 3 D printed mode maybe, since that's a little bit of a longer question I'll address that, maybe at the end.
Um,
so,
um,
with that,
we see that our approaches are very successful,
and we wanted to take this to the next step and actually,
um,
go around 1 of the,
uh,
assumptions that the data have,
um,
or that these methods have,
as we are developing them that there is a slice of slice correspondence between the and histology.
That's not always the case. Many people don't have 3 different modes. Many people don't do 3 D printed modes for routine, uh, clinical care. Um, so, um, and our idea was to perform this as a registration.
However, the problem with performing this as a treaty registration is the data is not really we actually call it 2 and a half D and we call it 2 and a half fee because we have a very high resolution in line.
Like,
this is a,
um,
and try so,
the resolution is very good,
but when we look through plane,
um,
that that resolution is 10 times lower,
um,
both in the pixel size,
but of course,
level of details and this is even worse for the pathology because in the original resolution,
the pathology scan with a pixel size of 0.5 microphones.
Um, the distance between slices is roughly 4 millimeters. So now, even when we process it to do the registration, that will bring us to 60 microphones, uh, by 16.
um, this is still very, um, high resolution in place, very low, low resolution on a plane. What does that practically mean? So we are looking here at the same, um, axial.
T2 weighted. We are looking in the axial plain. We see a very nice resolution. We see the prostate. Um, but when you look in the satchel plane conditions is rarely look at this in this view. Um, um, we'll see.
Kind of the difference in pixel sizes between the 2 dimensions. But also if I'm, um, I go to the next. You probably are going to say well, but there is a yes, there is the surgical gateway.
That has beautiful resolution into in the satchel plane. But then, when we look at the axial, we have the same problem. Um, and then as the correspondence here, I'm showing you the pathology nice, um, axial resolution.
Um, but then, um.
Uh, when we look in the surgical plane, the pixel sizes are, um, a lot bigger.
So, um, and just kind of as a refresher here, or as a reminder on the top row, we are seeing exactly the same slides across the 3, different data sets.
Um, same with the with the lower 1, um, same data sets and prostate, uh, within this entire, all these panels. But just now we are looking at the surgical plan and the different resolution.
So our approach to address the fact that there is a different resolution between the in plain and out of plain information has been to use Super resolution and more specifically to look at Super resolution, generative, adversarial networks.
Um, and the idea is that, um, we want to input 2 images.
Um, and then we want to create the data, the image that will be coming between the 2 of them and we use a generative adversarial networks because allow allows us to actually generate this information.
While also, um, in a, um, adversarial fashion is being trained to see whether the generated image, um, is similar to the ground truth.
When does discriminator cannot distinguish whether,
um,
they are,
they're the same or different,
then,
um,
we,
um,
essentially have done are done trading this generator because the generator is actually good enough to create intermediate,
um,
slices.
So the way we apply this is that we take 2 consecutive slices from the plain, um, we create an intermediate slice, which will come here in the middle spatially. And then, at the same time, we also create multiple iterations of that.
So, we have multiple slices being created in between the original 2, uh, slices. Um.
So, um, here we are, I'm showing you some of the results of how this looks like. So we start with this is our input we push it through the Super resolve, uh, uh, Super resolution Gan and we get the Super results. Em, right.
And then we can compare it, definitely with the ground truth, it's not yet at the level of how the ground it looks like but it's close enough for doing the registration, which is the goal of this work.
Um, similarly, um, we have applied to some pathology now, um, in the case of we do have that.
So we have actually ground truth of how this, uh, intermediate slices should look like, um, in the case of pathology that information does not exist.
Um, but nonetheless, we can apply our train network to create this intermediate data, intermediate license between the existing slices. And this is the input volume how it looks.
And then here in yellow, we can see the outline of cancer and then, um, when we applied a Super resolution approach, we can actually, um, see.
That it nicely catches and creates the data intermediate, uh, within the cancer and we start actually seeing some of the, um, features of the process.
So, with that, um, once we have applied our Ganz, we can, uh, create this data sets that I mentioned original data is not.
And then now that we do have 3, uh, 3 data sets, we can apply traditional registration methods to align this data in 3 dimensions. Um.
And this is just a representation, kind of how that registration happens in, which we, we take, we bring the pathology into the space of Ryan.
We do this registration in 3 dimensions and in our work, we have shown that this registered data, um, that uses Super resolved.
Uh,
or this registration using Super resolve Emory histology achieves a better alignment between the prostate,
um,
between the,
uh,
emery and histology compared to other approaches either using directly to 2 and a half D,
and consider that as being freely or using other,
like,
simpler interpolation methods.
Um.
So, to summarize our work in this space, we have developed Rhapsody and a variety of different approaches, following Rhapsody to address the different aspects of registering radiology and pathology at the beginning.
We assume that we have a slice to slice correspondence and we continue to assume that for some, some of our work.
Um, but now we are starting to actually look towards, uh, dropping some of the other requirements, um, that this approach is have some of them being the fact that they rely heavily on the prospect segmentation.
So, this year, um, where you shall post doc, in my team has show how we can register radiology and pathology data without prostate segmentation. That's actually, um, the segmenting the prostate is not very challenging. Yes.
Segmenting a process that is consistent between the radiology and pathology. So, we make that aligning alignment better. It's a different it's a different question. So he has been showing how we can do that in a weekly supervised fashion. Um.
And then, um, the other part is kind of another feature of our approach is, um.
Is that we test this data and this approaches or multi institutional data uh, very often using the cancer imaging archive as our source of additional data with that also a shout out shout out to way.
Um, he will be part of my team, uh, going to become an assistant professor at the University of Florida. Um, and I'm going to be very sad to see him work, because he has been instrumental.
I've seen going, um, because he has been instrumental in developing, um, some of our registration approaches. Um, but with that being said, um, let's go to the exciting part of how we use this data.
So, we have develop a variety of methods to align the radiology and pathology data. As I said, our concept here was that now that we have is registered.
We can use automated, deep learning approaches to do the decent rating on the pathology side through the registration. We get that onto the, and now we can just simply take those labels and start training our deep learning networks.
So,
the 1st,
um,
network that we have developed this called SBC net,
stands for Stanford,
prostate cancer network,
which is based on a holistic nested edge detector network,
takes as an input,
that image and then the ABC,
and at the end,
um,
through a series of convolution.
And batch normalization, we are getting the prediction of, uh, in the London aggressive cancers.
Um, so that was our baseline. We have shown that with this approach, um, we can already create, um.
Labels and and separate within lesions that have Co, existing indolent and aggressive, and we can, uh, train these models to actually identify the aggressive components within these lesions.
Now, um, that that became kind of our, um, baseline approach to do the detection of aggressive and indolent. Um, lesions, but 1 of the things that we had was the fact that we don't only have labels.
We actually have pixels 2, pixel correspondences between them. Right? And histology. So, then you run it by bacteria.
A post op in my team has developed this approach called that learns from pathology information in a 2 step approach in which we learn and memorize signatures of aggressive and in deland cancer. And then we use that in a deep learning network.
So, I'm going to tell you now, how we do it.
So, the idea is that we develop features or extract features from, um, and, um, for this that we do features, um, we wanted to keep it simple.
So we wanted to just have some textural features on a per pixel basis, um, from the we wanted to have the same thing from the pathology. So, um, we pushed the pathology information through network as well.
And now we can have features on. And then we can have feature on pathology and with this features now we can start, um, using this, uh, correlated feature, learning approaches specifically. Cornell is the model. That is using that.
That is looking at. How can you combine features such that they correlate with a combination of pathology features?
And, um, what the network does is, um, learn.
How 1 can, um.
Um, essentially, um, combine these features in a linear fashion.
We have a very simple model,
um,
such that they,
they correlate with the pathology features and when we learn that we can use those correlated feature with the weights that have been learned in that,
uh,
coordinate approach as an additional source of information in our network.
It becomes kind of like a texture, um, but it's a texture that is learned to the correlation with pathology 1 thing that you notice in this schematic step 2, there is no pathology input per se, in this learning.
We a step because we already know how these features look and how to combine them to great, um, features that are correlated with the pathology. So now we can apply it. Um.
And we can learn again, how to distinguish between aggressive and interline that improve on our previous, uh, on our previous approaches.
And the reason why I mentioned the fact that pathology is not an input, because of inference times this approach can be used before the pathology has been acquired, which is, of course, what we wanted to do. We don't want to have the pathology as an input.
Now, showing you a little bit more how these features look like. So this is our example of, uh, features extracted from these are example of feature extracted from ABC.
So as soon as you look at them, they are enhancing and emphasizing certain features.
Textures within, within this image, the same, we can see a feature extracted from pathology and we can see how this.
Amber, I biomarkers, so this correlated weighted averages of feature look like.
Um, relative to the original input data.
And we can see, um, that the cancer part, which is actually in here and, um, pointed by the red arrow is emphasized compared to the rest of the of the tissue. And then, um, this.
Set of feature correlates with this set of features from technology.
Um, so well, with that core Signia our approach to do, that has been shown to, uh, produce an outperform our original SBC, um, approach and and here we can see the different ground through.
Then, we can see also we test this in normal patients, patients without cancer to also make sure that, um, that that step, uh, that still works in patients without without cancer.
So, we compared, um, our approach with, uh, with all, um, with other approaches. 1, our baseline SBC net, but we also look the unit. We also compare this with a more fair representation of unit that we call branch unit.
Um, and what we observed is that, um, very often core core Signia. Um.
Um,
it's it's the best and then overall comparing looking at all this measure,
um,
always,
always wins and,
um,
we do this,
uh,
with a variety of ways of assessing how course Signia how well,
persignia works and,
um,
and also 1 of the interesting things about our studies is the fact that,
um,
we,
uh,
training our patients will always training our patients plus minus.
Sometimes we have normal cases. Sometimes. Um, um, we include more often than not likely we have, including we have been, including them, but we trained in cases and we always testing patients that get biopsy. That's a very different distribution of cancer.
Um, those, uh, everybody in the cohort will have cancer. Um, but not everybody in a biopsy cohorts will have cancer.
So 1 of the questions that we have been asking ourselves, and this is a study that, uh, has, um, uh, got published, uh, uh, last.
May 28, so a few days ago, it came, um, uh, available as.
Okay, we do have the pathology label. It takes a tremendous amount of effort to get this, uh, this labels. Um, does it really.
Provide an advantage compared to the radiologist labels. So then, um.
Um, so 1 of the questions that we have addressed is what happens if we start training these networks with the different labels that 1 can create for them when we could take the radiologist labels.
We can obviously use the ones that are, um.
Uh, confirmed to be cancer 2. we have the, the pathologist outlines of cancer. Um, they don't go and outline glycin grades.
It's just too much work so from the pathologist, we have, um, cancer, no cancer, but we can use the digital pathology approaches to say. Well, this lesion has a lot of a pattern for or above, and then it's, it's considered to be aggressive.
Um, and then similarly, definitely we can train as we have been doing it using the deep learning labels, or we can also do a more legion level.
The deep learning approaches, so instead of saying, on a per pixel basis, this aggressive indolent, um, we actually can say this entire lesion has a lot of aggressive pixels. So let's call all the pixels within this lesion aggressive.
So, essentially, we have 4 conditions that we wanted to compare and answer the questions. What are the best labels to train? Um, and the interesting thing about, um.
Um, doing this study and have these labels, is that we can actually start looking at how well, the radiologist labels. Um.
Match the pathologist,
we can look at,
uh,
what is the,
which we see in this,
uh,
this panel number B but we can also look at how good the dice coefficient is between the radiologist labels and the pathology labels for the same lesions.
Um, so that's the number that we see here so the groundproof labels no learning at this moment, we're just looking at the labels and assessing them between the radiologist labels right here and then the pathologist labels, the dice coefficient is 0.24.
so, when we look at our deep learning approaches, and we tried to assess whether we have a good enough, um, direct coefficient to the ground truth, we also have to keep our expectations realistic. So sometimes we see low desk efficient.
So we can, um, we can actually answer those questions so we have looked and compare all these situations and obviously we have looked to see how well, the deep learning approaches compare with pathologist. The dice coefficients are.
Not 1, they are 0.8 and that is because of that difference in resolution between the labels uh, or a lot of that difference is caused by the fact that they have a different resolution.
So, the level of details is a little different and then, um, um, so that's when we look at the dice coefficient. But when we look at how well, they match, we can see that the so there is a very good concordance between the, uh, the digital pathologist.
Uh, labels and the pathologist labels. Um, and then also we can see kind of, uh, how many, um, lesions that the radiologist has, uh, outline match the ones from pathology. So that's 0.75 for the radiologist.
We do the same we can do the same with aggressive cancer. Um, so looking at those lesions that have a decent.
4, and above at least, uh, some of it, so with that we have done the experiment and said, okay, let's bring 4 models, we're gonna train 1 model with the radiologist labels. We're gonna train 1 model with the pathology labels.
And then we're gonna train our 2 models with the different components, different versions of it more important. The 1 that I showed the SBC net is always trained with with this, um, per pixel assessment of leasing grades.
And, um, because we do have all these labels we can, um, go ahead and, uh, compare each of these 4 models that we're trained with different labels with the different labels that we have.
Um, so, if we train with the radiologist labels, how do they compare in terms of the ice coefficient with the ground truth that comes from the pathologist or digital pathologists?
Um, and similarly, if we train with the deep learning approach for labeling the pathology, what happens when we compare those with the radiology labels? So overall what we can see, uh, from this plot, is that.
The approaches that use the digital pathologies, the EHR approach to label uh, the pathology images are, um.
Um, detecting, uh, they have an ACC of, uh, 0.94 compared to the ground truth, which in this context is considered to be the pathologist.
So, this, this is a, um, and then we can observe, um, to that, um, the radiologist, the model strain with the radiologist. They still have very good, um, um, dispositions. So it's, it's quite interesting.
And then again, as I said, um, we do compare this our results, um, with, um, the, um.
Um, uh, or we look at what happens when we get these models that are training the cases. So radical prospective patients, um, into the biopsy, uh, cohorts now, the biopsy patients never get an article perspective. You mean.
The ones that we consider in this biopsy cohort, uh, don't end up having a radical perspective. So we don't know exactly where cancer is.
Um, so, we, the only label we have is the radiology label with confirmation, whether that's biopsy or not. So, in the 2nd cohort, the model train with the radiology label is still the 1 that actually gets the best.
But there is an advantage that being said there is an advantage of using the digital pathologist, which is this, um, this version here because this model, the 1 that has the pathology labels are the per pixel level of aggressive and indolent.
Actually, um, uh, is the only model that is able to learn to distinguish between the different components that we have within this within these lesions. I do see, there are a couple of questions the chat.
Um, I'll address those when, when when I'm done, just cause they're a little longer and that will.
Um, affect the type I, I have to to present, so, um, to conclude on this, uh, on this work. So we have, um, develop a variety of approaches, um, to distinguish indolent, aggressive.
Um,
and we are comparing always our approaches,
um,
between the different flavors of how we use the pathology data and then,
whether we look at aggressive and handle and then we use,
uh,
image correlation and then,
uh,
with that,
uh,
a big thank you for interrupting by the carrier another post doc,
um,
in my team that is driving many of this,
uh,
studies.
So,
um,
the last thing that I want to show to you today is a recent work that was just accepted on Friday,
um,
that,
um,
has been focused on taking this models,
um,
that we develop,
or cancelled that detection and start looking at how they work in the case of extra static extensions so we know that,
um.
Extra static, uh, um, extensions of cancer are cause of recurrence or or they increase the likelihood of recurrence of prostate cancer. Um, and, um, but many patients do have them and through a successful surgery.
Um, that may be a good, um, they still have a good outcome, but it will be very important to be able to say this cancer actually has extra prothetic extension.
So we took this models that we train to detect cancer. Um, we applied, uh, within the prospect, but also outside the prostate to see, um.
If we can find lesions that extend on the outside of the prostate and and be able to say, well, this lesion actually has has cancer. And because we, um, do the registration between radiology and pathology. Um.
Um, we have very careful labels of extra static extensions we know on a basis where this, um, extra static extensions are and we can also project them onto the, uh, emery.
So here, we can see some of the results of this, uh, net, um, in, which we, we have the ground truth cancer and then we have the in white and here. And then, when we look at our models, this is the same case across multiple slices.
We can see how, uh, the model actually says that there is quite a little bit of cancer outside suggesting that, in this case has and similar we can see more examples.
And we can see our networks, um, starting to actually catch some of that information while it might not be perfectly aligned with where the is located. It's, it's still very, very close.
So we have shown in in this, uh, recent work, um, that we definitely get a better sensitivity than the radiologist when they call on the, um, but we still have to do some improvements in terms of specificity.
That is a lot lower. And then, um, what we see what the radiologists do it. Um, but nonetheless is our 1st, um, uh, attempt to do prediction.
And it is the 1st attempt to repeat prediction in automatic fashion without, um, using radiologist labels. Um, so with that, I will conclude by saying that we contributed to the field of, uh.
Radiology pathology,
bridging the cap by,
um,
essentially developing a variety of radiology,
pathology,
registration methods,
addressing the different shortcomings of,
um,
the different aspects of how we go about registering the radiology and pathology data.
Um, and then we have used these labels to train a variety of networks, um, that distinguish aggressive underlined cancers.
And also, we're starting to look towards, um, other, um, aspects of of the pipeline, which, uh, for example, the extra static extension work.
Um,
and finally,
um,
when we have developed this methods to,
um,
do Oregon segmentation,
but we are going,
um,
and our intention is to improve the diagnosis of prostate cancer,
and,
um,
we are going in that space by focusing our attention,
um,
in on ultrasound.
So, um, 1 of our 1st work on work that has is focused on detecting the prostate and ultrasound. Um, and, um, more to hopefully come in the in the future. Um, but with that, we, we have, um.
And spend a lot of, uh, energy and our attention towards distinguishing England and aggressive cancer when they coexist within the same lesion. Um, hopefully helping, um.
Neurologist to identify aggressive cancer earlier and and use that information also to, to drive local treatments. Big shout out to all my team.
And then the collaborators, without which this work would it be possible? Of course, uh, our funding agencies are are very important for us and specific. Thank here. Thanks uh, to the r37.
That was a word earlier this year on this work. And then, uh, just mentioning that, uh, we are, um, we are looking at, uh, hiring. Um, so with that, I'll stop here. And, uh, thank you all for your attention.
And I am ready to answer all these questions that are in the chat.
Um, but I would I like a lot of discussion, so maybe, um, the, the folks that have asked the question, maybe they can unmute themselves. And, um, we ask the question.
Thank you doctor sue. I think Andre federal had the 1st question unanswered.
Sorry, I think Kevin is referring to the question I put in the chat, I can read it if that's easier.
So,
my question is,
how much extra time does the use of specimen mold at the standard of care clinical procedure for specimen processing and 2nd,
can you talk a bit about the logistics of utilizing mold routinely for every prospective specimen in large hospital which Stanford is.
Um, so, um, to be honest, I don't know all the details related how much extra time is needed for the 3 D printed um, modes.
Um, they do come from, from, uh, Dr signs off and Richard Richard fan. Um, which is a clinical assistant professor in neurology takes care of that. Um, so.
I, um, I don't know all the details about how complicated this becomes from routine, uh, clinical care, but it's not extremely, uh, complicated because it became routine standard of care.
Um, that's that's all I can say.
Okay, sorry if I don't have more details, but, uh, we can connect and we can collect offline and I can put you in in, in touch with the right people that can answer those specific ones.
Okay, thanks. The next question is from Jay.
Would you like to ask a question? I can read it. Um, so thanks very much for your presentation. I just wanted to.
So, in terms of the question I had, I think that highlight, like, if you're looking at, for example, the CDC statistics, in terms of prostate cancer instance, mortality, they really are clear differences when you talk about different by the race and ethnicity.
And so, when you're building these models and training them, were you taking these patient populations in consideration, particularly your central lab is focused on kind of the personalized aspect of of medicine. So just want to get a sense of kind of how generalizable this potentially is.
That's a very quick question and the direction that, in which we have been interested in, um, um, thinking a little more and see if our networks are, um, generalizable or not. Um.
The, um, as we are growing our cohorts, we're gonna have more, uh, uh, distribution, or we are gonna have more, uh, folks with different race and ethnicity.
Unfortunately, we have a relatively small, uh, cohort of patients with, uh.
Uh, with with different, the distribution is is kind of homogeneous. Let's put it this way. So it's a bit hard for us to do very that, uh, analysis with the data that we currently have.
The preliminary studies that we have performed, um, have shown, though, that we don't have, we don't see a difference, uh, in performance, based on the race and the city.
Um, but again, with the star that we have a very small number of black, African American, uh, patient in our cohort. So the lack of difference could also be, um, because we have a small end on that side.
Um, but at least as much as we have seen in the preliminary studies that we have done, um.
We don't see a difference in a ai performance. That's all all we can say. Um, for now.
But a very good question, and something that we are interested in looking, uh, further into.
Thank you the next question is from Dr macro.
From would you like to ask you a question?
Hi, uh, thank you for great presentation grit work. Um, um.
Actually, uh, I was curious to, uh, to know from you, if.
Um, they're any work, uh, on, uh.
Like, trying to match, uh, uh, pathology images, for example, with immune cell infiltration how how it looks.
Um, like, matching this with the geology, look of a tumor.
I mean, can we actually train, uh, also, um, or match, uh, re, geographic images with pathology images with regard to the immune cell infiltration?
Um,
so short answer,
we have not performed that study,
um,
and,
um,
but would love to do that,
um,
as long as we can get that information from the pathology,
um,
you know,
ideally,
automatic fashion,
but even if a human goes in annotate,
what where the infiltrating tumor,
um,
or infiltrating cells are,
um,
I think we could it.
It, it definitely is a very interesting question. And, uh, 1 that, um, I will be more than happy to look at. Um, so maybe we can connect offline. And and then we can see how we can make something like that happen. But as of right now, our team has not looked in that aspect.
Yeah, that would be a great, uh, advances actually would help with that now immune therapy. And, uh.
Um, yeah, I would love to communicate with you. Thank you.
Thank you thank you. The next comment is from Dr.
I think he's still online. So can you hear me? Yes, yes. Yep. So excellent. The congratulations for the excellent work. Uh, I have 1, quick question.
Um, are you, uh, did you ever deploy any of these into the clinical box environment? You know um.
W, W, W, what is your experience with that? Thank you very, very good question. We are in process of getting some of the, our approaches into our pack system specifically.
Um, our earliest, um, attempts are, are the ones using the process segmentation. Um.
And, um, that, um, is being.
The process segmentations on M. R. I used during the biopsy procedure. Um, so, and currently at Stanford that those are, uh, manually performed by group of, um, uh, technologists, uh, from the lab.
So we are looking at seeing if we can, um, improve their time.
Um, that they required in order to signal the prostate, and also, ultimately they get approved and modified by our, um, urologists before they, they perform the biopsy.
Um, so, also, um, if the segmentations are accurate, then, then the radiologist has less work to do. So, I think all in all, um, it would be very beneficial time wise and practically, um, to address your point.
Yes, we are looking at implementing this, you know, you know, PACs environment. I mean, uh, you know, in in more of this research packs, if you want, um, but it will go kind of closer to clinical care.
Um, and then once we are successful in our integration with that, we are looking for the other approaches as well.
Thank you and Karen, if I can ask 1 more quick question. Yes, please.
So, and, uh, you showed some results from your algorithms and mainly for, you know, like object detection algorithms that's similar to coefficient is, you know, like, kind of with an acceptable range.
But still, you know, like, not like very high. This is also what we observed on the other hand, the detection performance they use these are quite promising. Do you believe in that?
Those agencies are much more realistic performance metrics that we should consider when we do all of these algorithms instead of just going with dice similarities.
I'm I'm laughing, because I think that's a very, very good question and 1 that I'm always the reviewers are very concerned about. Right um, I think that's that's my efficient is very well known to be small for small objects.
It's just kind of a little bit how it works and many people have spent quite a little bit of time showing and proving that. That's the case. Um, I think from the performance.
Um, so the way I look at this, when, when I talked to our trainees about, um, what's best thing to do? Depends who you are asking if you're asking a clinician I think the condition is more interested in.
Obviously you can confirm that. Um, as a radiologist, they are interested in knowing whether that vision was detected.
Um, and, um, whether you will be able to guide that needle towards that region and finally aggressive cancer so detection.
So, you see, it's more important than dice coefficient whether you found the entire region.
Now,
that's,
um,
I think from a clinical perspective,
from a technical perspective,
for folks like me,
I think that's indeed an important aspect,
but I also want to know how much of the real lesion I was able to detect enhance the dice coefficient while understanding that low values are not sounding great.
That doesn't mean the detection is not correct. Um, so I think I think it's a, it's a mix of multiple things and that's the reason why I don't think we should look on the dice coefficient. I don't think we should only look at it.
You see, I think we should combine and look at all of them and the best method should should win across the multiple measures that we are looking at.
I think, um, the very recent challenge, which is the prospect API challenge is not even looking at dice coefficient. I think they are looking at the distance between the center of the predictions, compared to the center of the ground truth.
So, in other words, a lot of people have been thinking about this aspects. Um, and I think they all have reasons and they all should be considered consumer painlessly.
Thank you so much and I think I want to take and organizers for this. Excellent. Well, thank you again.
Thank you thank you. Thank you so much. Dr so, this was an excellent and engaging presentation. Um, I think that your our 37 beginnings 1st, here, there's a lot more to come in this area and we look forward to future developments from your group.
Um, we're almost at the end of the hour, I know some people have to leave. So, for those who can stick around to get the updates please please do for others.
Thank you for joining the decision and I, again on behalf of NCI, my colleagues agency, I'd like to thank doctor mirror Bill Russo for her excellent presentation.
With that will be Thank you all. Of course. Uh, with that will go through updates from.
Thank you Kevin, I just want to give a brief update, uh, about the developments at the imaging data comments.
Um, the imaging data comments now, office office hours, and I'm just going to put everything that's here on the slide I'm going to put in the chat.
So,
if you're interested to learn more about imaging data comments,
how it works,
the data,
you can find there,
please,
uh,
join the imaging data comments team every Tuesday on Wednesday at the set times,
uh,
using this link to,
uh,
the Google meet.
Meeting I also want to let,
you know that the imaging data comments has pushed out a new release and with the May release,
they introduced the concept of differential licensing for collections as you may be aware.
Uh, due to security concerns do to privacy concerns. Some of the collections in particular had net collections that have been previously, uh, been public collections have become limited access collections.
In order to be able to still publish the pathology portions of these collections. Different licenses needed to be applied.
And the imaging data comments has addressed this in this previous release.
So, uh, the pathology collections are available through the imaging data comments.
With the May release they also, uh.
Because of this change, they are able to share the full digital pathology collections for the CP tech and the programs. But in addition to this, they have also introduced to new radiology collections.
And they have added to the analysis results,
uh,
section of the portal,
and have upgraded and fine tuned the viewers that are used to display and visualize the radiology and the pathology images.
And we have.
Andre and David Clooney and other members of the team on the call. So if you have questions feel free to check them and, uh, ask your questions.
Okay, and I'll stop sharing and hand the mic over to Justin.
Thanks, um, share my screen.
So, I just put it that's a nice idea. I just put the link to and the chat as well if anyone wants to use it to check out what we have on hand. Um, and, uh, I just wanted to piggyback on andrea's point.
We have, uh, 13 or sorry 12 different prostate related collections. So, if anyone's interested in getting their hands on some prostate data, uh, both, radiology and pathology data is available. Um, in terms of our updates for the month.
Um, we have 2 new data sets that came out in, May that we wanted to advertise both our pathology datasets. The 1st 1 is, uh, her 2 and treatment response slides with tumor annotations.
Um, this is 273 patients that were annotated with regions of interest associated to her to positive and negative tumor area with the generated by a senior pathologist.
Um, and the data set is basically designed to help facilitate development of her 2 diagnosis and response applications.
Um, the 2nd data set is, uh, H. E.
And and, you know, his, the chemical stained images of 209 cases of diffuse large lymphoma linked with site of genetic features and clinical outcomes.
Um, so this 1 includes a tissue micro array data as well as, um, the decide slides and, uh, also a company with survival data, follow up status and a bunch of other clinical information.
Then, uh, the last thing I wanted to mention, um, we put out a call this month and are generally always interested in hearing about what you're doing with the data on.
Uh, and that could be publishing the data in journals, or, um, if you know, if you build software or commercialize a product or anything like that.
Um, we're always interested to hear about it so that we can help spread the word about how is being used and also, um, you know, use this as a further justification to continue, uh, funding this public resource for you all um.
In response to that call, we wanted to highlight someone that that made us aware of a recent paper that they published, um, using or digging through.
The 13 had net cancer collections that we have available. Um, they were able to find 4 data sets on that match the criteria that they were looking for for their study.
Um, and were able to use that to test the ability of deep learning models, um, for HPV status testing from routinely acquired diagnostic.
So, if you're interested in head neck cancer and deep learning, I encourage you to check out that paper and definitely either ping us directly on any of the social media channels.
Whether it's Facebook, LinkedIn or Twitter, or if you want to just send us an old fashion email to the helpdesk and let us know what you've been doing with the data we'd love to hear about it.
Thanks, thank you, Justin. Uh, again I'd like to. Thank Dr and.
All the attendees, uh, this brings this session of.
The community webinar.
Museum and Informatics company webinar to.
Conclusion I'd like to wish you all safe and enjoyable summer and we'll see back in fall.
We'll be back in October. Thanks everyone. And thank you. Lee.
Thank you for having me.
By course, thank you.