The 2nd, 1 on a platform for, uh, curating, uh, and harmonizing models for deep learning. Uh.
Based on imaging data for full disclosure, I should say that I need.
From the NCI side both projects. So, um.
And was involved in inviting the presenters so the 1st presentation will be on the imaging data comments and update by Dr Andre who's the copier and technical lead for the imaging data comments?
And his presentation is about 20 minutes +5 minutes for questions and answer. Thank you that. I turn it over to Andre.
All right, thank you very much. And, uh.
I have to admit, I thought this will be a 30 minute presentation, so I will try to fill it in.
Okay, can can you see my screen? Mm. Hmm.
Almost okay. How about now? Yes, yes. Okay. All right. Thank you. Very much for inviting me to present. My name is Henry shoulder.
I'm a researcher at Brigham women's hospital, and I, my of this project we did talk about, you mentioned the, the comments on this venue and it's my pleasure to.
Uh, give you an update of what happened in the past since the past presentation, I guess more or less. Um.
So, I want to start with acknowledging the leads and technical, uh, team of the.
Uh, these are the people who get, uh, who have, uh, uh, or behind the progress. Uh, we have we've been making in this project and I want to thank them for their contribution. So.
I don't assume that all of, you know, what you mentioned data comments here. So I will what I will do is kind of, uh, inter, leave a discussion of what is with the new updates that happened since the previous presentations.
So, overall, uh, you mentioned data comments is a cloud based repository.
Of publicly available cancer, emission data Co located with the analysis and exploration tools and resources. We are cloud based, which means all of our data is available. Right now within Google cloud.
We're also working to get our data into, uh, Amazon, uh, cloud.
Uh, our data covers all the entire range of cancer imaging data, so it's not just radiology, but also, digital pathology, multichannel, efflorescence, data and, uh, any other data,
which is about cancer imaging is a fair game for you mentioned data comments. Uh, very importantly. Uh, it's a, is not just about, uh, archiving the data and making the data available for download.
Uh, our strength is really in qualification data.
The cloud, uh, tools, cloud, uh, tools that are generic tools, provided by Google, but also tools that we develop and maintain, uh, to help. You make better use of the data. And, uh, uh, uh, also it's important to emphasize that image.
Data comments is just, uh, 1 of the repositories within the broader cancer research data comments infrastructure, which has been established. Uh, uh.
As the initiative of the cancer moonshot effort, I, it includes both, uh, repository of other types of cancer data, uh, but also, uh, resources to support access to this data.
Uh, compute on this data submission and such, but in this presentation, I will focus on, uh.
We started in the summer 2019. uh, we, uh, very early president with, uh, Louise in the pilot release of, uh, and at this point, uh, we have over 40, terabytes of data.
Uh, in radiology and pathology collections, initially, uh, our, uh, content was populated from the collections released by the cancer image in archive and I wanted to, uh,
definitely acknowledge a huge contribution to this community from and acknowledge this is a very valuable partner.
Uh, we are in a very close communication with regularly, and we are working with a productively together, but at this point, we have a data from other repositories from other.
Initiatives such as and more recently. H10 uh, here's another view on the timeline of our release. You can see we start with a small number of collections.
Uh, we had a huge jump here, uh, because, uh, with the production release of, we included the collection and then we had, uh, we are trying to get into a cadence where every month we have a new release and, uh,
for some of those releases we had those. those
Kind of, uh, highlights, uh, here, we start our collaboration with the Google public data set program. I will talk a bit more about why this is important later ingestion of pathology data pathology.
Ingestion of data, and in the latest, the 11 release, we, uh, included clinical data in our, uh, offering.
Uh, as you can see from this slide at this point, we have a broad range of, uh, manufacturers, uh, uh, body parts covered, uh, different types of cancer.
Uh, and very importantly, we have quite a lot of collections that are accompanied by annotations those, uh, sometime are plain orientations, uh, volumetric annotations but, uh,
those are the datasets that are more ready for the user to train deploying algorithms. Uh, I will share the slides after the tone.
You can follow all of those links to.
Examine this content on your own time.
So, uh, I will talk a little bit about in a bit more detail about the, uh, new data sets that, uh, kind of I want to highlight that are available in.
So 1st is definitely the national landscape and trial is by far the largest collection, uh, that we have in, uh, this collection, uh, used to be a limited Texas.
Uh, but, uh, it went public I, Boston and, uh, uh, we have images a subset of clinical data. Uh, that is quite variable.
And also we have digital pathology images that have been harmonized into a document, a presentation and, uh, are available, uh, for query and visualization in the same minor as images.
Our approach to managing, uh, digital pathology content is through it's a harmonization into dual personality representation.
Uh, the, uh, the reason we are doing this is because we want to provide uniform interface to search of the metadata. We want to be able to leverage, uh, Google backend.
Instead of maintaining our own servers that, uh, provide, uh, this data, uh, we want to be able to leverage.
There are 2 users that, uh, work with data. We want to be able to harmonize data from different sources and different kinds of data. We want to have common data model, both for images, uh, radiology, pathology and more. Uh, but also annotations.
Uh, so, at this point, we have these 3 groups of, uh, collections, uh, that, uh, are available in, uh, dual personality. And all of those pathology images are viewable, uh, through the, uh.
Uh, interface, similar to how we provide utilization of radiology, data, human human Atlas network was definitely a big highlight, uh, for us in a version.
Then I believe, uh, it took us a quite significant effort to, uh, work out the procedures and, uh, work for the productive live with the Asian community and the to ingest, uh, this data. Uh, and, um.
At this point, this data also is available in, uh, uh, representation and you can, uh, have a interactive visualization of the, uh, this multichannel data using.
Uh, this data contains a significant number of channels. I think, uh, most of those data sets have more than 10 channels and with the interface. We provide, uh, you, uh, have the freedom to choose, which channels you want to see and how you want to present them.
But also we have preset configurations that, uh, select a predefined set of channels, uh, that, uh, are meaningful in, uh, in the opinion of the experts working with us on this data.
So here's an example of interactive visualization of this data. Um.
Uh, you can see here that there are 34, uh, channels that are mixed in this lecture. You can see how you can choose 1 of those presentation states that, uh, uh, have the configurations of preset channels.
And then you can interact with the individual channels to adjust their transparency and color.
Be a highlight of 11 was the list of clinical data for those of you who, uh, worked with.
You probably know that, uh, this, uh, data has been, uh, available, only in the per collection format, uh, in Excel spreadsheets. Sometime, uh, those spreadsheets are accompanied by dictionaries.
Sometimes they have single sheets sometimes they have multiple sheets sometimes a CSV file. Sometimes it's in a zip folder. Uh, sometimes in PDF. Uh, so, uh, up until recently.
It was, uh, very difficult, I would say to, uh, even grasp the variety of data that, uh, is available in terms of clinical data, uh, not talking about being able to search that.
So, uh, what we did in, we didn't, uh, try, uh, to harmonize all of the data, but we, I did harmonize the access interface to this data instead of, uh, looking for specific attachments and individual collections.
Uh, downloading them parsing them loading them into some, uh.
Uh, mechanism that you can use to search, we do this for you. Uh, we maintain, uh, those tables as the, um.
As the big query tables that are available through interface, and we maintain the dictionary extracted from those tables. So the consortium both the dictionary and the content of the clinical data that the company is, um.
Uh, uh, data sets that we have in I want to emphasize that we do not have clinical data for all of the collections. We have not yet ingested clinical data. All of it that is available in.
Uh, but it's definitely for us, it was a big step forward, uh, in terms of making this data more accessible and we are looking forward to your feedback in how useful you think this is and how we can improve, uh, availability of this data.
Here's the dashboard.
This dashboard is interactive and available to you this URL. Maybe she will be able to access, uh, when I shared the slides, you can see a summary that at this point 78 collections, we have a are accompanied by a clinical data.
You can see, uh, collections ranked by the number of dictionary attributes that are available for that collection uh, here. here
And this this plot is, uh, uh.
Best appreciate it in the interactive mode because when you open this patient, a mouse over individual squares, you will see the number of collections where this specific attribute, uh, appears. And, uh, I think, uh, 1.
Uh, encouraging, uh, I think, uh.
Uh, take away, um, thought from, from looking at this point is that there is a lot of opportunities to harmonize this data and apply, perhaps ai or other tools.
Uh, to make it more uniform, uh, what, uh, our goal has been is to make it easier for you to access data as it is right now but we are very hopeful that, uh, this is a great opportunity to.
Terminate this data by the domain experts in the future.
So, as I said before, we are a repository, but, uh, I think most important about is not about what kind of data it is important. But I would say, it's not the most important thing, uh, what kind of data you can find in ATC.
But also what you can do with, uh, with, uh, with the data.
And, uh, our goal has been to reuse the effort. You need to put into doing something meaningful with the data. And we have a number of features that you can.
That are in our opinion, are quite powerful that you can, uh, achieve without having to download data. To your computer, and without having to install anything on your computer.
So 1st, and foremost, we want you to be able to search and explore data available in and we have different mechanisms to support.
That task, uh, if you go to our portal, you will, uh, find the, uh.
Uh, the user interface for exploration, uh, or for the data content and, uh, this interface.
It allows you to subset our data based on a very small set of metadata attributes from those that we have behind the scenes. It's definitely the best interface to quickly browse the data to visualize it. And it doesn't require any technical knowledge at the same time.
We also provide very, uh, low level, technical interface where, with SQL, uh, you can use any of the diagram attributes to define very precisely the cohorts that you use in your research.
Uh, you can have very deep understanding.
About the variability, uh, across acquisition parameters, manufacturers, availability of segmentations resolutions, uh, tissue processing techniques uh, you name it.
Uh, so all of that is, uh, available in, uh, uh, SQL interface.
And, uh, more importantly, our data is version. So, uh, when you define a cohort using our interface, whether it's, uh, portal or, uh, you should be sure that you can come back to that cohort, uh, after our subsequent releases.
Uh, and, uh, hopefully this will help you establish reproducible, uh, workforce and share your findings with, uh, with the community.
Uh, what is new about our search and explore capabilities? Uh, we, um, uh.
Edit the analysis results as a 1st, class citizens in the interface of, uh, our portal.
Uh, we think it's important because some collections have a number of analysis results and sometime intermixing and those results from different contributors is counterproductive. Uh, we also want to be able to give full credit to the contributors of those results.
Uh, we believe that, uh.
Annotation segmentations, X, presentations, algorithm limitations are as important as the images themselves, because those citations make images more actionable and more accessible, uh, to.
To those users who don't have the domain expertise to conduct those presentations. So I encourage you to check out this feature in our interface.
We added permanent links that allow you to link to, uh, filter definitions and individual collections. So this hopefully can, uh, make your interaction with our portal, uh, more satisfactory.
Uh, we, uh, did a lot of work on developing tutorials and documentation on how to use.
Uh, we understand that, uh, uh, the amount of metadata available in is really overwhelming and it's, uh, there is.
Learning curve, understanding how to use the. So we have, uh, uh, several notebooks that exemplify, uh, how to get started and hopefully, uh, motivate you to learn more and ask questions.
And these are some of the examples of queries that we developed.
Uh, so, um, which hopefully can help you in, uh, developing your analysis algorithms and understanding what is available in.
I see so next what what we want you to be able to achieve is a visualization of images and derived data.
Uh, I mentioned this before, uh, we have the radiology and facility viewers, so on this screen, you can see examples of visualization of ancient pathology. Multichannel.
Efflorescence you mentioned in radiology data uh, we are committed to support a support of the, uh, image derived data. You will see.
That for radiology data, we support segmentation objects are destruct. Uh, we do have support for plant orientations, uh, that we added to review, or we don't have any collections yet that can demonstrate this capability, uh, for the H. E.
and, uh, efflorescence emissions. We don't have any limitations right now. Uh, but we are actually working in, um.
Implementations for some of those collections, and hopefully next time we will be able to report on that topic. Uh, so, uh, our work is leveraging, uh, open source or and slim viewer.
And, uh, I always emphasize that every single feature we add to the viewers that are important for every bug. We fix we commit back to the, uh, public repositories of those users.
So, uh, if you use those open source tools in your work, you will be able to leverage the, uh.
From, uh, from our project.
Uh, this is the feature that, uh, maybe I have too much text on this slide, but, uh, um.
I think it's very important.
Uh, we have been repeatedly asked by, uh, our users, uh, how can I visualize the analysis results that I created using my tools and, uh, for obvious reasons we cannot take another results from anyone.
Like, you have to get the quality, you have to establish the value, uh, of those, uh, and those results in order for them to get into. But it's a very valid point that, uh, for a researcher, um, performance analysis they wanted to.
Be able to view those in the same way as they see images in image and data comments. So, the, uh, very excited. The feature we added to, uh, version 2 is the, uh, support of 2, different servers.
Uh, what this means is that, uh, we can configure the sorry about that.
Uh, we can configure the viewer to fetch images from static repository, but then, uh, fetch the segmentations or other type of presentation from the data store, which is controlled and owned by the user.
This makes the process much simpler. It allows user not to spend money on hosting and duplicating imaging data and really focus on annotation and, uh, understanding the performance of of their tool.
So, Here's a, just a.
Full diagram of how this looks like we have with the store, which is static not writable by the, by the, by the user. A subset of this. Um.
Uh, of the relevant portion of this data is done through SQL analysis can be done, uh, in the cloud uh, either in collab or in the Python notebook and then, analysis results can be converted into a data representation, uh,
populated into the store, uh, in Google. And then you can have the, uh, or if you were talking to both these, they can store with the results and they can store with the images to provide this combined visualization experience.
What I want to say in this slide is that, uh, we, uh, once you define the subset that is of interest to you, you can freeze that subset, uh, by defining the query. Um, or you can also save the cohort in the portal.
You can always come back to this cohort and, uh, HC reproducibility of, uh.
Uh, input, uh, for your algorithm in this way um.
Ultimately, all of those steps related to visualization, data selection are important in order to, uh, support the development of new tools, evolution of generalizability of existing tools and, uh,
the workflow we demonstrate in our notebooks and something that a human will actually talk today in, uh, his presentation how you can combine the capabilities of the open source tools.
We use commercial backend, uh, data repository and, um.
Uh, Google call up, uh, to have end to end reproducible workflows that, uh, allow anyone, uh, to.
To repeat the finance, uh, punched in the paper, for example.
Uh, so, uh, we, uh, released a number of new, uh, uh, call up notebooks and we have the, uh, we are continuously working on improving the.
Uh, availability of those, uh, tutorials.
And here are some of the examples of the notebooks that we released in the past, uh, 6 to 12 months, all of those notebooks are available publicly in this repository under image and data comments will help organization.
Uh, we've been quite active, uh, in our outreach activities.
Uh, we presented a number of, uh, Webinars and, uh, posters and also we had an in person, uh, workshop at the annual meeting, uh, just, uh, last month.
Uh, we will be at, uh, in person, and we will have a tutorial at the deep learning lab.
And the poster, so if, uh, for those of you who attend, uh, please stop by our booth and tutorial, and would be great to have interaction with you in person.
Uh, so finally, what you get by using, you also have the opportunity to identify.
Uh, I mentioned non emerging data for your image in cohorts, uh, cancer data aggregator.
Is this a component of the broader infrastructure that allows you to query and identify such a mentioned data sets and there has been a lot of progress in developing cancer data aggregator.
Uh, and, uh, that group is also looking for users and feedback.
Uh, but, uh, clearly there are a lot of unexplored opportunities, uh, in combined journalists and proteomics data with images as.
Perhaps can be seen by the overlap of those, uh, repository in the diagram on the right.
So, if, uh, after hearing all of what I said, in the past 20 minutes, you are not convinced that you are interested in using cloud or exploring those capabilities and you just want to download the data. That's completely fine. Uh, we will not judge you.
You don't need to ask permission, you can go to the documentation and we have hopefully clear instructions how you can just select your cohort and, uh, download it.
And, uh, use, uh, other other means for analyzing.
Uh, the data, uh, there are no, uh, out of cloud egress fees, and this is quarterly, uh, Google, public headset program and other very valuable partner of image data comments. Uh, so, uh.
All of this data is available for the analysis, either in the cloud, or on your computer, and you don't need to pay for this. And neither you have to pay for any of those capabilities here that are powered by, uh, the cloud.
But, but all of the costs of maintenance maintenance and compute are, uh, covered through the, uh, uh, uh, free tier uh.
Uh, through the 3 tier of those, uh.
Platforms, so, if you want to, uh, if you, uh, after exploring the free here, you are interested to do more there are a lot more opportunities what you can do with the cloud.
Uh, you can configure your virtual machines to have very powerful.
Gp use as much distant memory as you as you want virtually, uh, these are some of the capabilities that become available if you are interested.
To get a free credits that we, uh, uh, awards to anyone interested in using for cancer research. Uh, or if you are interested in setting up a build an account.
So, um, the cost that you will need to, uh, pay for, uh, doing this, uh, vary and, uh.
Uh, we are very happy to, uh, work with you to guide you how to accomplish those individual, uh, tasks, uh, our plans for the, for the, uh, next 6 months or so.
Uh, we are considering, uh, working on implementation of moderate.
Uh, access in order to provide access to restrict the Texas collections in.
I D. C. that's a necessary step that we need to do to comply with federal regulations. Uh, we started to work on replication data in Amazon. Uh, and we are hopeful that the next 6 months, we'll make a good progress in this direction.
We continue working with, but also with other, uh, sources of data to, uh, increase the range of the data offered by.
And, uh, I mentioned this before, but focused on is quite important.
We are actively working on creating annotations for radiology collections that don't have any limitations right now, and also, uh, ingesting pathology notations for collections that are available, uh, currently from USA in a non standard format.
So, if you are interested to, uh, try check out our portal.
Uh, um, we have, uh, uh, short paper describing, uh, main ideas behind that, uh, we have an application how you can get free cloud credits if you need those, but you can do a lot without them.
Uh, we also have a weekly office hours on Tuesday and Wednesday.
You can come with any questions. There are no wrong questions that you can ask about or or Google cloud. So it would be great to have discussion a freeform discussion with you at this, this venue.
Uh, and of course, we have the forum, uh, where, uh, we are building the community where both, uh, supporters, engineers, developers of the, uh, platform kind of work together with the users and users help each other, uh, in, uh.
Learning how to use cloud for cancer imaging the research. So that's all. I had. Sorry I ran 5 minutes over. Um, but hopefully.
It's okay, thank you so much Andre. Um.
I don't see any questions in chat, but if there are a couple of questions.
Okay 1, this is big Vikram? Yes. Yes. Go ahead. Yeah, so Andrew mentioned the, uh.
You know, the, the challenges with clinical data.
And that made it made me think of another program where they have the opposite problem, which is all of us.
Now, they have clinical data and is quite user friendly and extensive on more than half a 1Million subjects. But they have 0 images last time. I looked so I was wondering whether you or anybody on the call is involved with all of us do they have a plan for the images?
I mean, is there something they can learn from them or you can learn from the or vice versa?
That's probably a question for I know he's been working with that project.
I I don't have a quick answer for you. Uh, I haven't really worked with all of us, but I have a call actually with a referral from all of us tomorrow morning. So I will, um, I'll bring this up. I mean, just just by way of background.
When all of us, but launched several years ago, I remember talking to Francis Collins, and you said, because all of us is, you know, disease agnostic. Eventually, his vision was that everything should kind of fold into that.
So, it seems like that's why in the sky still, unless, you know, something more.
Okay, I'll try to find out more end and follow up.
But for a moment, I don't have any more information. Thank you.
Any other questions okay. Uh, since we are kind of running a bit tight, I would like to, uh, thank you, Andre. I'd like to.
Welcome it's presented to Dr Hertz from, uh, bringing women's hospital and another is also from the same institution. I forgot to mention. Uh, he will present on cloud based platform for dissemination of deep learning models. Thank you.
Sorry um, I hope you can all hear me well, um, and can also see my slides. Is that correct?
Yes, all right. Perfect. All right so, um.
In this doc, that's about, um, uh, 50 minutes long.
Uh, I want to talk about, um, um, basically what's fits very well with with the previous talk from Andrew, about how, uh, or we can also take leverage, you know, these kinds of developments in CO cloud based platforms for, um, uh,
in the field of AI. So, 1st, I will, um, describe you a little bit to give you a little bit of background about the problem that we're facing. And then I will go into details about.
The platform, that's, um, that we're not developing and this is also recently funded by an NCI contract. So.
I think you all know that there's a lot of attention, of course, in the eye and I'm still amazed how this whole field is really fitting this Gartner hype cycle very well.
Um, you know, so this also what you saw on a lot of conferences is that everybody was talking about and what you now see, and I think that is really happening is that you're, you know, we're underway on the on the way down where, you know, start ups starting fail, um, you know,
um, a lot of people find that some medicine is not that easy to transform using, uh, using.
And so, you know, there's a lot of realism that comes in. But what I think is also a very, very important aspect of this is reproducibility and transparency. And that is something which is really hampering progress in the field.
So, um, this was actually paper. I really, um, that really opened my eyes. That was actually published 4 years ago in science, where they did an evaluation, um, uh, of 400 papers that Bob is in the field in the eye field.
And they found it only 6% of all, uh, you know, published codes. Um, uh, some 30% includes some test data, so part of it of all of it and then 54 to half of them have pseudo codes, which took me awhile to understand what that was. But that is more like, um, uh.
more like um uh
Description that was detailed enough that other people have an idea. At least what what the original investigators that.
So, um, that's really, really worrying and, of course, also, this is still a very big problem.
Um, uh, today, right, a lot of people are not sharing code and not sharing a lot of details and it's a really big problem here, because also, very often the editorial staff and the, and the, and the journals they do not know what exactly to ask for.
So, I'm gonna highlight this paper that was actually published by a group from Google that gets a lot of attention in the media, how to use an system to improve.
Breast cancer screening, and if you don't look at the vulnerability statement that they did, um, it is something which we actually really opened our eyes because, uh, this is very surprising. Right?
So they said, for example, you know, the code use, uh, is so dependent on internal tooling. Right it's very difficult to extract that goes. Right. Which is surprising for say, a Google team that they cannot extract code from their platforms, but that's something else.
And also is that they said, you know, we are, you know, sharing all the details and extra for the experiments and all of these.
Different things in the methods, which were not there. Right there was over the whole description of the API architecture, for example, the deep learning architectures. It's like, 1 paragraph long. There was almost no details in there.
So, it was very difficult to replicate this and they also for sure they're pointing to public, uh, available toolbox, uh, platforms, like, tons of flow, for example, but there was no other data share.
And this is really something that, um, opened our eyes that they actually got away with this. Right? And a lot of people, right this is only 1 example.
Other people got away with this so I think there's a, you know, there's a lot of people now that are really having this hit and run mentality. Right? Let's get faster vacation. Let's be the 1st, and then run through your next problem. Right?
And try to not, you know, spend too much time on supporting this and, and answering questions and all of these things of previous studies and I also understand how this comes. Right? Like, in science, we are, of course, a little bit.
Ah, um, science is a little bit like this, but, you know, we think that this can be done better and this is also why we wrote this article, which is also.
Site now, as a response to the, to the, to the, to the, the Google paper where, um.
A lot of things in reproducibility and transparency in, you know, uh, in platforms and and technology, which is very, very standard or very, um, uh, broadly accepted within say,
by Informatics or genomics community is not actually that's existing in the in the clinical community. So, you know, we provide to, uh, also we mentioned this and we also provided some, uh, some, um.
Some advice how, you know, how we can improve this and how it's going to be done better.
Also to highlight this is, uh, 1, actually, 1 comment that we also made in this is that, uh, which I always like to highlight is that, you know, computational reproducibility is indispensable for high quality applications. Right?
So more complex methods demands greater transparency.
These these arguments are so complex, we need to know more to be able to, uh, to, uh, to try to reproduce them, uh, on, uh, on other data sets and really investigate their, um, their.
Potential, um.
This was also a paper that actually was published recently, which, um, I think a lot of people in the field were actually they're not surprised about. But so, what they did is they, they looked in the corporate, um, a large number of corporate papers.
That were used to diagnose, which is already very interesting, but diagnosed over 19 from a chest radiographs and CD scans and they, uh, and there was a lot of these payments are being published and they were also published in a lot of high impact journals.
And what is very interesting and what they also.
Concluded is that and I quote here that none of the of the models included in this reviews are likely candidates for clinical translation, uh, simply because there were so many details about the missing.
Uh, they were, um, uh, but it will also mistakes in their experimental design. And such, so, uh, it's almost like a lot of these papers are not that very valuable. So.
What we how we approach this right? So how we, uh, approach this whole problem is that.
In progress, and medically, I will likely not happen until we managed to solve this lack of transparency and reproducibility of published research, which, of course, is a very big problem and a, very, uh, to solve.
Uh, but also, once, once we do this, we, I think we also, as a community have to, um, uh, have to, uh, provide as easy as possible processes for others to actually upload their models and to provide these models to the community.
Because if there's a lot of barriers there, nobody will do it and we will just continue how it is.
If we do, so, so if they are easy, relatively easy and straightforward processes to share models, then this, uh, this will also lead to more, uh, generalization, uh, experiments of these AI, pipelines and hopefully also progress in the field.
So, in this whole realm, uh, there's also what is really good to see now, is that there's a lot of different, uh, guidelines and checklists, which have been reported by various, uh, different groups around the world and, um, more or less. You say the same thing, right?
Like, we have to provide all of these very, very detailed, uh, uh, descriptions of the, uh, of the model that we use it.
And the data okay, so that whiteboard that you saw, that you had taken note on too yeah. Sorry if you need to align that when you can use with your map.
Go ahead. Sorry. Um, so, you know, and this is really good at this happened because, you know, these, we have a disability of these, these, these guidelines and checklists can really help, um, um, and move the field forward.
So, building on this, you know, we said, um, why not also take advantage of this, this auto movement now in the field, where a lot of data, and a lot of medicine are moving to the cloud. Right?
So why not, uh, think about a cloud based platform for the dimension dissemination of deep learning models, and we call this model model. So what model hub can give us is in this transparent and reproducible API.
Um, it's, it's it's very interesting. So, how it's usually done, is that an model.
You know, if we're lucky, you know, they're reporting the checklist and the guidelines, and the, the different various, uh, uh, uh, um, aspects that need to be listed there. They store the weights again.
Uh, if they are willing to share the weights with the community, then they store uh, hopefully debate in some repositories. The code isn't isn't gets up. If we're lucky or upon request. Um, and also when public data is used, there's a link, you know, to the rule.
If, uh, again, if we're lucky, um, the results data are often shared in a nonstandard.
Formats right simple tips or or something else. So, we, but also, what is really important is that the connection between the items and original datasets is often very unclear.
So think about, like, you know, you have an analysis of a 1000 patients, uh, for a certain reason, right? Maybe you exclude 50 of them, which 50 were actually excluded. Right?
And that is often very often missing in these, uh, in the in, in the current process. So, how we focus model hub is that.
You know, we are really moving into anti reproducible, pipelines, API, pipelines to the cloud right? So the API model is up and running and runs in the cloud for free. Right.
Uh, but also all of the other codes before, uh, for Pre processing and post processing, everything is there so that from the original data, you can go to the results and on that whole process can be, uh, inspected, expanded on shares, use for your own research.
Is completely reproducible and the nice thing is also, is that if you don't link this back to the original data.
That we can almost reproduce full, uh, we can fully reproduce, um, um, complete research studies in the cloud and provide it to the community. And that is something that, um, if you don't focus, if you don't look at medical imaging data.
So, uh, because there are some specific details there that are specific to the field, so a regular workflow in the medical fields, uh, medical imaging fields, and it starts with the design of a study right? So, we defined a research question. We choose a certain data sets.
Um, you know, then we have to ensure the data quantity and quality are sufficient of this dataset. Right. Uh, we select a certain API model and, you know, I, architecture that we don't start building up.
Then we do an analysis and then, you know, we have to think about the environment, right. To all the hardware and software dependencies. Um, all of these things we need to, you know, we need to set up and we need to maintain. Um, but also the data retrieval and Pre processing steps are very important. Right?
So we have to ensure that the input data matches the requirements, and then we can also really replicate this and that is the data processing, uh, that is happening, uh, and the application of the models. And then, of course, there, you know, a lot of this, the results, uh, you know, we have to do a lot of reporting.
To document the codes environment, you know, release derived data, analysis, pipeline, um, and also document, of course, the retrieval and Pre processing steps because they are so crucial and very, very small details matter a lot. So, the nice thing.
Now, is, is that by moving a lot of this to the cloud is that we can really take advantage also, um, all of these cloud based research, uh, platforms,
like the imaging data comments and and Google cloud and Google lab is that a lot of these things become much much more easier, right? So we can do data.
Duration on the clouds, we can actually link to s, very specific and detailed subsets of certain collections that are in the cloud and that that relationship is always contains. Then if you go to analysis, we do not need to set up environments anymore. Right? These are already predefined.
Um, we can also share, of course, uh, Pre processing pipelines and we are building, uh, also these, and we, we are handling these to the, to the community. And also, um, uh, the nice thing about for the reporting part is that we have an exact copy. So we have an exact copy of.
Fully reproducible pipeline that is also documented versions and freely accessible. Uh, so that is something that, um, uh, can really, uh, in our view. Really help to feel move forward towards transparent and reproducible AI.
Um.
So, um, the nice thing is also, is that, you know, we are providing different ways how to interact with these models and these pipelines. So does publish models, can, you know, can be simply, uh, ran in a, in a, in a web, uh, web browser, you know, with a graphical interface.
So, um, uh, to quickly have an idea how that model works. But also, people can go into Jupiter notebooks and have very detailed seated line by line the code that is available, but also we are running them as APIs. Right? So you can really.
Yeah, for these public data that can be sent to these models and you can get a response back. So that is really to tell us the different needs in the community that we see.
Um, there is a 1st, uh, collection of models, not being shared a model of the API, uh, also, um, providing the Google kool app re, fully enter entry pipelines.
Uh, more of these will be added over the next weeks, uh, more of these pipelines, and also we will also add tutorials and, uh, very detailed, uh, documentation for these, uh, to the, to the website.
All right, so I have 2 more minutes, um, I just wanted to give you an example of how this works. Right? So, you know, we have input data that we can query from the imaging data comments. And a lot of these, this is not focused on the medical imaging data.
Um, you know, we want to have this reproducible, uh, workflow. And then, um, what we are building is, you know, we have these series, which is a standard, of course, in imaging data, but most models are actually not using as inputs.
So we have to convert these into other.
Formats that could be used by these as inputs to the, to the image processing. So, the deep learning models that are working there and there, you know, there's a lot of Pre processing again happening after that. The model is ran on it and post processing.
And then we have a standardized output that we then have to convert again to a diagram and then push that back to a viewer. So there's a lot of different conversion steps and again, every conversion step um, uh, very small details matter.
And we'll, um, we'll make these pipelines, uh.
And and small adults could make these pipelines completely fail. The nice thing now, is, is that we, if they have this cool app, is that we can really look at the, the, the notebooks.
And, um, there's also done really, uh, allows us to analyze the results from various models in a transparent and reproducible way. So we can really benchmark models against each other. Right? So the moment that more more models are included into model hub, we can easily run them.
Uh, on all of these collections that are so our W, the platform also becomes like.
Accelerator of science, uh, for, um, for in cancer imaging. Um, and then also, of course, we can yeah, so we can see how they generalize on seeing data. Um, and, um, we also want to, um.
We want to facilitate access to, but also other platforms that potentially could benefit from this. And then this allows us to do these benchmarking studies. So I won't go into detail about this is, but just to give you an idea. The nice thing is it's very quickly.
We can run all of these models on these standardized because the data standardized the platform is standardized. So we can we do have a comparison of performance of models very easily.
All right, so ex, exactly 50 minutes. Uh, I want to thank you for your attention and please let me know if you have any questions.
Great. Thanks. So much. You go great presentation.
Hi, uh, this is set for right now from NCI.
Yes, go ahead. Please. Yeah. Yeah. So I have 2 questions 1, is that what's the business model to make this sustainable? And the 2nd 1 is that.
Don't you have so much overlap already with the existing ones like hugging phase and others?
I mean, they, they, they, they maintain like, 75,000 ai models.
And, uh, they're reasonably priced, et cetera, so uh, huh. What's the attraction?
Yeah, so so about the, the business model, so it's a research platform, right? So, everything that we do fully is fully open, uh, open source and reproducible. So, um.
About the other platforms that's a really good question, because a lot of platform actually tried to address this. Right, and, you know, and video has hops and, uh, there, you know, there's Docker hubs and there's, uh, also Google has a certain hub and there's also companies in this space.
The thing though, is, is that, for for a person to disseminate their model into a standardized, uh, model into standardized workflow with existing data like, eh, that is not existing,
at least not existing in the cancer imaging field, or in the radiology.
Fields, which is a huge field, uh, that, uh, benefits greatly from these developments. So, um, the, the, and also be an academic resource. Right?
So, we are, um, main main goal is for other investigators to share as easy as possible, their technologies to the, to the, to the community. So, 1, that is up and running.
Of course, also, we will interact with all of these other, uh, uh, platforms as well. But a lot of these are not standardized, so, this means that if, for example.
We want to take the on video hub, um, these doctors that are running there it is, uh, it's required to work from our side to actually get them running into our platform, because most of them they use a different, uh, um, uh, in different interface. Right?
So, they call the same thing, uh, differently. Uh, so there's a lot of things that are not, um, harmonize and standardize and that's something that we also want to focus on.
Thank you other questions for Dr it does have a great presentation and is exactly direction. We need to go, uh, 1 more technical question.
You just alluded to this in media platforms all things. So how deep does the standardization girls? So, software, dependencies, libraries, even a computing architecture, how much does it actually have a real effect on the outcomes?
And how much do you see the need and the opportunities they actually standardized.
Fish.
Yeah, no, thank you. That's also very good question, so we started this with this ideas, like, you know, we need to make these containers. Right? So these containerized models as a useful for the community as possible.
So, we wanted to force contributors to, uh, put everything in specific modules. Right? So, the Pre processing, you know, the, the model reference, the post processing. Each of these should be done, be in different steps. We noticed that, uh, however, that this made it, actually, very difficult.
So, if you, uh, very famous models now in the field.
And a new nets, um, it's very difficult to actually fit them into this, uh, to this format. So, what we set is that we wanna, um, now be, um, um, almost we can even treat it as a black box.
So, we just want to make sure that the interface is harmonize and standardize, so that we are, um, we're very pragmatic here, right? Like, we're not gonna, you know, spent years on defining the best optimal standard. 1st, right.
We will, you know, we, we see what the field is doing and we, we will follow the field on this. We tried to harmonize this.
Across the containers that we curated and then upload, and we provide to the community so that, you know, bring them or at least, you know, be calling these different sequences, the same name. But the segmentation is also called the same name that we are.
You know, harmonising these things and of course, we can also learn and the imaging field a lot from the there because they are facing very similar issues.
I think we got to start somewhere, right? So, it's, it's standardizing and actually putting them in a standardized box might alleviate some of those problems. A follow up question.
If I may another big issue, especially in the medical field is explained ability of the results of outcomes.
So, you know, the old physician card, we don't trust anything that we that we don't think we can understand, even though I think the transparency, even for physician's decisions these days, it's probably not up to the standard that we require a fee.
Solutions, but never mind. How do you think, uh, sort of a very transparent, be fine grained.
Uh, uh, documentation of the platform as a whole.
Can also lead to a better understanding of individual parts and how they contribute to the answer.
Potentially enabling sort of a more human understanding of how the, a I came up with the specific results.
Yes, so I, I, I also agree with this comment a lot. So, of course, you know, a lot of people are investigating, you know, explain ability and, you know, trying to look at the different nodes and, you know, activation maps and all of the different aspects, which, you know, there's issues with them.
Uh, but, uh, you know, the fields moving forward, uh, significantly, and there's a problem that's not solved yet, but also what I, how we can contribute to this is indeed, you know, the moment that a, a team uploads their model to, uh, to model hub.
The nice thing Dan is, is that.
Uh, if this model, you know, the model runs in the standardized interface, right? Eh, and can interact with all of the data that's in the cloud but then we can also do very detailed interrogations. Right?
You know, if the, if they're the field of view is different from the city scan, for example, what will happen to the models outputs um, if they are, uh, we use a different sequence, right? Like, uh, we are using, uh, uh, small, uh, artifacts that are in the image.
The nice thing is with having this enormous resource of data that is not uploading every month more and more data is getting there the more evaluation.
We can do we're using these models and the better we can interrogate do these, these models actually work and is it or it does do they only work on the, the, the smaller training data set for example, that that the original team um, um, uh,
develop them on a test amount so the nice thing is, we hope also by doing so that we can give a lot of these models longevity, and also by demonstrating generalizability that these models will also have, um, uh, more impact in the future.
There are a couple of questions in Chad 1, is, uh, from Tracy Nolan, I think asking what are the most frequent non icon emerging formats that you see.
Um, so, um, so we.
And, uh, I think that's probably related to, uh, Andre, right for the, or for us it's for you. I guess maybe we, you know, we mainly work for the team, the team, uh, our team works mainly with medical imaging data.
So, we work with, uh, radiology database, mostly, um, as well as, uh, with pathology data.
Um, we do, um, but, of course, you know, the in cancer also, dermatology data, and even facial pictures we're working out with, that can be used.
I think the question might be, uh, Tracy please feel free to to jump in, but it might be regarding formats like nifty. Probably, you see a lot of different. Okay. Sorry. My my apologies. Yeah, so there's a lot of different, uh, formats in the medical imaging fields. Right?
So, they are, you know, this diagram is the only format we believe that is the most important, because that is what the machine outputs and that is, uh, at least a standard that everybody agrees on.
Um, but after that, uh, we, we want to be very flexible so we, um, you know, we are providing 2 nifty converters to, uh, to, to nurse to, uh, snap and I. T. K. conversions.
So, we have all of these different, uh, uh, we want to be very flexible to users. Uh, so that, um, uh, because we also understand that, uh, most or even none of the models that we've created so far, actually inputting directly.
So, they always need to go to a different format.
And again, in Python, that's a nice thing. All of these are readily available and Slicer has developed, uh, many of these as well.
Okay, uh, thanks, uh, couple of questions from George. How do you scale the compute? What about authentication and authorization.
And how do you version the data, or do you version the data that's more for Andre? I guess.
But we only have a couple of minutes if you could quickly address those, you want me to, uh, to answer those.
Sorry uh, yes please. Okay. Yeah, sure. So how do you scale the compute? Uh, right now what we've been doing, uh, we were still in compute by increasing capabilities, so the single virtual machine that we use, so, with Google call up, you start with a free here.
Uh, you can do some meaningful work on that feature, but then we scroll up, you can also migrate your workflow to a custom and there you can go crazy with, uh, and, uh, RAM.
Uh, we have not invested a lot of effort into scaling up horizontally, like Ryan parallel multiple tasks at this point we didn't have a use case for that.
Um, uh, but that's something we plan to do in the future, um, for the authentication authorization right now uh, IDC only has public data sets.
Something that we would need to address in the future when we have non public data sets and the last 1. uh, yes, we do a version of the data that you have an ABC so you can access past versions all versions are available.
When we release a new version, I think George, uh, if you join office hours for Tuesday, Wednesday, it will be great to talk more about those topics. It would be. I think if you joined.
Okay, thank you so much. Thank you. Both Andre and Hugo for great presentations. Uh, the last piece is, uh, usually we have updates from IDC and and we already got an extensive update on IDC.
So, Justin, can you give a quick update on please?
Yeah, um, so, uh, we had a few updates, uh, from September.
Um, we published this, uh, multi modality, annotated cellular carcinoma, uh, cases without.
Uh, with them without advanced imaging segmentation, um, this provides scans and organ segmentations and patient outcomes, uh, to help people study whether radio can predict tumor response. Um.
Then we also wanted to highlight the utilization of 3 data sets that were part of the project. Um, and so these were.
This is a project that was meant to combine a whole bunch of different data sources to create a large scale chest data set for 19 detection.
Uh, we also published a, an updated version of the pancreatic SEG dataset.
Um, and so that included some new, uh, radiation therapy dose files as well as some kind of, and, uh, radiation therapy structure sets to, uh, with repeat organ at risk delineations from 2 observers.
And then we also had a new collection CC ventilation as a functional imaging modality for lung cancer radiotherapy nickname for that. 1 versus pet ventilation imaging.
Um, this 1 is, it's been made available to help build a international capacity for prototyping and evaluating new ventilation imaging technologies.
Uh, then we had a.
Well, you already heard about the clinical data that was, um, released as part of the last IDC effort um.
We also wanted to remind folks that, uh.
About the impact of challenge competitions over the years, and the role in helping organizers, uh, prepare their data and identify it.
And we maintain a page that you can click on here to be able to go see a list of all the different challenge competitions that have been supported by. This is really useful. If you're looking for a super well curated dataset with labels for.
Uh, AI purposes, um, and then we also last 1, um, had a a 3rd version update for the dynamic contrast, enhance magnetic resonance images of breast cancer, uh, from Duke.
Uh, and this added some, uh, 3 dimensional breast and FTC MRI segmentation data.
Answer any questions if there are any. Okay. Great. Thank you. So much for that. Update, Justin.
So we are just on time and like to thank the presenters again and and all of the attendees.
Well, thanks for joining us today, or next presentation, or webinar would be on Monday, November 7th and we will announce that speaker.
Soon, if you are interested, or have, uh, suggestions for speakers, please feel free to reach out to the partner or myself, uh, or Dr Nordstrom and cancelled meeting program together. We manage the.
Monthly webinar, thank you all and have a great rest of your day.