I'm sorry.
Thank you. Hi, everyone. Welcome to the November edition of imaging. informatic community.
This is kept on for a honey from the center for what we Informatics and information technology.
And it's my pleasure to introduce the guest for today's topic on tonight, medical open network for. So we have with us.
Dr, Steve Albert from, who will give the introduction to Mona and the moon community Dr George corridors from kings College London.
We'll speak about when I medical deep learning capabilities and roadmap and finally Mr Brad general from NVIDIA, we'll talk about Monet and clinical workflows built on top of the system.
Without further ado, I'll turn it to the 1st speaker Stephanie work please.
Thank you very much.
So, today I'm going to be talking about and keeping just introduced the.
Medical and medical open network for a framework that we've been developing for the past several months.
They stopped actually is comprised of the to talk to that.
Comprised the freeze toxicities mentioned, they're listed here I'm going to begin with an introduction to Mona where they going to go into the technical capabilities of it.
And then talk about some of the clinical applications that from.
When thinking about making this presentation, this 1 question kept re occurring to me.
And that was why is deep learning succeed? I think we've all generally accepted the answer to this being that it's a performance of our systems.
That is making it such a great success nowadays. But what I want to argue is part of this presentation is not only talk about the capabilities of it. But also it's still in, you.
That is the openness to the acceptance of open science.
Which is driving the success of deep learning within the medical field where, uh, Steven may I interrupt you for a 2nd are you sharing slides?
Yes, we cannot see them yet.
I gave you presenter rights and I okay. I think they ended up.
With somebody else. Okay.
And you should be able to share now. Okay. So, are you seeing my application? Are you seeing the window? Yes, but they are very small. Is there a way you can increase the size of the slide?
Uh, let me see, let me try sharing a different way.
It is presentation format that I'm sharing, so I apologize since this is not, uh.
I see your view of it, it is stretched down.
Yeah, it's still quite small now. It's better, much better, much better. Is that acceptable?
I would think so. Okay. Thank you. I'm not able to see what you are seeing. And so when I do it this way.
Okay, I apologize for the difficulties.
So again, medical open network for a.
3 talks are being presented today.
And performance as well accepted as 1 of the reasons why I think planning is succeeding, but openness open science, I believe is another components.
And so, when I'm talking about performance of these deep learning systems, and that is leading to their acceptance as being so successful.
What exactly do I mean, well, that's actually being left as an exercise to the audience, right? And that everyone in the audience here as well aware of the many successes that are going along.
Coming out day, after day with the application and deep learning to medical problems.
I was just reading in LinkedIn article about this 1, small business.
That was on 7th FDA approved product that is applying to the interpretation of oxygen.
And so, the small business with 7, 8 food products in the field of deep learning medical energy.
It's just phenomenal how this performances come about.
But I believe underlying all of this again is the open science capabilities of the community.
The acceptance and embarrassment of open science that is really driving the success. And the just played a big part of that as well.
So, it's pervasive in deep learning, this notion of open science, which is really reproducible science that we're talking more.
And that involves open access publications, which is system such as our guys, and the policy of open next.
It's open access data, such as image net as well as funded systems from the United agency as such as and the imaging data comments.
And then it's open access algorithms and the sharing of those algorithms.
Which is being accomplished really to put centers so open science and deep learning. What is Mona?
While the goal is to accelerate the pace of research and development.
By providing a common software foundation, and a vibrant community for medical image. Deep line.
It began as a collaboration between them, but in case.
But really has grown much further beyond that, as they intended.
It was really meant from the start and then succeeding in becoming.
A community support a platform that's freely available.
It's based on type paid torch, but a timeline optimize for medical imaging.
And it provides not only effective algorithms solutions, but also reference implementations for best practices.
So, when it began with college and video.
The way the field look was, there was a variety of open source tools already out there for deep learning and medical energy.
Nifty, not deep neural and a variety of other tool kits pie torch itself existed for doing a processing on medical images.
And Mona, instead of going off on its own to develop an independent solution consulting with and actually integrated a variety, these toolkits together.
In order to define it, scope them to habits and mission lives.
And when defining that scope, that didn't try to do with everything that every tool can do. But I tried to identify what was the best niche that would make it an effective contribution to the medical imaging deep learning research field.
And there they identified data augmentation, neural networks, lost functions and validation rooms.
As the key components that they wanted to offer within.
And then they want to communicate with the other toolkits that were already out there, addressing things, such as data data, sampling and evaluation.
And so I had a scope that was based on expertise from developing all of these other tool kits that were out there.
But you could as well with all these other toolkits that already exist out there. Why was the 1st, 1.
Well, biomedical applications have very specific requirements.
Perhaps 1 of the most obvious of those is the fact that.
The imaging modalities involved are not the same computer vision.
And so data augmentation techniques, which are so central to deep learning training.
Those augmentation techniques, which are valid for computer vision, such as slipping rotating images and so forth.
Are really not applicable to and.
For the more the data formats, and which has the images come are quite different and different for very important reasons because they maintain the physical aspects of these images, the pixel spacing, the orientation of images and so.
So, mono from day, 1, allows you to treat a medical energy as a physical safe enough space.
All right, and this notion is pervasive throughout the to okay.
And allows the clinical application of the results, proper interpretation of the results. Actually the proper generation of results.
It's possible because of this base of knowledge, that intense.
Definitely certain that architectures are designed for and not suitable for biomedical applications, making certain that those are available.
And then things such a sample size limitation, annotation, uncertainties that impact metrics and so forth are taken into consideration because they really aren't domain specific as well.
Well, with all of these different challenges and capabilities within the system.
Again, it goes back to open science is reproducibility that offers beyond the majority of these other shortcuts.
By reproducibility, we want to make it so that 1, if 1 person generates a set of results, using online, someone else is able to reproduce those exact same results. If they have the same data.
All right, and it takes a lot of effort within a random and on certain environments of a deep learning systems and this happened, but what we're paying attention and making it up.
So, not only is this reproducibility good because of the pieces for implementation?
But also provides baseline and limitations.
Forward she researched and then they compare so you can conduct the census studies within your own lab by using.
Also demonstrates best practices for how that implement algorithms and send it out.
And what I think of all this wraps up to being, if you want to be a little bit cool with it, you can say it's a matter of standing on the shoulder of John's.
It's really some outstanding developers. It's the people who invented.
Um, variance on the unit architecture who are implementing that unit architecture and expense.
What is the inventors the, the researchers, the innovators within the field work and children to this to okay.
And we buy more of them to come along. So, how is Mona adjusting these needs? How is it addressing reproducibility?
Begins actually by having a very design.
And this is for handling not only IO of data file formats, but also the transforms that are used to process regular as and augment image data.
And the metric used during, and after training systems, all of these are designed and ways to emphasize reviews and reproducibility.
There's also a variety of revenue training paradigms within Mona.
Support for multi and multinode multi cheeky and training.
And then extensive tutorials in documentation, and not just written text, but also live tutorials in the form of Jupiter, notebooks, ignite workflows and so forth.
Whole repository within that IO framework website have been created just for hosting all these tutorials and G.
And Additionally, as I mentioned before, Casey and video realized that they wanted to be team supported.
From the start in a way that they chose to pursue this community support was to establish an advisory.
Now, this advisory board, their goal was not to just give advice, but their goal is actually to collect advice from the key.
And then follow that advice to the development and design team on.
And so, this advisory board, you know, is a wonderful representation of academic industry of deep researchers and application developers of clinicians and software developers as well.
So wide variety, but they have meant to.
Gather the feedback from the community and bring that to them.
And it's been working quite well 1 of the 1st things that this advisory board realize is that even gathering this feedback was what? That group and the previous slide was capable of doing. And so they broke into the smaller working groups.
And he's working groups address, imaging, data, diversity, reproducibility, transformations.
Federated right. Advanced Research and the adoption of the.
And leaders from coin,
and for each of these working groups,
and these working groups are meeting very frequently once a month,
once every 2 weeks to gather the feedback from the team will guide developers,
answer developers questions and really working the exchange of ideas back and forth between the developers and the community.
So that 1 is most effective and can have the greatest impact.
And it's being successful if you look at when I was released in April of 2020.
All right, so not all that long ago. Um, we just had this 0T.3 release, right? Um, so, just a couple of weeks ago.
And in that short period of time, it's gone to having over 1400 likes or thumbs up on.
We have 39 developers from variety of different groups. Well, beyond kings conditional and video began with these people from throughout the world.
And it's beginning to integrate toolkit, such as advocating a variety of other tools that and developed in other groups again is part of this community.
Integrated, rather than in place approach to software development.
And another great example of power succeeding then from the user point of view is.
We offered a boot camp right before we guy the 3 days before the conference began.
And we opened it up for applicants and weren't certain how many we were going to get. So we were a little conservative on our estimates and we were hoping to get 60 applicants.
Well, within a couple of days, we had great rate exceeded that count of 60 and ultimately we ended up with 563 applicants to attend our weekend.
The central 3 day tutorial, help before.
And so we ended up doing was accepting 60 primary attendees. They had access to cloud based European resources 1 on 1 instruction, and so forth throughout this Bootcamp.
But we also a lot of 140 people to participate as observers.
And of these 200 people that were represented in this way, for the different countries for represented.
We also then work with the student board at the guy in order to run a challenge.
And develop tutorial materials, and the student board, and the guy judge that materials, and they identify for a 2nd and 3rd place winners of it. And I think it was wonderful that a 2nd prize winner of this was from Africa.
Um, that contributed outstanding tutorial material to the system.
And so really, it's been a worldwide participation development of this team.
And so, with that in mind a little bit more detail, then on the technical capabilities, and I, I'm going to turn it over to, you.
George, you want to share your I will share my screen will probably be good.
Sure.
Okay, can you see my screen now?
Yes, perfect brilliant. Excellent. Thank you for the intro. Um, right so I'm sure I come from, um.
And but my role is not only as a senior lecturer or assistant professor.
But also, as the CEO of the London center for validates health care.
Which is a consortium of 4 universities, the 4 major London universities and 10 different hospitals.
And our group, and our university actually has a tendency to do a lot of platform development from the very beginning.
We, I've been very involved in the development of open source platforms for many years.
We release many of those, not under a very open source.
Non income licenses, and as part of that strategy, we actually have a previous work, uh, previous to pitch called nifty net.
And at some point, we realized that that was 1 of a.
Plethora of of many different tickets that were out there.
And what we decided to do was to really take this.
Cool kits talk to the other 2 kids that exist out there.
Find out what worked what didn't work what the community was actually after.
And what were the tools that we're actually missing and tried to somehow combine these altogether?
Into and you standardized copays that is of higher quality and more flexible and more compatible and more composable than what it was before.
So, with this in mind, we basically actually took, I think, 5 or 6 of them, the neuro the.
And a couple more, and we, we kind of brought them all together.
Into what is now online, and the things that we identify as critical that we're missing in parts, or actually we're missing in full in many of these packages.
Where are the 6 items that we see here, which we believe are the.
Design goals that we have for Monday, so 1st of all, we want them 1 night to be really customizable. We wanted it to be useful for.
The clinician radiologist.
Also, scientists that is not experts in coding, but wants to train their models, or maybe knows a little bit of coding, but not too much.
The junior researcher that is just starting, and they want to train certain networks and know what you're doing, but they have limited. Know how.
Are really the very advanced researcher that wants to have control over every single step of the.
And they just want to use my, and I, as a way to.
Attract building blocks and to use certain tools within 1 eye.
So, you really wanted it to be very, um, very customizable.
And this is really the idea of possibility if you can break the, um, the package down into many components.
And these components follows tender protocols as defined, for example, by pie torch community.
Then it's very easy to use and reuse certain things from an I.
Given the to the full framework headphones, but obviously the full framework has even better capabilities when.
Are intertwined with you with 1 another. So mama is also the main specialized 1 thing that we notice is that there is obviously to 3 major software packages for.
Machine learning and out there and even the deep learning set and you basically have 2 competitors, make sure.
Uh, 10 circle, and and we saw that the community was was shifting towards pie torch and.
We went through this process of trying to recognize which 1 of these.
Foundations was best, we thought about creating an abstraction on top of that, but that's actually made it more complicated. And then we just thought, okay, let's go for because this is primarily a research phase.
A software package, and this is something that is actually evolving is now also moving into the direction of deployment and real world, uh, use.
Because we start from these research phase, a software package, we thought, okay, let's go for a research platform.
But, let's just take everything that pie torch has and give it all the capabilities that are missing for anything that you want to do in a healthcare setting.
With an initial focus on medical imaging data obviously, because everything that we're doing here is based around the learning and it tools around that we wanted to be very optimized.
So, we wanted to make sure that everything that we do is a properly benchmark or properly testing.
And so that the code that we provide, that there's the effort before the code has been released to make sure that the code is a good quality.
Ethical is optimal enter that code can be used by other people without having significant bottlenecks, getting their own core base.
Another objective was, as Stephen was saying, we really wanted to make sure that we could create.
Frameworks that make sure that people can share their code.
And that could could then be used by other people in tested and compare it against other people.
And we also wanted to make sure that the cockpit itself was easy to use well documented with training material, but also having all the testing continuous integration between this deployment set up in the background.
That allows us to make sure that everything that we're delivering is a equality.
That is higher than most academically design piece of software.
So, why don't I really has a full set of workflow modules. You see here many of them.
And we're really trying to take on the whole ecosystem of training for now.
All the way from data ingestion transformations. So what most people tend to talk about locally as augmentations.
Making sure that you can attract a download data sets and train and things of this.
That you have levels and by models, I mean, architectures that can be purpose specific or general purpose that are defined per.
So, for example, you can prescribe that you want your unit to have a specific type of block as the main core block and X number of layers with X amount of features at different dimensions.
And the networks are scripted are programmatic so that they can be automatically generated by a code.
Uh, loss functions, which are domain specific and again, the medical, the main has.
Huge amount of help us functions that are very appropriate for medical imaging, but not necessarily for other types of data.
But we also wanted to formalize the way we can provide ways for people to, for example, do evaluation to make sure that their algorithms work.
It can, it can send often very seniorly but for example, the way exactly that you implement a sliding window or the way, you estimate a dice for the way, you estimate, for example, tasks are fitness and things like that.
And have very, very different impacts on your copays.
And doing this once and doing this properly is something that we thought from the beginning that would be very useful for the community.
And you also we want to be able to provide these tools, so that as a research, or you can plot statistics can pump curves you can know what your algorithms are doing.
You can, for example, show images and the outputs of algorithms as the algorithms are training integration into different.
Visualization environment, such as sensor flow and etc, so with all of this in mind, there's actually quite a substantial technology stack that we need to think about when we think about creating all of this.
There's this layer of what are these fundamental components.
And in the fundamental components layer, these are the tools.
That you as an advanced research or someone that really knows what they're doing and must have full control over every single step of the way.
These are very likely the blocks that you would use. Right? You would take some data readers and writers. You will take place to.
To, for example, in instantiate certain networks, define certain loss functions transforms.
You would create your own training loop, but you, you will real certain metrics.
You will also take, for example, visualizations and interact directly with that, but different visualization systems and there's quite a few that we support.
So really, this is the base foundation tools. This is the toolbox.
That you would use if you want to build something from scratch by yourself.
Then the next level up is what we call workloads.
So, workflows have the engines on 1 side.
Even handlers and metrics so engines are the, uh, basically the Super systems that we define the series of sub components, such as the model, the data, the training loss, and all of these things.
And then you start training and the end, the training engine, for example.
Uh, automatically that's for you things like the training of remote it automatically, um, sets up the training loop properly make sure things are tracked and locked properly.
And if you then instantiate certain event handlers at different parts of the training loop, you can perform certain operations.
So, you can basically transform your training system into a single functional call.
And you can make that function called very powerful by basically.
Uh, introducing a new event handler set, there's something specific to the way you are train your algorithm. For example, you might want to do a, um.
Learning rate scheduler, which means 1st, but at the end of each, you might do something to your training. Right? And that is going to be part of a handler.
And obviously, then define also these metrics. So the target that we want to achieve.
And maximize 4 and these 3 things together, allowing you to find a work flow.
Where you can either train or evaluate a network.
And there's other workflows that are in engines that are coming out in the next few months, which are.
Will also change what we can do beyond just training any validation.
The next level up obviously is the examples. So examples you can think about them as fully complete pipelines.
These are things that we're working for still at the moment.
We do have good baselines for the segmentation task.
For a classification task for again, I'll twinkle their cost.
We have some examples for favorite learning and we have quite a few get started notebooks that really give you examples of. How would you.
Put these different components together into something that is end to end.
And that they can deliver and that you can deploy and that you can train and that you can trust.
What we are heading towards is actually to try to go from my.
Baseline implementation to a strong baseline implementation and that sounds like a, just in addition of a word. But actually what we're trying to do is to bring the community together.
And try to understand what are the best practices that will make you go from a.
Week baseline to a strong baseline. What should you be doing? What should be the standard.
Foundational pipeline that you should be doing so that, for example, the task of image segmentation performs well enough out of the box on a large variety of texts. How would you set up a system to do that?
So that every time a new person starts thinking about help segment and image.
They will start from the strong baseline and then you knew theoretical improvement or actually engineering improvement.
Would be always on top of the strong baseline, rather than all this, when comparing it to a old implementation that might not be working very well.
And the last very last level of obstruction is the research level. So these are less about.
Tried and tested a piece of work they're more about how we make sure that we can provide the functionality.
And the tools and the models to be able to do what.
The the field is currently things as the blue skies next step, right?
How do you make sure that you can handle multiple types of modalities and data? Not beyond imaging for example, how you bring in.
Genetics, how do you bring in structure information in a form of a 7 fire bundles and things like this?
The next 1 is, have you, for example, make sure that you can have a.
Models that can be trained across multiple, a large set of clusters. How can you make sure that the models themselves can be optimized?
In a non prescriptive way, using things like neuro architecture search. How do you create the frameworks to make sure that.
These things are in place and that again, simple examples access so that the community can bring on can build on top of those.
We're also thinking about the end to end research lifecycle.
Beyond just training, so we're looking into how do we make sure that models can be exchanged?
That we standardize how data looks like. So that's.
If that the systems can actually be deployed within the hospital setting.
When the appropriate safety around the top, obviously.
But we need to make sure that the systems know, for example, how to speak icon, they know how to handle fire messages. They know how to, for example, Logan, specific data sets of specific formats for certain types of modalities that we might not initially support.
And also we're thinking about, how can we use this package for comprehensive decision making.
Which is part of our deployment strategy is more about how can we make sure that has the capabilities.
That that enables this algorithm to when it is deployed to be actually used for clinical decision making with all the safety that that entails.
These last 2 points are still in the early stage of developments, and there's quite a few things that we will announce to the next few months. That will make this.
A little bit more clear, so, in terms of if I go into a little bit more specifics, for example, if I look into transformations in augmentations.
Again, medical imaging is very specific. It's not like in their computer vision setting.
That any transformation makes sense when we are in the medical imaging setting, or in the healthcare setting.
Most transformations need to be either biologically meaningful, or are possible.
Or they need to be physically meaningful, impossible and by physically, I mean, the physics of acquisition or the.
The, the technology that is used to require the data, so obviously we need to support basic transformations that everyone would use.
Things like, introduce you noise in the images, making sure that the images can be augmented by.
Spatial transformations of multiple forms, and that all of these things are conformance to, for example, the geometry of the data.
Sizes of bottles, 2 forms and that's for matrices and all of the complexities that you might have by going from bottle space to millimeter space.
You want to make sure that certain things are done in ideal ways for example.
The amount of papers that I've seen that, um, say, oh, we do these 3 types of augmentation on data. We 1st rotate and then we translate and then we do something else.
And the only thing that I can imagine is, oh, my gosh, if you are going to the sampling 3 or 4 times, they're going to completely kill your resolution.
But that's what everyone does, because the tools don't support those things.
So, for example, we've been looking to how do you concatenate and compose transformation so that you can do 1 off.
With some planes, how do you make sure that you confuse certain transformations? In a way? That is not only you optimize, but also optimal from the point of view.
Of the signal that you want to preserve, uh, and also how do you chain? Multiple transformations.
So, as I was saying, there is all of these transmissions that we are implementing more 9.
But there's also many other transformations that exist of part of other, um, tool kits that we are integrating with.
For example, these 3 tickets here bat generators.
Uh, and Phoenix rising there are 3 quite popular toolkits that I like to do many things. 1 of them being augmentation.
Uh, and rather than saying, oh, no, we're going to augment and do all the augmentation by ourselves and we're going to do it properly.
Actually, that's not the point. The point is to make sure that we're creating a community.
So, yes, we do want to have good implementations of all of these tools.
But there is no reason why we will always be at the edge of research that we will always implement all the things that people would want.
So, we actually work very closely with a core developments developers of these teams to make sure that create these adapters.
So, for example, these transform here that you see on the on the top, right is actually a composition of 9 transforms. We educate transforms with that generator transforms with rising transforms with portrayal transforms.
And they're all composed with each other in a single set of transforms and you can pass that that transform object.
Into any of the online pipelines.
This is basically a way that we use to make sure that these packages could be used in any way. We want.
And that we can adapt them, and the way that they deal with audio, and the way that they.
Do you expect the data to be in to make sure that they are all interchange.
And actually, some of these packages are likely now, um, maybe not all of those, but that was the stuff that set up to those.
Are likely to be deprecated because many of the developers are actually transitioning into more themselves, which is also something that is very exciting.
In terms of data now, obviously we are starting with imaging data and when you're looking to imaging data, there's many many ways that you can do.
Because we are looking into high throughput, high performance.
Uh, systems, we need to have ways to read data, but we also need to have ways to make sure that we can support many formats. And if you're looking to the medical community.
Obviously is the standard, but in the research setting, you have many other standards that are used right?
And making sure that we can implement all of those in the interactive way. And in a way that is consistent.
We decided to go with to support many of those, uh, formats either via advocate itself explicitly, or via secondary packages like.
Uh, in our support icon, obviously all of these things are still ongoing. We still do not support anywhere close to all of the functionality that we can have.
And that's because, I mean, tackling itself, it is a complicated standard.
And the requirements that we have in a deep learning setting can be quite tricky in terms of how do you define 3 dimensional sensors and what the geometry of those are.
Uh, there's some discrepancies there that we need to handle, and we're doing our best to make sure that we're covering the most that we can in terms of that comes standard.
It actually goes beyond that, right? It goes about it goes to.
How do you make sure that your data can be aggregated into the structure collections? That includes, for example.
Your images, maybe some bonding boxes around objects.
Some classes that those images might belong to, for simple, a diagnostic class for a specific.
A biomarker that you might be interested involves tours in the form of segmentations are, or, for example, DAC marquee contours.
Many of these things will be interesting. So those you can think about those as structured collections of data and how you represent all of those.
But also, how do you make sure that you can represent other types of trends of, of data formats?
Or sorry, data, storage formats. I wouldn't call them data.
File formats, but waste that you can store data to make sure that we are compatible with things that the community is now starting to accept.
For example, we have in the near imaging domain, which is actually where my research comes from. There is something called the bids format, which is a way for you to define.
What your subject sessions and time points and modalities look like.
And there's a very well defined standard on how to do that. Can we just by pointing at the bits folder load the full dataset?
Into memory and feed into neural network doing something like this will be really useful for anyone that is starting their dating its format.
Same thing can happen with 7 fire bundles. Same thing can happen with tools such as, uh, which is.
Uh, really well known tool now for, uh, for medical imaging, research trials and, and clinical research studies.
It's really about how we make sure that we cover all of these things at once.
Also, when it comes to date, the other thing that we realize is that there's all these challenges happening out there and actually many people are releasing large data sets. That's.
Uh, researchers are using in their own research 2nd rights.
And what's happening is the process that you actually have to find where the data is download the data format in a proper way. So that you can feed it into your network and your network can do something with that data.
That person is actually complex, and he has a lot of know how that is neither.
And he has a lot of particularity on how to do that properly. So, what we decided to do was, okay, can we actually create a way to store these data bundles.
And a API that people can use to actually that automatically download data sets.
That are that have very little restrictions in terms of license?
Directly to their learning pipelines.
So we decided that this was a barrier that we wanted to tackle we decided to standardize this data set level aisle. So I'm not talking about.
Single element I'm talking about the full data sets and also harmonize how babies tour for these open source data sets environmental data challenges.
And again, this is something that is evolving is not complete, but you can already use it. For example. Um, there's 3 data sets that are available. Actually 1 of them is going to be soon available.
Ah, but more are coming now, for example, you could just.
Call these mobile apps, and you can import the catlalyn data sets and after doing that, you can, for example, you say I want to have.
Task 5 of the capital dataset, which happens to be a prostate task.
And I want to get for some of the training sets, and I do not want to do any transform. So the data, and I want to download it.
And by doing this, the system will automatically downloads cash that they'd on your local system in the folder that you prescribed.
And it will persist and give you a pie torch data set that you can then use.
Uh, and the good thing about doing this is that when you do this, not only there's the system behind and this is an agreement that we have a, with a WS.
To make sure that we can have, we can serve large amounts of data.
For example, that that customer, I think the total is something like 100.
50 gigabytes, or something like that. We can actually serve that a month of data anywhere in the world.
But that we are formalizing, how these data store so that anyone will start from exactly. The same data sets.
And this is something that we want to promote as a good foundation for challenges, moving forward.
Is that obviously data can be served like, this challenges can have a much more standardized training and testing pipeline, which will then increase compatibility and deploy.
When it comes to networks again, I'm not going to go into details, but we really went through.
And assess the, which were the most common.
Blocks layers, architectural networks and losses that we're using different parts of the pipeline.
All the way from really simple things like conclusions and the residual units. 2.
Squeeze at sides, all of the crazy kind of layers that people have been working on.
Uh, attention is now also there and a few other things. Um.
You can also define certain types of layers, which are, can also be transforms. For example, you can have a fine transformation. That's part of layers. So possibly could use that in your registration setting.
There's really many many things that you could use on these layers for, by yourself.
But if you not want to build your own network, we also have Pre implemented well, validated networks, for example.
You have things like units for no new, which are.
Probably, the most, the 3 are 4, most famous general purpose networks out there.
And those ones are all implemented in a very programmatic way.
Where you can actually use many of the blocks and layers that I was discussing just before as the base foundation and then you describe, for example, the, that form the number of steps, or the numbers of features that you want that different stage.
The last thing that we also worked on from the point of view of this kind of training set up was really how do we.
Train and the new influencing and evaluation.
Uh, not only from the point of view of making sure. Um.
Our networks and train on the size of data that we are expecting our systems to have.
So, for example, if your image is very large, you might need to attract catches from the data.
But when you're doing influence, you then need to take those patches and you need to assemble them and aggregate them back into a whole volume. And there's again many ways to do this. And if you do it in the wrong way, you can actually create a lot of these continuity that make your prediction. Not very good.
Uh, so we implemented several ways of doing this, some of them, which are better than others.
Uh, and again we provide some recommendations on how to do this properly.
But we also provide metrics and the main specific metrics that you can use to assessor on. Uh.
Algorithms to make sure that these things are well tested well implemented and you can trust them out of the box.
If you look into the full workflow again, this is really a very easy way to look at the workflow.
But you can see that you can do a very easily in just a few lines for a couple of here is an application.
That takes the moon, this data sets and loads up a bunch of transforms.
Composes all of those transforms 1, not the, at the other.
It downloads the madness data sets to your hard drive specifically the training section of the amendment.
And then the next step takes that data set, passes it into a supervised trainer.
And the supervised trainer takes the data loader.
And network, which in this case is advanced net 1 to 1.
A Optimizer a loss function and influencing system.
And those 5 things together are the only things that you need to define a supervised trainer.
And he's supervised during their engine can then be called with a trainer adopt run and that is all that you need to be able to train the system.
To be able to, for example, do a across entropy classification of and madness benefits.
Again, this is a very simplistic task, but actually, if you replace for simple dense that with unit.
And if you replace costs, cross entropy, lost by.
Dice segmentation loss and you feed, for example, the data.
You can certainly do a segmentation network with exactly the same set up in the same number of lines of code.
So, this is actually very powerful and again, I wouldn't expect many of the advanced researchers to use this.
But he's actually a very useful tool set to have available because a lot of what supervised trainer does for, you.
Is things like multi he actually starting to do some, some degree of model parallelism, which are things that you really do not want to be implementing by yourself unless you.
Are really, really keen on banking your hand I guess the computer screen.
So, just to finish up on a couple of last things, um, we are also providing.
Some research baseline implementations for certain papers that we believe that.
To be importance. So, these are still not very well defined as to which papers we choose. But at the moment, there's a research group as Steven discussed before that is discussing, which ones are the papers that we believe should need to be implemented either because they will be.
And useful tool for the community, or just because it would be a nice resource to increase the visibility of mine.
For example, this paper here on covey, 19 pneumonia, legion segmentation is implementing 1. I think even more interesting than that. Is this model here called.
Which is a way for you to do level parallelism on the fly.
So this is where you can take a single model, let's say a segmentation network.
And split the segmentation network across multiple in a seamless way.
With small batch sizes so that you can actually have larger parameter sets so that you can have largely input patches because rather than being constrained by the memory of 1.
You can be moving that they are moving the activations and, uh, the different parts of the network across multiple.
And this model will implement a way to do this in a very optimal way. That doesn't, for example, wait.
For the 1st, to be flushed so that the 2nd, you can take over because that would be really, really, really slow.
So, again, he's engineering jointly with research that is trying to push my my horn.
Brad is going to talk much more about this, but we also have some early implementations of learning again. How would I be used within the Clara federate learning setup?
Uh, the in such a framework, and also the open mind framework.
We are showing that 1 nice flexible and we are providing implementations of how to do further learning.
With all of these different packages, so that you as a user can choose, which 1 is best for you, because all of them are have completely different capabilities.
We're also looking into really working very closely with biomedical chat, Grand challenges and the BIOS initiative.
To make sure that the models are reproducible that the most can be changed that we can.
Understand, um, um, how to hobbies most are working and that we can understand their bottlenecks and their efficiency.
And we're also looking quite a bit into, um, into repetition from state of the arts networks.
To make sure that the networks we are providing our indeed stick to the arts and provide strong baselines for what people are doing.
So, just to finish up last line I promise is the features of point 3, which I've been released very recently.
So this says, as Stephen was saying, this is released middle of October so, a couple of weeks ago.
During version, 0T.3 D, what we did was we released quite a few more domain optimized networks, last functions metrics and optimizers as we always do in every release. There's always more functionality to that list that I was showing.
But we started to provide some early foundations for to distribute the training.
Uh, where you can have a data parallel and fully data parallel across multiple machines already implemented.
Automated mixed precision either as implemented by pie torch, but also by providing tutorials to make sure that.
You are optimizing your networks in the right way and that they're handling precision properly.
Because that's surprisingly hardly again in the medical setting we have a completely new way for us to do that is now in, it has kind of a factory, uh, template.
Associated with it, which allows us to.
To find loaders, um, beyond the ones that we defined within 1 night itself.
But I think 1 of the most important things that we're releasing version is, 0T.3 is a fool.
C, plus plus could support within 1.
What this means is, and again, if you have done medical imagery.
You probably have realized that certain operations are not available.
In, let's say pie torch that you might want to do in a medical imaging. 2nd, for example, if you are doing.
I'd say object detection and you want to, for example, do non local maximum suppression.
Type torch does not implement no local maps and suppression. E3 be.
And it's not like he's easy to implement in actual pytorch called. You would probably need to do it.
In our in C to be efficient. So we actually created the full continuous integration and continuous deployment pipeline.
That allows us not only to serve those pieces of code.
And compile them when they're submitted, but also to making sure that the kernels are available.
At the time of training, so that you as a user, you're not just calling pie torch functions. You'll actually now calling 1 night kernels.
We are using the piper API in the background to make sure that you can do things faster.
Uh, and in the more consistent point, and again, we have many new examples, notebooks and things like that.
I think that's it from my side. So I'll now pass the ball on to brown so that he could give us a little bit of a.
Of a brief on the Monday and clinical workload side of things.
Can you hear me? Okay I can. Yep. Excellent. All right. Let me share my screen here.
And you can all see yourselves.
There can you see my screen? No, not yet. And I made you presenter.
Okay, let's see, maybe I need to hit it again. Share.
2.
Perfect fantastic. Okay. Thank you so much. So, this is I'm thrilled to be able to be here today and present a little bit on.
Uh, the clinical workflow side of things, I won't be focusing on Clara the framework and then how, uh, models and other things can flow from an AI into the clear framework.
So a little bit about me, medical imaging alliance manager with NVIDIA.
So, the NVIDIA Clara framework, it's an application framework.
That consists of really 4 components, uh, to help enable, um, workflows. It has both.
Uh, Pre, train models a place to start from on. I've.
Uh, miles have been trained on various, uh, data sets that are available publicly.
It includes a component for assisted annotation to help.
Create annotated data that can be used to train models. It has a training framework.
Uh, that is using just transfer learning, uh, and looking at in the future and the last piece is the deployment framework.
Which is where we can take models that are trained using our training framework, or or really anything else to connect that into a medical imaging ecosystem.
So, in terms of looking at the, uh, the workflow on the, uh.
The annotation site unlabeled data and annotating it. We've got labeled data.
Feeding that, to training, uh, doing things like February and learning, which we'll talk about in a minute to kind of build out what collaborative learning really is.
And then feeding all of these outputs to connect that into an application that could then be deployed.
Into into an ecosystem.
So, lots of cool places to get started, whether it's.
On brain tumor segmentation, there's a bunch of these from Pre train models.
All of these are available for download uh, they're, they're free done NVIDIA dot com.
But you can see what medical tasks they are, whether they're a segmentation or if they're a classification, the data sets where they came from and the types of networks that were used to train of them.
The trending framework.
Consist of, as I mentioned the access that annotation.
This is what can take a dotcom imaging study as well as a particular organ or a condition that we want to annotate for.
Feed that to an API create.
Uh, a mask, like, we then rendered inside of a medical viewer application.
So, whether it's 3 Slicer or MIT K or oh.
Whatever that happens to be, it's an API that can be called to help rapidly create annotated data.
On the training side, the declare train piece, which is pie torch based Claire. R. T. today.
This is something that we're also recognizing the direction of the community tours pie charts looking to encompass. That is really important.
We created what we call the medical model archive the M a. R as a means of creating portability of models that are created that we could then take to other sites as well.
So, I think of it as kind of a packaging that I can take from site 1 to site 2 or workstation 1 to workstation 2. so that we're all working kind of off the same page in terms of the models that we're using.
And then we could take those things and then connect it up into an imaging workflow.
I includes things like February learning, which we'll talk about in a 2nd.
Auto mail, automated mix, precision and other things that domain optimized, uh, in the medical imaging space.
February learning is a pretty exciting area of development we put out as in Pi,
2019,
February learning tooling that can help enable this if,
you know,
in terms of what Federated learning is,
in our context.
This is where, rather than pool all the data from all the hospitals together into 1.
Site or 1 collection to instead leave the data in place at the individual hospitals or repository, or and move the training on premise to where that data is.
So that different hospitals participating in the same training run. They don't need to expose that data.
They don't need to worry about all the, the actions around anonymization and showing it in a public forum that data remains private on premise.
And what happens instead a model from a central location gets fed to all the participating sites.
We do all of the training, collect the model ways, and then send a subset back to that central location where just the waste themselves are average and then we do multiple rounds of this training.
Way to to help a.
Work together and creating model are actually sharing data itself.
Part of the framework that enables this it's all done through just restful type protocols. Very firewall friendly in terms of how do you participate from enabling your IT departments to communicate?
It's done over SSL. It's all encrypted end to end.
Claire deploy on the flip side. We've talked about access annotation. We talked with the Pre train models. We talked about training.
Uh, deploying into an imaging eco system is where Clara deploy comes in.
Uh, and what this allows us to do is construct operators, which are single tasks and string them together in a pipeline to accomplish a particular work flow.
So,
as an example,
I could create a pipeline that takes them die from a Pax,
or does some,
sort of Pre transforms runs in France creates a result as a dotcom or fire object and then send that to whatever the destination system happens to be there's a bunch of different pipelines that are already
available just to,
as reference implementations to help get people started and there's a bunch of different tools like monitoring and visualization that we can use after the fact to analyze what's happening and those pipelines.
So, an example, 1 that we have up there for coven, 19, here, we've, we've got a model that can do classification as well as a model of segmentation before we actually do the classification and end to end pipeline would look like.
I received com, I transform to nifty. I segment out the lungs.
Uh, I classify the lung tissue, I create a object, and in this case, it was a PDF wrapped object that I then send to a packs.
I can construct whatever objects that I happened to need for my particular workflow, using operators that are written in whatever language that they want.
It's all hung together using Docker and cubernetti's and then how we take that pipeline and connect it up into a imaging ecosystem, whether it's a, a Pax or a.
Or image manager, or with a router or.
Uh, some sort of a marketplace, all those things are completely valid and how we move content to and from that particular ecosystem.
And then, so if you're interested in learning more or want to get, you know.
Experiment with these this framework, you could just go to developer NVIDIA dot com slash paramedical admin. So lots of different content there really?
To help accelerate get you started free to download and do your research and experiment on.
So, I got Thank you very much. Thank you. Great presentations. Lots of.
Interesting developments. I'll like to open the floor for questions.
So,
I have a couple of questions,
which may go either to Clara or Monday so a lot of training,
especially for larger models are done on high performance computing resources sources,
which generally have a batch coding system you might have mentioned it.
But, I mean, I'm always double tasking, so I probably missed it. Do these things tie him to high performance computing resources?
George, you want to take that.
Yeah, I, I can, um, it's part of the, it's part of the pipeline. It's not there yet. So, at the moment, most of the things we do both within kings and and video, which.
Start with the package is mostly Docker so everything is just submitting Dockers to some training system and that will be work on the background. We are looking into many other. Um.
And that's doing systems, um, either kind of the classic.
Somebody the engine replica, but also some of the more modern ones that everyone is using now.
We don't really have a a kind of a requirements.
A, well designed to do that if you have some ideas in mind as to what kind of functionality would you be missing?
That will be something that we will be delighted to bring on board.
The things that we currently have we've been thinking about is, for example, if you are doing things, like hyper parameter tuning, or if you are trying to.
Train on local data sets how would you make sure that that can be automated from the point of view of the queuing system?
But we haven't really gone much further than that much be on the kind of the hyper parameter training type of job.
Set up that some people do you have any, what were you thinking? Exactly in terms of functionality.
So, for example, some researchers, particularly for training of really large models, just using a single NVIDIA.
Graphics processor on a local, even if it's a fancy workstation, the horse power to do the training, you need to get into a nice big array. And most of those are behind Super computing walls.
That's kind of where I was coming from so we, we do already support data parallelism as to the degree that is supported in pie torch. So, pie torch as the data parallel, and the, uh, kind of the fancy data parallel.
So, they have 2 versions 1 of them, which is mostly in the same physical machine.
And the other 1, these multiple cues over multiple machines, and both of them are support that 1. I.
But they would be supported in the same way that you would train any.
Any pipe Parch work workload in any other cluster, which means, you will still probably need to do a little bit of work, set up the communications between multiple nodes, and making sure that the job is issued in the right way.
But Piper should cover most of those things by itself and we are fully compatible with that.
Okay, sounds good. The other thing, you know, Brad started talking about Federated learning, and it seems like that's, uh, you know, from other.
Forays into that area, it seems like it opens up a big can of worms. Um, so.
Are you, are you taking steps to try to improve some of the follies of Federated learning?
So, George can address some of that, but I also know that that is Daniel Rubin was on a 2nd.
He's still on the exterior, so he actually hence the Federated learning working group for Mona and so perhaps we get to that opportunity.
Daniel, hi, Larry are your is your concern.
Protection of patient, privacy security? No, not not. So much that, um, you know, certainly, no expert in this area. I value the greats.
Like,
I'm Brad and Steven you and others,
but the,
uh,
um,
from what I've been reading about Federated learning,
uh,
the part part of the,
you get different answers,
sometimes when you do learning in a federated fashion,
as opposed to a centralized fashion.
Yes.
Fantastic point. I just wanted to make sure I unpacked the, your product. There's many challenges.
In terms of this isn't the panacea. This is still a research area. I think the great thing that when I project is doing is trying to move, there's a lot of complexity underlying setting this up.
Setting up a training of a deep learning model by itself, but then to federate that.
Then there is more complexities in terms of.
Communication across the centers, the, the training of the models, and then the whole approach by which it's done, you're getting at.
And we've been looking at and you're absolutely correct.
You do not necessarily get the same results or the same model. The performance of that final train model is not necessarily the same as when you centralize the data.
And there's a lot of and that's actually a research area.
My lab is focused on as well, but what is the extent of that? How do you mitigate that? And there are strategies that are there so this is in evolving area. I wouldn't represent to the same.
Oh, here's the greatest thing in the world. This is going to solve all of our problems, your own data sharing.
You're absolutely right that at present you don't necessarily get the same result.
And the is still work to be done through research, use cases to fully understand.
How much of a problem that is and we've looked at the segmentation task and classification tasks.
Not so much a problem classification test a little more of an issue with segmentation.
Um, but it's still work to be done. Does that answer your question or? Yeah, yes. Thank you. Sure. Yeah. I can just add to what I was saying. That's the reason we are working with many of these other packages on the other 2 learning side.
So many of these packages do provide.
And they have ways for you to set things up. So, for example, if you're really interested about some of the.
More theoretical aspects of fabric learning, including the ones you are describing just now, but also.
Differential privacy, privacy, preserving querying data and things like this.
Things like, would probably be the best platform for you to use if you're going to if you're with what you want to just deploy.
A bunch of Docker machines in multiple, um, hospitals in a way that has been tried to test.
Then you have things, like, then be to Clara or the such a platform that are actually more adaptive to that kind of large scale deployment and less about the theoretical, um, kind of considerations behind it.
We are trying to support those packages as much as we can. But the idea is that we only implement things that the packages and not provide.
So, if we identify that there's a component that is missing, that components is then something that we would implement ourselves.
There's also a question, uh, I, I hope Laurence, this answer to your questions and then you'll, thank you.
There's also a question in the chat from the while and the question is what are the plans to add tools for explainability.
Exactly, yeah, yeah. So you know.
Activation map, visualization techniques and variety of techniques can grab and so forth.
Clearly, they're critical to.
Clinical applications and medical, clinical applications and learning systems.
And again, we're following from other tool kit. Suddenly pie torch always already has action and support.
For the systems, and so it's a matter of integrating them with them on that.
George yeah, more to say. Yeah, so, I mean, pytorch supports a lot of that we are using also the packages that kept him and working with the developers.
Make sure that the 3, the world is fully support to.
Again, the kind of mantra that we decided to go for is.
Let's try to choose the best and if there is no best, let's choose 2 or 3. very good.
Platforms that are out there that our purpose specific, and let's work with those developers to make sure that we can add the functionality that is missing for the 3 D world or the.
And the worlds, depending on what kind of value of use.
And as George mentioned, you know, it's the 0T.4 release and has a 0T.3 was just 2 weeks ago, 3 weeks ago. So 0T.4 scheduled for joining.
I believe these somewhere in December, but I'm not entirely sure I would be. I don't want to be quoted on this. I think it was some, some time around that so real soon.
And but also as a community again, the number of contribution to receive is phenomenal. And there has been contributors in that general direction.
So, if you can have your favorite tool kit, you're lucky is and how that should be done.
We run a Slack channel as well as the wiki page and then the issue tracker is heavily used on the get of account.
And the issue tracker is used for feature request from the community as well as suggestions on how to address those features.
And, um, it it's amazing how many tens of times a day. Um, there are messages posted there that are discussions, not by developers, but by users.
And so welcome ideas to be submitted there as well.
Absolutely, Andre just posted another question. And his question is how is supported why does the sustainability strategy for the project?
And he apologizes because he was a little bit late to the meeting in case. This had been mentioned earlier.
So, our goal is definitely a community supported library and it began in video.
And, um, team scholars on and quickly realize that.
It was something that they did not only have the resources to initially develop.
Um, but they looked elsewhere for expertise and really their goal is at stake up front, too. Just want to build this reference software library.
But to really establish his team and community support, I think that's kind of why I was brought in.
Um, we've seen with 50 K and so forth that if it really is day 1.
A community supported effort with, um, open and transparent policies and procedures that are not just, uh, touted, but actually applied.
Uh, these toolkit can exist on their initial funding.
And also, 1 of the great things is the number of letter of request just got another 1 today.
For people who want letters of support for grant proposals that they're submitting to the now, we send them my own shortcut.
And this is where we see because of the community support and the nature of this, it's everyone getting their individual fundings to go after their.
Special type of network, special application and so forth and then contributing to the tool. Okay.
And then people taking different levels of workers and their support that toolkit, as you've seen on 3 slides or another platform to yourself.
Shoulder George, and Brad perhaps Brad and maybe you've also been working on a large number of volunteer projects.
From the di comment beyond, um, do you have any insight that you want to offer.
Well, only just to echo Stephen, what you were saying, I mean, it's.
We don't want, you know, a bunch of different projects, all doing similar things to go off in different directions.
By working together, around the idea of NY to pool, these efforts across communities and build something great. We all win. And I think it just makes a lot of sense a lot of really great projects.
claire's 1 of them and all, you know, all across the other participants, we're, we're building something great.
And the fact that we had such a already big community around this with our boot camps and things like this just emphasized just how great this initiative I can be. So, that's something I'm really excited about.
Justin is asking a question and a chat and it points to every pipeline data. So, his question is.
Uh, 1 of August slides showed a single line of code for importing a handful of data.
What is the process for making additional data sets in this way?
What would be the best way for tca in the imaging data commerce collaborate with 1 eye on doing this.
Outstanding question, and we've tried to make that easy as possible to define data sets with pie torch approach. And so George yes. So, I mean, this is a collaboration with WS. They have a open data sets policy.
Which allows you to basically use S3 buckets with almost unlimited.
Data query as long as the data adheres to certain standards, and they are very open to, um, to be flexible around what the license would look like.
That data, and then if the data is available there, and we have a specific pathway, we didn't need to make sure that that process is prioritized.
We would then work you would then work with the I'm on my team.
To define the wrapper around that data query. Right? So there's a S3 API which query you need to find the data set in a specific way. And if you do that, then you get back a pie torch data sets as the standard.
And it's just a question of making sure that the data started in the right way and that that's rapper's. Correct? And he's basically put data in S3.
Right the data kind of the derived object from the dataset object and after you have that you send a full request and is there.
But we would also be able to, to help you make sure that that's available. And by working with you, rather than just telling you is done here. So just do the same thing. And this request.
But I think it's also feasible for the just jumping in to offer a similar API to the S3 system,
or to provide,
um,
its own method for making sense of its data available as.
Pie torch, datasets and advertising that makes sense.
And so you don't necessarily have to go with the data system. You just have to support the API at some level, 3 level, or at the torch dataset level.
In order to participate and is the 1 who is a been in charge of challenges, and the guy for several years, and he's been working towards standardizing them and she leads the challenge is working group.
That's addressing many aspects of this for them on our system. So, she leads the challenges working group.
Recommend having a conversation with her as well as, as well as the technical team to identify where.
To do we have sorry for jumping in? Do you have.
Uh, documentation that instructs how to build such.
I don't think for data set specific. I don't think there is extensive documentation.
Because that's very new. It's actually a good idea. I think we should take note of this and communicate back to make sure that people can contribute.
So, we are currently working with the grand challenges team to track together a list.
Of the all available open source.
Data sets as part of many challenges, and the plan is to email all the kind of the leads of the different challenges to explore the possibility of them, making those data sets available.
Under the, um, data download, but obviously the same thing that happened with any other institution, including.
What that we should add Justin to that email list. We definitely.
I'll save it. Sorry I didn't mean to interrupt you.
No, no, that's okay. Just really quickly. I mean, 1 of the things that we've talked about in the month data group.
Uh, is that we don't really want to pick.
A format that represents all data, uh, because the, that it's such a big expanse. Right? It could be. Someone just wants to say, here's 100 images. Here's 100 labels and that's it.
Whereas others might say I, here is a cohort of data that has the current study.
The prior study, the lab results, the disease indication, and I want to feed all this to 9 to do training and so by providing and extensible Transformer based mechanism that people can write.
Uh, their data structure in a way, that could then be important. That is really important.
So, I just wanted to learn Starbucks since. I'm the chief architect that tca I'm really interested in these discussions.
So we talked about providing dotcom web interfaces into dotcom based data tca but TCI is now getting data in all different kinds of formats.
Not just com and we are migrating starting migrations to are using unnecessary back end not 1 from Amazon, though, but it's.
We're keeping it well, hidden from users. They don't see that at this point in time. So I'm intrigued by this notion. Well, what about exposing it directly?
You know, there, there are security concerns there, because some of the data sets in in 2 are restricted, they require a data use agreement in order to access them when we have to make sure they stay well protected.
But but but I'm interested in the concept. It sounds interesting.
I'm going to mention your name, Larry, I'll name, drop to Atlanta and I think that your participation cause you had been doing this since you and I've worked on that with.
Yeah, couple of years ago we'll leave it at that. I would love to. I think your input would be huge with all the experience that you have.
For dealing with these different data formats in a web based system. I mean, that's outstanding. We need that.
I'd be happy to to participate in discussions. That'd be yeah, it's a really active working group. 8 am on Wednesday mornings and so time changes are always challenging and so.
Pacific time or Eastern time Eastern time Eastern time even earlier. Yeah. So, uh, so we'll see what we can do. All right. All right.
No worries but a really active group by the way 1 of our most active working groups.
So, I think that's where a huge impact can we add? Yeah.
So, this is came on, we're sort of coming up to the end of the time, but there are a couple of questions that I want to, uh.
In the chat that I want to just point out 1 from Dave Clooney, uh, he points out that on Google.
Also, can I be ported to Google healthcare platform?
And the other questions from spirit spark us about how can contributors to the system get acknowledgement from paraphrasing his question. It's, it's in the chat. You can see it.
But Google,
Google,
healthcare,
George,
you want to give that to Docker or your response to that a brief where I don't think where we are at that stage yet if you have any inclination to help us support that we will be the license.
I don't think we have gone that far. I mean, we're still thinking about the.
Training ecosystem inside a machine on private clusters.
If you're going to be integrating this with all these guys and things like this, I mean, that's a really interesting question, but I'm not sure I'm the right person to answer.
I don't think I have the knowhow. Yeah.
Again, without acknowledging I'm sorry go ahead. No, no, I'll jump on a far acknowledging so that is a partner who's been addressing out to acknowledge our outreach and communication.
We know how important it is. 1 of the things that we've identified are already done this established a separate repository called the research tool for cutting edge algorithms and so forth.
And that's 1 way to get acknowledgement. And George. I think he wanted to.
Say something about that, but but we realize that a community.
A lot of reasons why people release and participate in open source software is.
To have an impact and have that impact recognize.
We have a variety of mechanisms in which we recognition to people for any mechanisms for this.
And George, you can give some personal that, I mean, it's always a little bit complicated with a project like this that is evolving to make sure that every single person is recognized to the same degree.
Definitely, every time we implement a or.
Network think like, from a specific paper as well document, the patient where that idea comes from, um, if you, if someone is to jump in to, for example.
In year, 4 or 5 of development, which is a few years in the future uh, they are likely not going to have the same level of recognition as maybe the original would have.
Uh, by developing a lot of the base foundation, but that's also something that we're always trying to get, um, uh, opinions from people is is what would be needed to make sure that.
You can be recognizing that you, you will be good to contribute to the system that.
This discussion is happening with several people. Several people have completely different ideas of what that might be.
So far we've been very open to all of those and actually we've pretty much accepted all the.
I wouldn't call them conditions because that's a strong worth, but all the, all the suggestions that we were asked in terms of that kind of recognition.
Everyone has been very, um, very reasonable about it.
Uh, so this is something if you will like to contribute and you, if the thing is, that is stopping you to contribute.
Is the lack of recognition, uh, if you think that you have a solution for that just get in touch.
We will be more than happy to to make sure that that happens.
Okay, thank you so much. This was great.
Obviously, this group is very engaged in your presentation, and we'd like to follow up with you fully have you back maybe in a year or so to to report progress and by then hopefully, we have some collaboration going on with stories.
Thank you again. This was a great presentation. We had speakers from the West Coast, East Coast, and the UK. So thanks technology.
Um, I think before closing the session, uh, Justin has some announcements about developments and then we'll announce the next couple of meetings that are coming up and alternate to Justin. Thanks.
Yeah, just real quickly since I know we're running over. I just wanted to mention that we're real proud of the fact that surpassed the 1000 publications based on data sets that are available in the archive this month.
So, it's a really great achievement for everyone that's submitted data and who's written those publications about the data sets. And then I also wanted to mention, we put we published our 1st ultrasound dataset. So you can take a look at that.
Now on the collections page. It's over a 1000 patients, so it's a pretty large 1 and.
Sort of piggybacking on the earlier conversations about the challenge competitions. There's a new guy endorse challenge competition, using the cobit data set that's in TCI. And so you can go take a look at that.
I put a link in the chat to the TCI Twitter, which has more information about all these things.
Okay, thank you, Justin. So I just wanted to say something quick about the upcoming.
I see W, webinars on November, 16th, 2 weeks we'll have a special additional W with a presentation binder better off, uh, about the imaging data comments.
We will be sending out announcements about that. And on December 7th, we have our colleagues from the National Institute for allergy and infectious disease.
Artifacts Institute to present their portal TB portal that's a combines imaging and genomics data.
And be interesting for for us to interact with them. So, with that, I want to thank the presenters, uh, the attendees and we.
Again, and hope you have great rest of your day. Thank you.
By Eva, thank you. Bye.
Thank you.