Hello good afternoon. I'm coupon for honey, but.
Center for by medical Informatics and information technology at and.
And it's my pleasure this afternoon to bring you to excellent presentations by, um.
Free prior from University of Arkansas, medical sciences, and also by then Coptic and Dina with daddy from Deloitte. So these 2 presentations today to an overall.
A project that we serve a year ago, and I'll just take a few minutes to give an introduction.
And put the presentation in context. So, uh, what is the need for medical image? The identification will the health insurance, portability and accountability act, or as is known regulations.
Require removal of protected health information, um, from medical records, including images before they can be shared publicly.
Uh, through open access, media, such as, uh, the cancer mission archive, imaging, data comments, or other.
Public repositories as their rate of data increases and, um, interest in continues a need for sharing up more data.
We are interested in finding base that are automated or semi automated, um, to the identify images.
This is Twitter is large data sets and also find solutions that are scalable.
Potentially cloud based and and always allow for human in the loop.
So,
last year around May of 2020,
we started this or conceived of this project called medical image,
the identification initiative or metaphor short,
and it basically included several projects,
including the 2 that you hear about today uh,
project,
1,
data set for validation and evaluation of.
The identification and.
On project to which relates to the 2nd presentation you will here.
Today on image, the identification using automated cloud services.
Hey, Brian, what's that? I'm sorry.
And, um.
I have a lot of time, and I just want to I want to make sure that I get to the acknowledgement's.
Because these projects, although they were relatively small. Uh, they.
Benefited from cooperation of many teams and many individuals as well as colleagues from across campus. So, 1 acknowledge a bit. Um.
The cancer program colleagues from,
uh,
and resources from an that were facilitated through the anti strides program and,
of course,
individuals and experts at Arkansas and Deloitte,
um,
colleagues from Frederick National Lab.
For cancer research David Coney with Matt and of course.
Folks at Google, healthcare, so without further ado, I turn it to.
That afraid prior from University of Arkansas to give the 1st presentation.
Thanks, Eva. Uh, let me share my screen.
I think you need to give it up and let me make me presenter perhaps.
I make you presenter there we go.
That should be, we should be seeing the screen now. I believe. So, as Kevin said, I'm going to talk to you about the Mini project, which is the creation of a data set that allows for the evaluation of.
Identification or anonymization.
Tools and pipelines were all very much aware that open access or shared research data has to comply with with HIPPA, which.
Provides regulations that govern patient privacy.
They, these regulations are private removal of both protected health information and protected.
Are personally identifying information? Sorry?
From beta sets before they can be shared there's a growing number of image and anonymization and analysis pipelines or curation pipelines and.
Algorithms that are being developed. Some of them are commercial products and there's really no good way of understanding what they do in comparison to 1 another. And it's it's to me it's less about, which is the best.
And as how are they complying with some standard way of understanding what they're supposed to be doing? And what did they actually do and so.
The idea here is to be able to.
Run some type of validation process, which I'm, you're going here. I think perhaps later more about that, but.
Any kind of validation or analysis compares algorithms requires a data set.
Tci has been asked many times. Can you please give me the data before you anonymize it so that I can test algorithm.
And the answer is, no, that would be illegal. But now the answer is, yes, here are Here's data that is synthetic. That looks like patient data that you can use for validating algorithms.
Before I move on though, I want to address something, which, I don't know, but the rescue, but it drives me crazy notification anonymization and.
So following, because she lovely paper identification of medical record data, refresh the removal replacement of personal identifiers.
So that would be difficult to re, establish a link between the individual and their data.
Hippa, at least 45 CFR part 160.
Says that get invocation requires the removal of specified individually identifiers and.
hibit happily gives us 18 classes of such identifiers.
As well, as the absence of actual knowledge, by the covered entity covered entity is a healthcare system, like the University of Arkansas for medical sciences lack of actual knowledge by on the part of the covered entity.
The remaining information could be used a loaner in combination to identify the person. So it's a.
Workable standard, it's something that you can work with.
Anonymization, and now I'm going to switch to definitions that come from the use general data protection regulation, or GDPR.
Anonymization refers to the irreversible removal of the link between the individual and his or her medical record.
To the degree that it would be virtually impossible to reestablish the link.
Actually, GDPR says identification would be impossible so it's a very high standard.
Um, it's so high that, in fact.
If you actually can prove that you've anonymized the data it's outside of GDPR. There is no regulation because it's no longer considered to be.
Personal data anonymization in under GDPR is very difficult to achieve and I'm going to talk a little bit later about how we're trying to approach.
Getting you to get its head around what what anonymization might mean in practice, because they do not provide the equivalent of hip as 18 identifiers.
Sit optimization again following GDPR.
Is the practice replacing the personal identifiers with non identifying references, which is what does.
In its standard D identification process.
So, within GDPR, most data is.
But there is still a set of legal documents that have to be in place between.
And health care institution and anyone who's analyzing that data. So these 3 terms are frequently confused. We tend to use the identification within the TCI world because we work in a hippo environment.
But I wanted to make you aware that these other 2 terms are there and to try to keep them straight. And so, for most of my talk will actually use anonymization. And because.
I tend to live in live with in this international framework largely because of this.
Dotcom, as you well know, is the.
Um, international standard, perhaps you didn't know it's an AI it's a 1252 as well as being the com standard. It's, it is the international standard for exchange of digital imaging.
Information and related information, it includes a standard for medical image. Anonymization.
And that's the security assistant management profiles. So this standard defines.
In detail what data elements you need to modify, and in what manner to achieve specific levels of anonymization and optimization.
So, to me, this is critically important, because it is the only international standard that we can use for.
Helping us to understand how we might define anonymization get edification and sit optimization.
Within we have the past 5, actually more than 5 years used a set of tools called pasta.
That are open source freely available and around those tools, we build a scalable workflow that allows us to.
The identify and.
Data not just com data, although we primarily focus in that kind of data but we actually also deal with non Dotcom, form, image, formats for radiology and pathology as well as clinical information.
But primarily we focus and paused on dotcom because we have this standard for defining what curation means. And so.
tca and paused it use the basic application confidentiality profile from,
from the security assistant management profiles standard with this set of profile options.
So, to me, any tool that's going to claim to de identify data should be able to tell you the equivalent information. What profile they using.
Are they using any options and that would then specify how they would modify any real data or synthetic data.
So, you could predict the outcome, and I think this is important that it should be a requirement that any algorithm provides this kind of information about what it is that they do.
All right, so, whether we do to build a database, this is the general work flow. It's so a lot of details, and in describing the full database that we created for.
And I'll talk to a little bit later about a subset that we've actually published, but.
We developed this icon data set to be used for the evaluation of demonstration algorithms.
We selected dotcom objects that had already been identified and published and so they were open access and we created some tools to generate synthetic that we then inserted into
header attributes.
So that it would, you end up with an output that mimics a typical imaging data set that you would find in a hospital.
We rely on the standard which defines.
What data elements should should contain standard elements and then TCI curation logs where we've found a number of.
A large number of instances where.
Duck into places that that icon standard prohibits its existence.
And so we have this database of such information it happens in real life. And so we insert some of these odd cases into the data set as well.
So, we create the synthetic data, then we use the tca process, completely independent set of tca curators.
Curated the data set as if it was brand new, they never seen it before. And that generates an output which.
We believe is properly curated in accordance with this profile.
We generate then an answer key, which is based upon what we know we did in the synthetic data.
And that answer key says what should have been done to modify.
Uh, the data, if you were following the profile that uses.
And we double checked to make sure that the tca output actually matches what it says it should in all cases. And then finally there's a Python evaluation script because.
As you'll see, there were 426 patients.
With 40 when studies and about 57,000 images so you're not going to evaluate this by hand you need a a script that actually does it. And so that's a script that processes the answer key and the 2 data sets to evaluate what any new algorithm.
Has done so the.
This is the description of the large dataset. This is this data. So the by NCI to be used for.
They're on purposes when evaluating.
Algorithms so, the total data set contained, as I said, 57,000 images that were pet and digital radio graphs.
18 equipment vendors were represented in the sample, including a small set of synthetic data from the victor collection.
From tca, in some cases we burn Phi into the pixels into particularly into radiographs and some of the radio graphs contained implants with serial numbers.
And as you can see up here, there's.
Phi stuck in places that it wasn't supposed to be. This is not a body part.
Obviously, but it's part of the, the synthetic data, so.
That gives you an idea of the size of of this original data set.
These are some of the unusual places that we have found again by analyzing the tca curation logs. We keep track of every time we find 1 of these unusual things.
We build it into automated processes to make sure that we.
Uh, always check in these locations and so s, a sample these numbers are how frequently this occurred in the tca audit logs. You can see in some cases.
Like, patient comments, they're always full of paychecks so we sent we took some of these to include into the data set.
There's also private tags or vendor specific tags where they can extend the data set and they, of course, are owned by specific vendors.
Hence their, the identity shows up here and these are just an example of places where.
Again has been inserted where you wouldn't expect that you would not expect patient data in a coil ID data field but.
We found it there,
so that's to give you a flavor for the kinds of of not only standard mappings,
but also non standard mappings that we put in the data set in order to test algorithms for how robustly they can deal with the real world.
Experience that we've had over the last decade of curating data for.
I mentioned we published the dataset. This data set is available on it. It's only representative of 21 patients and their.
Associated images, and so this is the original evaluation, synthetic data.
This is after the tca processing and then this is a crosswalk of patient identifier since they're modified as part of the.
Process and so you can.
Use this to test your own algorithms it's a small data set, but what we want to do is get that sample out so that people can understand and have experience with using this data.
And as of this morning this manuscript is accepted by nature scientific data. So, there will be a paper describing the process out shortly.
So, in the remaining time, I want to talk to you about a little experiment that we've done on our own in, in conjunction with our European colleagues and a project called. You can image that we're part of. And it's very similar to what.
Um, what planning to do in this case, we created a separate.
Data set, it's a small data set and mammography only at this point.
Specifically for this experiment, and it was an experiment to.
Be presented to the European Union to try to get them to get their head around.
The question of how do we actually define anonymization under GDPR? And so, in this case we ran through the entire process of we created a synthetic data set and then we.
Uh, de, identified it through or anonymized it in this case through and pasta.
We had a collaborators from Sam Rad, which is commercial company, use their processes and then we went through the complete evaluation of the test script and answer key to compare the 2.
Now, because this is for GDPR.
We did not start with images from TCI. We synthesize the images.
This is work that with our collaboration with him on energy who's got cut off at the bottom sorry? At Emory. So these are not real mammograms. These are generated by cycle Gan or generative, adversarial network.
The reason for this is to make sure there were no questions about the validity of the data within GDPR. And also, because we have colleagues that want to look at differential privacy, which can be calculated using.
Uh, it's again method for generating synthetic data, so.
This data said is not the same 1 that I've just been talking about. It's completely separate is quite small, but it was to go through the entire process. And again with the, with the purpose of trying to get the European Union to.
Deal with how do we really define anonymization?
So this is a real example of a of an answer key that was generated for this small data set.
Um, and.
A set of actions, which are defined that become based on the diagram standard of, do you retain the this data element or do you remove it to shifted the section for dates, which are shifted, et cetera?
So this is what an answer key would look like again. This is a small 1, the 1 for the data set that I just described as much larger.
And this is an example of what happens when you at the results of comparing 2 algorithms and so I want to call your attention to these 2 things. These are.
What tca produced 1.
It modified these original values, and this is what the other vendor did. They're quite different.
That it's not the case that 1 is better than the other. The issue was that they are quite different.
And so these are the, the key differences that when we came with 2 pixels that had burned in.
Tci replaces those that removes the by blacking out those pixels where the other vendor just removed the images.
You can see differences in how institution names are handled, how patients size and weight were handled dates and times were shifted and or retained differently. And private data elements were all removed by the read process.
Where retains.
Private data elements, because we know that there are scientifically valuable information in, except for those, which we know contain.
So those are the differences that we identified, and the reason for showing you this is to make the point that.
If you go through the entire process, you end up with a result.
Similar to this, and you can then analyze that like this to see what the key differences are. And 1 hopes that if each of these algorithms has specified its icon profile, you could predict.
Okay is that what they should have done? Or is this something that.
Yeah, was not what you would expect. So, um.
In summary, we all know there are a number of tools for anonymization and curation that are. And this is a rapidly growing area.
We all want much more automation in this process,
but we need a way to validate the tools,
or at least if not validated at least,
compare them and understand what they're doing using synthetic data does based on real world experiences,
not just what the standard says,
but also,
the unusual cases that we have seen in real life.
We provided in with an initial data set that I described a 2nd, larger data set is currently being developed to add to that. 1st, 1, and then a small subset, or example, we've, we've published both on TCI and neutral scientific data.
And, as I've hopefully impressed upon your GDPR represents a whole new set of anonymization related issues that are quite different from those. We deal with with, with HIPPA and.
That's what I had to present and I'm happy to answer any questions.
Thank you Fred. Excellent presentation. Uh, there are a couple of comments or questions in the, uh, in the chat. Uh, Andre would you like to ask your question? Repeat your comment.
Oh, sure, yeah it's just a comment probably because.
I don't know if Fred can still share the definition. The slide with the definition of anonymization and what's the definition of state that I, I don't see how it could be.
Feasible to completely and minimize the data because.
I would think that given arbitrary patch of pixels from the image. If you have access to the packs, you should be able to.
Trace it back to the medical record, which should not be possible according to this definition.
Right.
Yeah, is the distinction upon the immunization, so it says irreversible removal of the link between the individual and his or her medical records. Sorry, Andreas, I was trying to get I was fumbling around. Tried to turn off mute. You're absolutely right. In fact, it's worse than that.
Uh, the EU has the death clause that at any time, any European.
Person can opt out and demand that their data not be used.
Given that clause, it's impossible to anonymize because you always have to maintain a link so that you can find their data. Right? So, basically the way GDPR is currently interpreted.
Anonymization is impossible. You're absolutely right.
That's what we're trying to convince the you that they need to adjust got it. Okay. Thank you.
Yoga has a question about the figure of merit for to evaluate the identification, I think, and automating. Um, so I think the next presentation.
Should partially address that if not completely.
And there's a question well, can I, can I just make a comment on 1 of the things that is important to realize under HIPPA if you.
The University of Arkansas hosts the cancer imaging archive on behalf of of NCI, but we are a covered entity.
So any data breach that was caused by data leaking through our processes, and being published on tca would be a data breach.
Held against our institution, so we have a 0 defects policy and we go to great lengths to.
Tripoli confirm everything, because we can't put our institution at risk. If you are not a covered entity, you don't necessarily have that level of requirement.
And I think there there may that has to be factored into any figure America in terms of how good the identification is.
Okay, thank you for um, there's a question about how much of the process is fully automated versus requiring manual intervention.
The manual intervention comes for the inspection part, so.
It's belt and suspenders. We do identify everything before it leaves the.
A submitting site, then we, we make sure that everything we do the curation make sure everything is in proper form diagram format.
We run all of our automated processes and then a human has to do final inspection to make sure that we didn't miss anything and it's that final inspection required by our 0 defects policy. That's time consuming.
We're.
Very interested in any automation that's available and we continue to automate our own procedures, but.
There still be time consuming manual steps of, of visual inspection of all of the image data.
So, the, the can image project, uh, 1 of the sub projects there is to look at trying to apply to try to, uh, to, uh.
Not fully automate those steps necessarily, because I don't think we'll ever be able to get away from the manual inspection part, but at least to be able to flag it sort of like computer aided detection of cancerous lesions.
The idea is computer aided detection of suspected and free text fields and so that, that's part of that you can image project.
Thank you Laurence. Uh, so in the interests of time, uh, like to invite the next presentation presentation.
Then, yes, uh, let me bring up the slides.
Thank you and I believe, uh, Dana, you can go ahead and start.
Yes, thank you. I'm just getting off mute here so thank you. Everyone for the opportunity to present here and thanks, Fred for that. Great segue.
I think,
obviously,
given all your work around the creation of synthetic data,
allow us to really hit the ground running with testing in this case,
specifically the Google Cloud healthcare API,
which aims to identify dotcom images specifically.
So we can go to the next slide.
I can give you further context before Ben dives into the meat of things not going to go into too much detail because much of this was covered by Fred and obviously this community knows that the,
the act of communication,
especially when it comes to dotcom images is a prerequisite for secondary data data sharing of any time,
and while this active data and if the unification,
or in some cases,
anonymization is not new,
what is a bit newer these days is the cloud based tools that are used to do this.
So, Google has relatively view counter API, which does this in a data fashion has certain components that had been used by a number of different groups someone at the same thing, but a somewhat different way.
And we'll our intent here was to work hands on with the Google Cloud API to understand. Does it work number 1 number 2?
How would 1 structure that within a larger engineering framework where you're trying to do an end to end solution for biomedical imaging?
And so much of the work that Ben will we'll get into is really aim that already describes our analysis of how well, this tool was able to work within a cloud environment using the synthetic data from. So we can go to the next slide.
Covered much of this already, but just to recap the healthcare guy itself was is meant to natively work with dotcom data.
So,
unlike other cloud based tools,
uh,
you know,
the dotcom format is not something that trips up this API at all and in doing much of our analysis and our evaluation of how well,
this is able to work what we had in mind was thinking about how this can be natively engineered to eventually incorporate other cloud based web methods that lend itself to greater image
recognition etc.
But for our 1st pass at this, we're really mainly concerned with identification and the removal of from these images to understanding are there any false positives? False negatives et cetera.
Go to the next slide. So this was our infrastructure that we set up again. I don't want to spend too much time here because I am antsy to get Ben to review our our findings from this.
But the goal, as I mentioned was to understand how could this be set up in the cloud natively so that you are decreasing the amount of dependencies from an engineering perspective, and then allowing you to get a better sense for.
Again, how effectively these kind of operations are able to manage the ingestion as well as the unification of of images specifically dotcom images.
So would that I'll turn things over to Ben and Ben you can get into our findings.
All right, I hope everyone can hear me.
Let's make sure. Okay. Okay. Yes. Thank you. Dana. No, I would like to just get into the meat of the workings of how the pipeline performs, uh, as well as how we ran it. Uh, just to begin with as a Fred.
Kindly went over all the data sets that we also received from tca and the formats they were in all dye com,
uh,
where we can see,
we had header data that we had to look out for as well as a burnt in pixels on the images as well,
so what we want to do is make sure that we can properly shift the dates.
Make sure that we remove any institution names, institutions, addresses as well as any references to people's names. Uh, for instance, in the study description, we can see where it has Douglas Davidson in here.
We recognize that as a person's name is able to remove it while keeping any other research critical tags, such as X Ray chest portable additionally on the image itself.
We would want to remove any, uh, any that's found there while also keeping, uh, anything that's also research critical, such as semi upright portable and.
Uh, the 1 data set the 1st day is that that we got as Fred explained contained, just over 181,800 images and that's the 1 that's currently published on.
And that's the 1 that we will mostly be talking about. Today. The other 2 data sets were meant to be used for validation, uh, as well as throughput analysis.
Uh, so moving on to the actual de de identification process, uh, in you have, uh, the healthcare API, where you can upload data to within data sets.
Uh, so you can create these data sets if you have access to, you have access to their cloud healthcare API. Uh, you can create multiple different types of data stores within these data sets and actually identify them all together.
But for this project itself, we're just concentrating on dialog. You also have consent forms, fire data, or a 7 data as well that you could that you could use. And then, as Fred mentioned.
Uh, we also are able to define what we are doing with the different tags and transformations. So we have the option, uh, within to specify if we want to keep specific tags.
If we want to remove specific tags, if we want to reset these tags are and what we're mostly interested in, where the takes place is in the actual clean text tags, where it searches specific tags.
Uh,
to actually go through,
especially anything that can contain a string and find any that is located within that specific tag,
uh,
we can also specify what type of transformations we want to do with in those tags,
uh,
such as we can replace with the infotype configuration.
Uh, we can shift all the dates so that we maintain that. So, if 1 date it for a patient has shifted by 10 days, uh, all the dates for that same patient can be shifted by the same amount.
Uh, and then Additionally, we can specify how we do these options by the tags themselves by either their tag name.
Uh, we can use their Hexa decimal ID or just by grouping multiple tags together by their value representation and over here to the right this isn't a complete a config that we use. Uh, but this is an example of how it would.
I would look in the platform.
Uh, so moving on to the results, uh, I will 1st, talk about this left image with the throughput. Uh, this is where we used, uh, over 14,000 images. Uh, specifically, the images are the dotcom slices themselves.
Uh, and this is a subset of the larger data set.
Uh,
this is for a 93 patients,
and when we specify in you can have your data set in a single region,
or multi region within the United States and clearly,
whenever you're using a,
just the single region here,
where you're using the Midwest region,
you get your best inconsistent results for how fast it takes to actually run the D.
D. identification process for us. We ran this over trial of 10.
Over 10 trials, and on average, uh, these images were de de, identified informants and 6 seconds.
Uh, so that's just an idea of how fast it takes, uh, how fast this, uh, process can run.
Uh, Additionally, now, for the accuracy, as was mentioned, also earlier, there were multiple actions that we were meant to take, uh.
From tag retain a retaining text making sure a text was not. No. Make sure we're removing a text.
That is most important because that's where the located additionally we want to make sure we shift dates and we change the proper a week to see that.
We always have s***, we properly shifted dates throughout the whole data set, and we also changed all the new ideas properly. The 1 concern here is this 1, a text removal that we missed and I will talk about that in later slides because obviously, where.
The shortfalls are are the most interesting. So I'll get into more of those details later.
But and, uh, overall we did a achieve a accuracy of approximately 99.8% when we had taken to all these, uh, all these actions into account.
So, I just want to go over what we saw on the actual images themselves.
So, for this benchmark, 1 of the image are a few of the images that have a data.
Uh, burnt into the image itself and here's an example of what we saw where we saw the name date and the date of birth.
All properly get removed and put a black box over it as a is recommended by, uh.
1, publishing to while at the same time.
Uh, this text was recognized, but it was recognized as not being. So it was not removed.
And so it correctly identified what was and removed it while keeping non data correctly retained.
Uh, we did experience a couple of false positives on images, uh, where the D identifier incorrectly identified.
And, for instance, on this image, it put a black box, right? In the middle of the image. For some reason it police that there's there uh, this is obviously a mistake.
Uh, and we identified it as as a mistake.
And we are a solution to fix this is where we include our human in the loop. So anywhere that text would be found.
Within a image, especially not on the corners where it could, uh, interfere with. Actually, uh, the understanding of the image should be sent to a human in the loop to recognize whether this is actual.
Okay, so now getting into the actual text results of in the actual header information.
Uh, where we found our 1 false negative.
Here we see where text was failed to be removed on 1 of the images and the header information. And this is a patient ID with found within a study description. Uh, the, I failed to recognize this and this was our, uh.
Our 1, main downfall of using this of using this platform.
Uh, additionally there were a couple of false positives. Uh, 1 was an actual serious, false positive where a software version was accidentally mistaken as an address. Uh.
Obviously, having having the correct software version can be very important.
But we also want to be able to recognize where addresses addresses are to remove them. But, in this case, this was a mistake.
Uh, additionally there was some unwanted text changes, for instance, in the largest image, pixel value, the Max number 6,500,535 was changed to a negative 1 these are technically equal to each other.
So, they're not counted as necessarily a failure as in the standard, changing it to a negative 1 is permitted.
Additionally, we had a couple of issues with the text, not in all fails and tag retained fails. However, if you look at the original values, these tags were were originally.
Not available or Nance, and so the system is not going to guess what that value should be. So it's going to just leave it as a blank or also leave it as Nan. And we see that. The same thing with the tag retained fails.
If the tag didn't already exists in the.
In the image, it's most likely not going to create it.
Uh, we could do some back end work where we would add in these tags if it was necessary. But right now, as.
The stand alone system that, uh, has that is running on. It doesn't do this.
Uh, and that brings us to our final conclusions here, uh, where we have shown that this, uh, healthcare API service, uh, is showing great promise as a viable option. There's still further testing.
That needs to be done. But if it's going, if data's going to be uploaded to the cloud and already, this could be a secondary identifier to make sure that everything was correctly found.
A couple of notes I want to mention is.
That what the tools that we use are, a lot of them are still in alpha and not currently available in a stable release. Uh, but.
There's, as we're told by Google, there's going to be further software changes that are meant to improve the system and improve the accuracy of the pipeline.
And these obviously all need to be tested before, uh, it's ever moved into the actual pipeline.
Additionally, we identified areas where we want to automate analysis, especially where pixels are removed. Uh, so that we can identify where there are false positives being found and we can send those 2.
Our recommendation of including the human in the loop to review the final images as well as the final header data.
And combining these efforts of a human expert in a,
the cloud native data,
the identification service,
uh,
can increase our accuracy of making sure that we are properly,
uh,
sharing data that has been Dia de identified while as well as speeding up this process.
Uh, so that is all for me, I want to acknowledge all the work that people have done on this.
And, uh, thank you very much for your time.
And inviting us here, we're happy to take any questions.
Thank you Dina and Ben. Excellent presentation.
Um, we have time for some questions obviously we can't go through all the.
Chad comments, but, um, if there are questions from the chat.
Postal place then please feel free to.
Ask a question, I think we have some if I may have not been answered in the chat yet. Uh.
1 of them is, um, on some of the images, so indicative of particular pheno of a particular phenotype that they are inherently P. H. I.
With their classification, the class of images that comes to mind immediately is any of the head that can contain a face and because there's lots of.
They did indicate that such phases are equivalent to photographs under HIPPA. So, um.
That's the easy 1. the harder ones are, are there patterns of brain folding patterns are.
From a chest radiograph, the red patterns that could be used as as.
Um, identifiers, biometric identifiers and you I think the.
You can go crazy with this and certainly the more information that you add along with the images, the easier it is to identify the person.
But there has to be some, if we're going to actually have data sharing, there has to be some reasonable legal standard. And I think that HIPPA so far has been.
Reasonably fair and providing such a legal standard.
Thank you Fred.
And not a question is.
Does this intelligent data set contained pixel data with burned in? Ph either is rotated if so, how well did the Google process handle this?
I'm going to have to ask Michael Rutherford or Larry Starbucks just on the other members of our team. Whether it actually.
Whether it did contain rotated because I don't remember.
Uh, neither do I have to admit we were very interested in this, but we didn't see any in the data set.
Uh, at least down on the answer key that we received.
So, another question that came through the chat is was the Google process using pause that, or some other tool.
I am unsure, uh, the 1 thing that I wanted to mention on the, uh, was when we sent the data through, uh, we none of the data we used was used to train any algorithms.
So, it, the algorithm that Google has was trained by them by themselves, and their own data elsewhere, and any data that we uploaded was our own, it was never used for.
Training or retraining an algorithm, thank you Ben. And, uh, the, the pause that tool is the tool that is used and has been developed by the.
University of Arkansas tca team.
Another question was, did you try.
Ultrasound with masking and I guess that's a question to Fred.
This status, the data set that we've been talking about, did not include ultrasound.
I believe the next data set does so that's a question for the next day to set, which we're still building.
Okay, David clone is asking what about hand written.
Burned in text I don't think we put any. In this case we have seen that actually in pathology images where you had to remove.
Handwritten text, but it's something to think about in future.
Data sets there is another question from David and David. I do not quite know how to interpret this. Maybe you can ask yourself please.
So if if I have a dotcom study and there's a brain or something like that, and there's 1100 slices.
And do you really need to run all 1100 slices or? Sorry? Oh, 1100 slices through or can you sort of sample?
Because, essentially, if you're just looking for burnt in things, I don't know how common it would be to have text on only 1 image in a series out of a 1000.
And eventually it would either get expensive or just a waste the CPU time to, you know, essentially look for the same thing. 1100 times.
I know David, you have to check every slice some radiologists and techs have the ability to annotate individual slices. So, given that 1 needs to check every.
That is good, but the guys have a way simplifying that they, they do a sort of collapse as a whole lot into 1 image. So you can review a 1000 slices all at once.
That Thank you, David, that was that was my answer as well.
So, um, it was nice to have a.
Cooperation from David Coney, who just spoke in these 2 projects and, uh, David, not to put you on the spot. But, uh, I.
Do want to invite you to make any comments overall about these projects or lessons learned.
Thanks for asking. Yeah, it was it was a pleasure to participate.
As a helper and some of these projects and see what everybody was doing 1 of the interesting aspects of the Google platform,
which people may not be aware of is as I understand it is based on their data loss,
prevention platform,
which is a text oriented tool.
Which can be used for many applications, not just medical imaging, and it has many configuration options, and as people found, depending on how you configure it, you get different results.
You can specify certain kind of regular, regular expressions to be used to match certain things. It's not just an artificial intelligence space toward it is quite configurable. And also, just as a question that I think David mentioned is.
You know, if or similar can be anywhere, you know, what do you do? Well, the answer is you look everywhere. So, regardless of whether an attribute is supposed to be removed or retained.
If it is text, you really need to clean it. You need to check it. And that's what the folks got 2 and Fred had a slide showing how certain attributes that you would never expect to be in still need to be checked.
So the approach that a lot of people are using is end of the Google platform is using is to be less concerned about what the tag nominally is better either. It's text to check it.
But of course, the text checking routines are not perfect. And sometimes they have false positives, false negatives.
Another thing that guys had mentioned, that was a fascinating aspect of their project was the fact that Google knows where their address is actual, as opposed to synthetic because it had a database of all addresses. So.
If you have a dummy address in your data, Google might not remove it because it knows it's a dummy.
Which is unexpected from testers perspective, but of course, doesn't influence real world behavior.
So those kind of 2nd order issues are really fascinating when you're using a general purpose tool to address and medically specific problem, and then trying to create a synthetic data set to, to address it.
So, that raises the interesting question is, can you create a synthetic data with real addresses given the fact that real addresses are in a sense by definition.
But David, you just scared me. It really proves that big brother really is watching.
Yes, well I'm going to assume that at some point, Google will know not only are we real addresses, but are we real people and it'll say, no, you can leave a made up name because we know they're not real.
In the case of addresses, I think it's.
It's easy to think that it's probably linked to Google Maps.
And can figure out that this is not a real address. I wonder if you could use a real address from 1 record with somebody else's record with that.
Be preaching privacy still. I have no idea. It's an interesting philosophical question, but if you just pick an address out of the phone book, if there were such a thing anymore, is it okay to use that as a certain?
I guess so. Because it's publicly available information.
Yeah, and these projects are very interesting. Me, I think we all learned a lot from them. And, of course, uh, these are not perfect tools as.
Been mentioned, they are they're in the alpha version, but, uh, it's nice to have the collaboration of the developers to address shortcomings.
Uh,
so I'm happy to report that we are moving both projects to the next phase and expanding on collection of more data and,
um,
try to fill in missing data as possible ultrasound and,
um,
and also hardening the tools and,
and the pipeline.
Um, so.
Please stay tuned in data, got any questions.
Yeah, there are a few more questions in the chat so 1 is how do we deal with private tags that encode information and non string format.
Yeah, it was just answering that 1. okay. What's in vendor? Specific data elements should be specified in the vendor's conformance statement. It's not always true. They're not always up to date.
So a lot of it is based on experience, and we actually have had the instance it wasn't in a private tech was actually in a standard tag where the tag was defined as an American.
But if you interpret it correctly, it was a patient's name.
And so you have to be very careful and check them carefully and not a question is how many seconds unneeded per image for a human to spot as possible issue.
For example handwritten text that needs. Yeah. How long does it take?
Do you know how long you're curation teams take to review those images Fred because of the tool that David described,
where we collapse a whole study and to essentially essentially a single image reactions all series inducing Lemme show that you can spot those differences quickly.
It only takes a few a few seconds to look at a series.
It takes longer if you actually find something, because then you have to go out and go in and figure out which image it's actually in. But.
They can go through a lot of data quickly. It's just, it's it's tedious and it's 1 of those things that we would to me. That's a low hanging fruit for trying to automate is we've combined. We've created this special image now.
How can we automate the search of that image?
Or large collection of those images for anything that looks like.
Well, also in looking at free checks fields and doing the double check, their pasta has tools to collapse everything down. So you're, you're not looking at thousands of duplicated things. You're looking at.
Just the unique phrases that have appeared in in a series or study.
Thank you there's another question for the Deloitte team, and I don't know if that is known. So, the question is, when you pass a study to the, does it perch after processing or does it retain.
Yeah, so currently we have it set up to retain it because we're doing this testing, but we have the option to purge it after it's done processing.
So,
we can just delete it straight from once once the data's uploaded to the cloud,
uh,
it becomes the identified that original data set can then be purged and that that can just be an automated process as soon as it's done did doing the application process.
Thank you Ben.
Questions I wanted to add 1 other option with a Google health care, and we haven't tested this out yet, but they have a streaming service where you can stream data to the cloud automatically.
And so it has never stored in the cloud.
Itself, but it's stream there and then automatically deleted after the after the notification.
Interesting. Okay. Um, would it be okay to move into the announcements?
For the announcements yes, we only have a couple minutes so.
Yeah, let's do that as fast as possible. Okay. Yeah. Uh, I try to be very quick about this and I'm trying to find my.
Point to share.
So, the most important announcement is the, uh.
And C, I imaging and Informatics community is going to take a summer hiatus. So there will be no calls in July, August and September.
Um, this decision was.
Made because the July and the September call both fall on holiday Mondays.
So, it wouldn't have made a lot of sense to shifted and mess up everybody's schedule but the webinar series will continue in October.
We will notify you about the topics that will come up in the full portion of the series.
To find the information, please go as always to our wiki page, which is we keep dot. Gov. And and you can also find the webinar recordings on this page.
So you can go back and listen to previous talks.
Came on, that's it for the announcements. Thank you so much and thanks for an excellent job. Um, in helping with this. I like to thank the presenters for excellent presentations and the audience within these.
For for nice library discussion and I'm.
Issue all the happy summer risks with summer and see you back and fall. Thank you so much for attending today.
Thank you thank you.