And, uh, K1, I'll hand it over to, you.
Okay, thank you. Good afternoon. Everybody I'm keep on for Honeywell with it and.
It is my pleasure today to have Dr David Clooney to present on this topic uh, the medical imaging, the identification.
Initiatives task group, um, for best practices and recommendations for image the identification, uh, just.
biograph little background, uh, the medical image, the identification initiative is something that we started when I joined in.
Uh, spring of 2020, and it's a number of several projects under the banner of media, including, uh, the,
the key project being developing a cloud based pipeline for the identification and development of data sets for validation of that pipeline, using synthetic data. using synthetic data
Injected into clinical images and these were provided by, uh.
Dr Fred pryor's team to us and in the past in this webinar, we presented on both media pipeline and media data sets. And, uh, today we will hear about a 3rd activity and under media and that's the task group.
Um, so without further do I turn it over to David?
Please.
Thanks, Kevin. I'm sharing my screen. Are you seeing it?
Yes, thanks.
So, thanks gave on for the introduction, and thank you very much for convening this task group, which has been very interesting project over the last almost 2 years now, I guess, and in this call,
I am planning on giving you a briefing on the report that the task group that cave on described is about to finalize.
We've been working on this for some time and we're hoping to have finished by the end of the year and distributed in early in the beginning of the next year.
So, uh.
Uh, this has been funded by NCI, of course.
And here is a summary of the mission and the goals in the charge to the task group.
So, what we were told to do, and our focus has been on the technical aspects of secondary sharing, public secondary sharing with an emphasis on dot com, and to document strategies and best practices,
to try to reach consensus on some of the best practices, or to identify areas where we didn't find any consensus 200 ninety's findings, both to the general public and. public and
In other words, as well as provide input to see that an, and other other centers.
2 aspects that we wanted to focus on, are to make recommendations on the performance evaluation of tools because as a description of the other activities has highlighted,
there are a number of ongoing projects that seek to enhance the scalability and increase the cycle time of the identification of activities.
And to that end, we want to evaluate automated as opposed to manual activities and look really look at the scalability of these things.
So who's in this task group? Um, I don't want to go through the list individually, but you can see it consists of some of the people who are on this call right now.
Um, some of the usual suspects as well as some new players, and some people drawn specifically from key areas, where we lacked expertise, for example, in whole slide imaging or in radio therapy.
And as you can see, it represents a fairly good sample of the people mostly in academia who are interested in this project.
So our scope was human subjects in bio specimens, medical images and the goal, as I mentioned before is to reduce the re, identification risks sufficiently for unrestricted public hearing for any purpose commercial,
or non commercial and for secondary raise some well, beyond the primary reason for the acquisition of the images, and we wanted to cover, uh, to the extent we could all medical images.
Obviously our key focuses on cross sectional radiology as we use some.
Typically, in cancer, but to also include in a clinical pathology in particular as well as external photography and a number of other areas.
Uh, both because it's important than the cancer application as well, as of general interest to the, uh, the other institutions centers.
We also included non image objects that are related to structure sets and so on structured reports, the presentation States.
Typically stuff that's acquired in the image realm, but are not images themselves and particularly, but not only to focus on on dot com.
The deliverable is in the form of a report with best practices, which are what you should be doing now and recommendations for further research and it's quite a long report attempts to be fairly comprehensive in order to support the best practices and recommendations.
It's about 80 pages. And many references, and it has a best practices and make recommendations and in this short call can only touch on some of these that are primary interest.
The methodology we used was, um, typical, we did an extensive literature review to support our preconceived notions and or to change them.
Uh, it's not a systemic systematic review.
It was much more informal than that, but it is fairly comprehensive, and we wanted to discuss areas of concern or things that were novel or nascent and so we had monthly calls over extended period of time.
We looked at traditional methods, we focused on brand and text. We looked at faces. We looked at the threat models and attempts to quantify re, identification risk, which sort of led us into.
Greater understanding of the so called statistical disclosure control community of which many of us were frankly unaware or unconcerned and learned some lessons from what's referred to as the micro data community.
This is this sort of like the people who do censuses and other other types of activity, which, which this.
Uh, disseminate both aggregate information, which is not really at risk for re, identification, but also micro data, which means individual person level records. And as you can imagine, they have a vast experience in that area.
Now, we also, of course, looked at potential automated approaches, some of which included AI and some of which overlapped with some of the other activities that keva mentioned.
So, the reports content covers a fairly broad range of subjects. I'm not going to go through every 1 of the bullets on this slide.
Nor, I'm going to cover every best practice or recommendation and the subsequent discussion, but just some of the highlights we had to get our terminology straight.
So, in the US we tend to refer to the identification in Europe, they tend to refer to anonymization and there's uncertainty about the various overlap of the various terms in the various different jurisdictions.
Tried to clarify that and being fairly U. S. centric we, we picked the terminology, we were comfortable with different file formats.
Not just com, what needs to be identified in those files rule based approaches and where there were standards for rule based approaches FTC stuff.
I've already mentioned that took up a considerable amount of time to consider all the different types of areas. You might find offending content, be it in structured data elements where things are what they say they are, or unstructured.
Data elements that are essentially plain text carried along possibly with instructor text, possibly within, um, reports that may accompany the images stuff that's burned into the images dates and times.
What significance did they have as you're I'm sure all well where the community has been studying, how much information can be derived from images, including, you know, your age sex and race um.
And how does that factor in now to our, uh, addressing indirect identifiers as such whether it be derived from the image, or from structured data elements metadata looking in obscure places we'll come back to,
that we looked at modality specific issues. Um, radiology has issues radiotherapy has specific issues, whole slide imaging, of course, as specific issues, which we address in the report.
We looked into has used for the identification, not just as a customer for the identified data and so on.
So, as far as the best practices are concerned, um, best practices is a much bandied about term. But, essentially, it means, you know, what you should be doing, or what the community feels you should be doing, uh, which may, or may not differ from what you actually do.
And so our number 1 best practice focuses on the notion that you should.
Take care to de, identify everything, and that if you retain certain kinds of information, you need to quantify the risk. Um, and so both direct identifiers and indirect identifiers.
And the report goes into the description of what those mean, sensitive information in all collection descriptions.
Um, and supporting data, so not just the dot com header not just the burden text and the images, but also any accompanying spreadsheets or clinical data or data in other formats and data linked from other places. And we'll come back to that also.
So, overlays presentation States and so on and so forth. May all be places that you need to pay attention to.
Uh, and if you decide to retain indirect identifiers, things like age sex race, um, then, um.
You have to make a determination, you know, are satisfying HIPPA, privacy, rule, safe harbor requirements, sufficient or do you need to think more about this address? The potential uniqueness?
The potential issues for re, identification risk, and we recommend that you quantify the re, identification risk, or at the very worst.
At least document a risk assessment with respect to what you're retaining in this respect this is, to some extent pushing the limits of the state of the art and raising the bar.
But we felt this is important enough and not widely known enough to raise to the level of, including in the best practice and we can talk about that in the question and answer session afterwards. If you want.
Another best practice, and I'm not going through these sequentially. I'm not covering them. All obviously is to remain compliant. Don't screw up the doc on file in the process of identification and many poor quality. The identifiers do this.
This is something that's extremely important in terms of preserving the utility downstream.
on a subject preserving the utility you don't know what secondary uses this data may be used for there may be all sorts of anticipated uses for which information that you might have not considered relevant to your primary use case is useful downstream
and its absence may completely undermine the utilities so provide the preserve as much information about the image acquisition as possible to maximize that we use potential
Except, to the extent that you need to consider any potential residual identification risk that retaining this information contains so be thoughtful. But at the same time, make the effort to preserve as much as you can.
Use the standard profile, a lot of people who know a lot about dot com and the identification. They've put a lot of effort into standardizing the approach to the identification. So, use it if you.
Try to do this from scratch.
There's a high probability that there will be use cases that you fail to consider at the very least use the dot com standard for this as a reference and make carefully considered decisions with respect to additional information you may retain
or additional information that you may remove. Don't forget to consider the private data elements in this respect and, and there are sources of knowledge about which private data elements are safer unsafe. We'll go back to come back to this later in the.
Um, but remove anything that is known to be unsafe, retain only what is known to be safe. Uh, there are options in the dot com, profile for various different scenarios beyond the baseline, and consider those.
But in choosing those options carefully, balance the preservation of utility against the residual identification risk.
For those who haven't seen the doc on profile on the subject, it's, it's a very long table several tables for their doc on data elements and dot com structured report content. I want to go into detail, but you get an idea of what it's all about.
Non dot com images. There are non Docker images around and you should use the same approach that is described in dot com to apply to them. So, for, for images that are stored in doc on drive form.
It's like bids or something where the metadata is just an arbitrary translation of dot com, data elements into a different format, use the same principles, or at least extend those principles for data elements that don't exactly match for clinical data elements.
Um, stuff that's in text stuff. That's.
There are other standards, the farmer community, the farmer research community in the commercial world, aligned with the people, clinical data interchange standards, consortium, study, data, tabulation model.
They have developed their own identification standard and that's well worth reviewing if you're going to look at clinical data elements that are beyond what might be described in the doc on the standard.
Everything all data elements anywhere, and this means not only the material that you have control over, but anything that is linked.
So, in our case, for example, in imaging data Commons, we have shared the identifiers with generic data Commons and proteomic. Data columns and other potential data comments. All of that data is public. All of that data is available to a potential attacker.
So when you're quantifying your re, identification risk, you need to consider all of that data.
Uh, even if it's beyond your control, it's available, and therefore it needs to be considered in your risk analysis. You may choose not to include the patient's age but if the patient's age is readily available from GDC.
Then you have to consider it in your risk analysis and furthermore it behooves you to look for it. Don't just assume that it isn't there.
We believe that it is a good practice to to look for any potential link data.
Burned in text, uh, obviously an important topic, particularly when you're looking at screenshots or prediction radiography not just reconstructed or pet images.
Uh, these are synthesized a, obviously illustrating the application of an automated pipeline to find and automatically redact that. But sometimes these tools have failures and so here you have some part of the patient's pelvis that's been inadvertently redacted.
So, the state of the art is not yet.
That we don't need human QC on these kinds of tools, but they're increasingly useful and their application means that you will find more images that you can share whole slide.
Imaging has specific concerns in this respect the label image in particular may contain. And there had been documented cases where has leaked in this manner and it's not just a question of names as you can see on the left side of the screen.
There is a date and you need to satisfy yourself that the state is harmless. And it's not a patient related date.
But rather some arbitrary data scanning or replacement of the label that occurred in a, in a data coordinating center,
downstream you need to know your material and know what's been done to it by people whose hands has passed through before it got to you.
This also illustrates the prospect of containing handwritten information on a slide that is scanned in the overview image.
So, Brandon text, all sorts of strange sources of Brandon text. Even jewelry, for example, is run summers, identified in his chest X, Ray collection.
This needs to be detected, redacted or the images discarded in some cases. For example, screenshots in a series. May not be directly applicable. And may not even be useful, but they often contain patient's name in the corner.
You might be able to discard that. On the other hand, a chest X Ray, that's fundamental to the.
Utility of the dataset, you may need to redact it. And furthermore any redaction efforts should be subject to human review. If they've been performed by a machine and review by a 2nd person.
If they've been performed by a human in the past, we've tended to limit our checks for sending information to a stratified subset of images, you know, things that say, screenshot in there.
Uh, description, that's probably not sufficient anymore. So we recommend that you consider the risk of this, and the prospect of needing to review every image in some.
Efficient manner private data elements we've talked about some of them are discussed in dot com directly.
The ones the manufacturers have contributed, but other sources of the safety of private data elements are available, the Arkansas people, or the or the, um,
people have a disseminate a large carefully curated set of private data elements with information in this respect. But if you don't know, it's safe, private data elements have to go, even though we know that the utility of a dataset.
They will hinge on some important private data elements.
Obscure metadata, you would be amazed where metadata can hide, even in a image from a consumer camera, or through some other means may contain metadata.
Um, because many of these formats are not designed for metadata. Some of them are extended to include additional metadata support is extensible.
Um, just as many other formats are, and so these so called or application marker segments may contain metadata and either a structure or.
Structured form, and generally they can be removed without compromising the utility of the image.
They may contain useful information that might be externalized in some other manner like, access information copied into the dot com, header and discrete data elements, but don't ignore the possibility that compressed. The bitstreams can contain metadata.
Faces, um, obviously, these have been concerning for many, many, many years, the level of concern has been increasing with the increasing availability and more reliable facial recognition techniques.
And so these have written to the level of a concern, and the need to disseminate the best practice for potentially reconstructive facial information in the report.
We address both external photographs that are old faces as well as reconstructed.
Pmr and pet images that contain facial information that may, or may not support human or facial recognition for re identification. The literature is.
Uh, uh, improving in this respect there are many recent studies in these areas. Um, some of which, um.
how should i say this sort of have a chicken little sky is falling you can never share another image of the face again others are more balanced in terms of their statistical analysis of the actual re identification risk and we'll come to this in the recommendations
but we think that it is the level now where if you're going to retain faces you need to quantify the risk and if you can't or to quantify the risk then you need to remove facial features in some manner
Hopefully, that doesn't completely undermine the utility of the images otherwise not share them.
Quality control I've alluded to before.
We all know that machines and people are available and having someone else downstream or some other system downstream make another check never does any harm, but it has a finite cost attached to it. But the identification at scale is an industrial process.
And just as all industrial processes have quality control.
Percent of images, or percent of activities reviewed based upon the risk of failing to review them or a failure.
You need to include this in your process, and you need to document the risk analysis that leads you to choose a certain percentage of re, identification of quality control. I should say and failing that.
If you don't perform QC, you should do a risk assessment justify. Why you choose not to do QC and the QC process may address very different types of image, or different parts of the process that are perceived to have greater or.
That's a risk, um.
Uh, but the bottom line is, do QC and documents are QC on the subject of documentation. Uh, as I say, in regulated industries, if it isn't documented, it wasn't done.
So it is extraordinarily important for you to document your process.
So that people can confidently use your data in the knowledge that it is safe to use and safe to distribute and safe to include in their research.
It shouldn't this documentation should be shared it should be open. It should be published with the data collection since we are talking here about public data collections.
So that's a sort of brief overview of some of the best practices that we addressed and some of the perhaps more topical or controversial ones. But what about the future? What are we going to do going forward? And what recommendations do we have?
So, 1 of the areas that we were tasked to investigate,
but found little information on is the question of how do you quantify the performance and in terms of quantifying the performance I'm talking in this recommendation about the reliability.
Of the process and not so much the re, identification risk. So you can consider quantification of 2 things. 1, what is the re, identification risk and their statistical methods, which have their pros and cons and some are more conservative and some are less conservative in this regard.
But here, I'm talking about the quantification of the actual technical process, did it do what you wanted it to do?
Because not every system is perfect and we have often been challenged in producing tools that do this sort of thing is.
Well, how good is it when you, when you finish running your tool, did it work?
Did it remove everything we wanted and as we explore machine learning driven approaches to D identification, this qualification has become even more important you know, is there a score that says, you know, we've D,
identified 99.5% successfully or whatever and then the question would arise? Well, how successful do you have to be. you have to be
To be considered successful we understand that this is an area of active research. There are various different proposals in this respect, based on various different methods.
It's sometimes hard to relate the relative importance of various different successes and failures.
So we think this is an area that requires further active research to assist both consumers selecting a process to use in their production environment as well as a comparison of different potential processes. And.
Even to the extent of having some kind of competition or challenge to compare the performance of different processes to drive innovation in this area.
What about the actual risk of phases if I sounded skeptical when I was talking about re,
identification of faces this is because most of the recent literature has done small experiments that have been been extrapolated to the real world without actually,
having much in the way of evidence that this is a real world risk, and we'll know that there have been no successful facial.
Uh, reconstructive face, the identification attacks, conducted to this date, not to say that there couldn't be a successful attack in the future.
Um, but there seems to be a paucity of information on the real realistic level of risk from this recognition.
Potential and so we think there's a need to conduct further research and to realistically assess the need for restricted access versus public sharing or skill stripping or defacing versus public sharing without
adulterated pixel data. So that we can prevent an excessive ad reaction to this risk diminishing.
The utility of or limiting access to images, particularly for head and neck cancer applications.
Your therapy applications in that setting, um, radio therapy data sets that are not images, you know, definitely has friends and so on, you know, can they, um.
be quantified in terms of what their actual level of risk here is and we just didn't feel that there was enough material in the literature to specify an approach the other thing that's not included in this recommendation is the need to then include that risk assessment along with all the indirect identifiers you
might include so we want to try to find a framework where you can
Include reconstructed the faces as just, you know, potentially get another indirect identifier and attempt to include them in the same quantification of risk framework as the indirect identifier are. If that makes sense to, you.
So, uh, this is just a taste if you want more information, there is a lot more in the report, as I said, and we are hoping to get it out there by the end of the year.
Um, or at least get finished by the end of the year, I should say it is going through external review at the moment feedback is due by the middle of this month.
Although we will, of course, accept comments that come in later, we hope to complete the report by the end of the year, and start disseminating at, at the beginning of the new year. It's far too large to distribute as an academic paper.
So, it will be some kind of preprint or something, but we will write an academic paper serving as an executive summary so that it can be promulgated in the literature and reference appropriately.
And then hopefully advertised widely either to serve as a useful template or to solicit.
Feedback on issues where people disagree with our findings, uh, also plans and he can talk to this better than I to hold a workshop, which will include discussion of the subject as well as others.
Exact time and date to be determined. But, hopefully, we'll see some of you there and with that, I will finish hopefully leading enough time for questions.
Thank you David. It was a great presentation. I'll just add that.
This working group, or this task group also has a steering committee, which is composed of our colleagues across NIH from.
Different programs, uh, in several institutes, including an.
And and and, uh, and and other Institutes that have been have interest in, uh, imaging or medical imaging and image the identification.
And this report was presented to them, um, just before Thanksgiving holiday.
As David mentioned, we'd like to plan, or we are starting to plan for a workshop, uh, in late spring of next year and we'd like to invite industry and to developers to that workshop.
We're also thinking about planning it, um, a challenge or open benchmark to allow 2 developers to benchmark their tools against reference data sets for.
Image the identification, and that will be announced later with that. I would like to open the floor for questions.
Please Hi, this is Fred prayer. David. I know you and I have discussed this, but maybe you could for the rest of the group comment on. How do we operationalize the.
Statistical analysis approach to quantify risk in say, or, or any other repository.
Uh, yes, this is where there is a bit of a gap in terms of tooling because we're just beginning to understand this area. So, um, there are quite a few tools available. Um, uh, that are described in the report, uh, that.
Are used by the experts, um, statisticians who specialized in this field that use various different models and various different statistical techniques, to quantify the risk given a set of identifiers.
Um, as many of, you know, I'm sure it's all a question of uniqueness uniqueness in the population or the sample and how that then relates to information that can be matched up with it outside.
So there are there are a whole bunch of statistical models ranging from things like.
The more sophisticated concepts that take into account different factors. Uh.
Indirect identifiers, like age, sex, race, diagnosis, those kinds of things, whatever you.
Choose as a potential indirect identifier and that in itself is an art, um, can they can be extracted from the doc on header or other sources of information like a company spreadsheets.
Fed into these tools, and they will spit out a risk. You can then attempt to mitigate that risk. If it's too high above some predetermined threshold by applying techniques to reduce the uniqueness.
So, for example, bending ages into greater categories along the same lines as the credit. And 89 age, uh. uh
statement in the hippa privacy rule safe harbor eighteen elements set and then having done that you need to be able to feed that information back into the means by which you discriminate your images either update the spreadsheets that you're
going to share it with the images or indeed to update the dot com header and that's probably where the tooling is lacking it's easy we all have tools to get information out of competitors but we don't necessarily have tools to conveniently in the course of the identification
Pipeline feed that information back into the dotcom header when it has come out of 1 of these statistical disclosure control tools that I'm referring to. So operationalize statistical disclosure control methodology.
Is 1 aspect another aspect is the purely human aspect, and that is, do we have the knowledge to use these tools to which the short answer I think is no and the task group I think in doing some experiments during 1 of our calls.
You know, we, we realize that this is not our area of expertise. Um.
You know, the other method of the identification identified in the HIPPA privacy rule was the so called expert determination method, which basically means hire a statistician who knows what they're talking about.
And that is an area where we may need to revisit our current state of the art to determine whether it's satisfactory who we should consider hiring. And what criteria those sorts of people should meet.
And, of course, we can take care of the technical aspects of the tooling.
Uh, ourselves and we also need to consider it necessary or is it worth it? There is a body of literature again reference from the report on the adequacy.
Of the HIPPA, privacy rule, 18 elements, safe harbor that looks at this. From a statistical perspective. A lot of experiments have shown that that still remains adequate today. Other people would debate that, like any community.
No. 2 experts will agree. So some experts you might hire will say, no, you can't do that. Others will say, yes, you can't do that or whatever. So it's just an area that we, I think the imaging community has not adequately explored.
Board yet, and so, as you imply in your question operationalize this is both technically and resource wise, somewhat challenging.
Does that we have to fully answer your question for it or not quite? Yes. Thank you. That's I, I wanted you to just review those points with everyone.
That's the time yeah. Has raised his hand.
Okay, so given that we can pretty much predict age and sex from any modality. Nowadays. Would those be considered as known? Always? Ah, well, that's a very that's a very interesting question.
And we did look at that in the report, and agent 6 can be estimated from some kinds of images with a certain degree of precision and or accuracy.
And the question is not a.
May be derived, but Robert, can they be derived with sufficient.
Accuracy and precision to then lead to successful re, identification.
So 1 of the techniques 1 uses for age, of course, is beginning, as I mentioned, and the state of the art, an age extraction from brains and chest X rays and similar seems to be around the sort of plus or -4 to 5 year. Um. um
Range, I hope I'm quoting that accurately, which is the sort of level of bending that 1 typically does when 1 is causing ages to prevent re, identification in a statistical setting.
So so it may be that they're not.
So, precise or so accurate that they're a concern yet.
If that answers your question, eh? Yes then it's sort of it, it's your numbers. You're, we're quoting and match my personal experience. So we're plus -5. so I guess we're like the rest of it like everybody else. else
So, thanks yes, now, sex sex is probably different in that respect, but depending on what kind of image, I mean, you can, you can figure out someone 6 from a brain image with a fairly high degree of.
Accuracy for 1 reason or another, but you might not be able to predict it from some other kind of image of some other body part. Obviously if you've included, you can predict with a very high degree of accuracy.
So so sex is obviously something that we we.
We can't even attempt to prevent disclosure off with certain kind of images.
Andre posted the question in the chat he wants to know how many public repositories.
All the experience of how many publicly repository Tories was considered about putting together the report. The short answer is none in the sense that we didn't go looking for specific public repositories.
Um, uh, on the other hand, we reviewed.
Or publications, we could find that mentioned the subject of the identification, either in an image, or other image con, non imaging context.
Um, many of the people involved in part in the task group are involved in the maintenance of various different public repositories. So so indirectly we hope to, uh.
Given this area broad coverage, but we didn't specifically go looking at any particular public repository.
Does that answer your question under? Yes, thank you. David.
David, I have a question, so we didn't really look into encryption as a way to the identify.
And would you like to share your thoughts on that?
Well, encryption well, actually, I'm not sure we mentioned this in the report so.
1 can certainly retain encrypted.
Identifiers in a data set that you are distributing with the understanding that.
People who don't have the decryption key won't be able to identify the data.
3 decryption, whereas people who do have the decryption key will and that's a means of.
Allowing authorized people to subsequently re, identified, for example, to associate with information from a different source downstream. It fits into this sort of optimization world as a technique.
I'm, as I say, I'm not sure we discuss it in the report, but that's a very dangerous approach because all encryptions games are eventually broken putting it.
Apparently, so so if this archive data is shared publicly, unless forever, then someone eventually will be able to identify any such key.
So, personally, I would never include an encrypted true identifier in a publicly shared data set as a sort of matter of principle and find some other way to communicate that. If there was an appropriate reason to communicate that. But I'm not sure if that's what you're referring to.
That is, I just wanted to to talk about it a little bit. Just right. We, we do talk in the report about hashes. So some people recreate their instance and so on by using a hash mechanism.
And the question is, could somebody reconstruct the input information to that hash function? Now we all know that there are many has functions like that is supposedly unquote broken but what that actually means doesn't necessarily.
Translate into reversal of it under all circumstances. So broken in a meaningful way is what we need to consider there. So.
The report does not deprecate these of hashing hashing remains a very common practice and we didn't really pursue that in a lot of depth because, you know,
some people are comfortable with hashing and other people would prefer to use different techniques. And that's sort of where it was at the moment.
Okay, thanks other comments questions for.
David.
Maybe if I since it was quiet, I could ask 1 more. Um, I know we distributed the white paper to colleagues in Europe for a GDPR perspective. Has there been any feedback on that?
Not yet, but we certainly hope to get that because, um.
You know, GDPR has all sorts of other concerns related to it most of, which are not related to the technical aspects of what we're doing to be honest.
Um, and and that's something we should probably make clear is this paper is not about policy related issues.
It's about technical issues, so whether or not, you should say things, whether you have adequate consent to share things, things of a certain type and so on and so forth. Those are not in the report. The report's purely concerned about the.
Technical matter of given a dataset with images. What is the re, identification risk? And how do you mitigate that to a satisfactory level to publicly share a period end?
So, in other words, in the identified data set in the United States, and the identified data set in Europe would essentially be the same, except whether you're allowed to distribute or not might be different.
Did I say that correctly? I'm not sure who you're asking, but I think so. Yeah, I meant to Fred. Yep.
Yep, because you're the 1 who has a lot of experience with the European D identification.
So has a question. Yeah, I'm I'm intrigued by the.
Tools available to do statistical analysis for those that haven't done statistical analysis.
Does the report recommend tools.
What's your? It doesn't recommend tools, but it documents 2 of the well known ones. And as you probably got.
An impression from my spiel, you know, we don't consider ourselves experts in this area so we sold outside expert advice. So there are various people who really know a lot about this subject and various people who have.
Uh, evangelize, if that's the right word, the role of statistical disclosure control in a health care, setting published,
widely written books lectured and who volunteered their time to explain the principles to us.
And so hopefully we've done justice to their explanation and their training, and we have referenced quite a bit of material that you can go look at. And.
When asked the question, you know, what tools are available, they identified 2 tools in particular 1 of which is familiar to the people in the our community as, you know, are as a commonly used statistical tool.
And the other, which is a, is a more graphical user interface, based tool that we played around with just to get a sense of how this worked and both of those tools are are widely used, publicly, available well, supported as far as we know.
And so that's where you can go.
Of course, neither of those tells knows anything about images. They just dealing with tables and data both as important as output.
Thank you David. Okay, thank you.
David for for great presentation and discussion and for leading this task group, over the past year and a half. Thanks, Dave on. And my 1 plan would be anybody who has a specific interest is welcome to join as an external reviewer and just sent me an email.
I had an email on slide. You can find easily enough and we'll be happy to.
Use of a list of people to read our monster document.
Great, thank you. I have the questions Kirk Smith display. Excuse me on the statistical analysis?
Does that approach consider both the description and the value at the same time or how does that work? No, and that's a very interesting point. So, uh.
As far as these tools are concerned, um, it knows nothing about the variable. So you give it a a table that has a column header. That is the variable name for referencing and it has values.
The only thing the tools look at is the uniqueness.
In the sample of of what you have supplied it compared to whatever statistical modals, it's, it's, it's trying to.
Applied to the task at hand it doesn't know that a column called age is age or sex sex and it doesn't treat them any differently, you know, any independent indirect, um.
Identify as supplied is the same as any other.
At least so far as I've been able to ascertain, which is an interesting question, because we might wait some of them differently under different circumstances. There there are.
Ways in which you can supply weights, um, but it's up to you to judge what weight to give them and so on and so forth. So, does that answer your question?
Yes, it does. Thank you. Thanks.
Uh, David, Fred asked the question if your slides will.
Be made available. Absolutely. Yep. Yep. I sent them to Ali who will put them in the usual place give them to Carolyn or whatever, and they will be made available. Yes.
Okay, so, for everybody who is interested, I posted scroll up in the chat and there is a link to the NCI wiki page and that's where I'm going to post a recording as well as the slides.
But probably not today.
Great, thank you. Thanks, David. And I think with that, we can move to the next.
Section of the webinar. Thanks again, David for great presentation.
And the next section of the webinar is me giving a very brief update.
About the imaging data comments.
And again, the imaging data comments team is on the call. So if you have more questions about IDC, please feel free to ask, I just wanted to point out, uh, a few changes to the interface of the imaging data comments.
The team has tried to make the.
Way how users find information about IDC, and how they get help on a little bit more obvious. So they changed, uh, the labeling of the menu items. The menu items were there before we just renamed them.
So you will now see, uh, user forum link. And if you have a question for the team, if you want to reach out to the to them, or anybody else in the community, because as, you know, the community is, uh.
Table resource for all users, you can just click on user forum on this link and then go to the next page.
And create a new topic, and it will be a broadcast to the team as well as to other users.
The other relabeling took place here, the link is now called help and when you click on the help link.
You will be, uh, brought to the documentation page on on the very 1st page. You will find a link.
To the support email.
And you can send your question to this email address, this email address.
Was always there, it was just in the footer of the page. We are trying to make it more prominent and again, from here, you can also get a link back to the IDC user forum. The, uh, 2nd.
Way or the 3rd way to get help on is by just attending the IDC community office hours. And as you can see, there is 1 set on Tuesday afternoon and 1 said on Wednesday morning.
Ah, feel free to join Andre is there he will, uh, ask your question.
Uh, he will answer your questions and, of course, help will always bring you to the documentation where you get information about itself. The data on dot com.
The portal, the APIs that are available to you, and all kinds of notebooks that the team has developed and Andre, you just posted something in the chat. Why don't you, uh.
Tell everybody about this yes. Yes, sure. I posted the link to the tutorial that we presented on last week.
We had good attendance, and I hope that this tutorial can be helpful to get started with.
It's really at the beginner level tutorial that helps you set up the prerequisites in order to use Google cloud in conjunction with data and then guide you over the structure of metadata tables how to
query this how to build cohorts, how to visualize images how to get information about licenses and how to download images to either.
The cloud, virtual machine, or to your computer, and in order to complete those operations or the tutorial.
You do not need any credits. You don't need the credit card. All of these operations are completely free, including aggressive data out of the cloud.
Thank you, I'm doing any questions, please, as we mentioned, either sent email to this support email or post a question on the user form. Thank you.
Sure, thank you. Andre.
Um, Justin, uh, I believe you said, you would prefer to, uh, give a update on the next call.
Is that still the case? Yes, that's a case. Yeah. Yeah. Okay.
Okay, thank you Kevin. If it's okay with you, then we can move on to the announcements.
Sure, please, um, as always the invitation to the webinar gets distributed.
We are the Google group, but of course, we are trying to reach a broader audience by with the help of the secret communications office and by using the E,
mailing list and the email list also, tweets went out. Um, this time as we basically have done for the for the for, for 2022, all information about the webinar can be found on the wiki.
Same link. the wiki same link
I just posted in the chat earlier, and we also have posted all the recordings and I don't know if it was mentioned during last month's call, which I unfortunately was not able to attend.
Um, Carolyn has done a tremendous amount of work of bringing in, uh, presentations and recordings from.
Previous year or so, the record keeping now goes back a full decade back to 2012,
and you are welcome to look at these slides or listen to these recordings and it just shows you how this webinar has evolved. webinar has evolved
Um, our next call.
Would be on January 2nd, 2023, but because it's a holiday, we're going to cancel this and we'll continue the webinar series on February 6th. Uh, Dr Michael will present.
We are still working with him to finalize the topic of his presentation. topic of his presentation
And since this is the last call of 2022, I just want to wish you happy holidays and a happy New year.
Great Thank you. Thank you. And, uh, with that, I'd like to bring this webinar to a close and give everybody 7 minutes back. I would like to thank you.
Presenter of course, and all the attendees and Lee, thank you have a safe.
I'm happy holiday. Bye. Bye. Bye. Bye. Thank you. Bye.