Page History
Anchor | ||||
---|---|---|---|---|
|
Date
Attendees
Name | Attended |
---|---|
Benjamin Bearce | Yes |
Carolyn Klinger | Yes |
Jayashree Kalpathy-Cramer | No |
Keyvan Farahani | No |
Linmin Pei | Yes |
Ulli Wagner | No |
Currently supported challenges
Name of Challenge | Organization / Meeting | Medici instance | Release | Release Validation data | Release | Number of | Closing Date | Will the challenge remain open? | Comments |
---|---|---|---|---|---|---|---|---|---|
DL-sparse challenge #2 | Ben is talking with Emil about using MedICI again for another DL-Sparse challenge | ||||||||
SIRPRISE Reconstruction Challenge | https://www.aapm.org/GrandChallenge/SIRPRISE/ | https://medici-codalab-main.eastus.cloudapp.azure.com/ | Submitters will upload images to Medici. Did decide not to use FTP transfer but upload zipped images directly to Codalab. It was found out that the data size is not as large as anticipated. | ||||||
Truth-based CT reconstruction challenge | https://medici-codalab-main.eastus.cloudapp.azure.com/ | submitters will upload images to Dropbox, organizers will provide the scoring. Ben will create entries for each participants and enter the scores so that they can be shown on the leaderboard. | |||||||
Breast density FL | https://medici-codalab-main.eastus.cloudapp.azure.com/ |
|
|
| The submitter will upload the wrapped Docker image to the Codalab platform. Ben will execute the algorithms and post the performances on the leaderboard. | ||||
MIDRC COVIDx Challenge | https://midrc-medici-competitions.eastus.cloudapp.azure.com/competitions/7 or |
|
|
| The submitter will upload the wrapped Docker image to the Codalab platform. Ben will execute the algorithms and post the performances on the leaderboard. |
Previously supported challenges that remain open
Name of Challenge | Organization / Meeting | Medici instance | Release | Release Validation data | Release | Number of | Closing Date | Will the challenge remain open? | Comments |
---|---|---|---|---|---|---|---|---|---|
Tasks / Deliverables
Task | Due Date | Status |
---|---|---|
Supporting at least three challenges including one federated learning challenge |
| Three challenges are DBTex2, SIRPRISE, and Truth-based CT reconstruction challenge. |
Support organizers and participants (help desk, FAQ, training, presentations) |
| Ongoing |
Maintain and update the website |
| Ongoing |
Capturing and providing metrics on the use of the MedICI website |
| Ongoing |
Capturing and providing metrics on the challenges |
| Ongoing |
Support at least three previously executed challenges |
| Done |
MedICI on the NCI Cloud Two (GCP) and Firecloud |
| Task was suspended per discussion on |
Operations and Maintenance |
| Ongoing |
Changes to MedICI to support federated learning challenges |
| See Google doc |
Support Codalab / Codabench open source development |
| Ongoing |
Discussion items
Date | Item | Notes |
---|---|---|
|
|
| ||
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
| https://www.medici-challenges.org/ https://medici-codalab-main.eastus.cloudapp.azure.com
| |
|
| |
| Meeting canceled, Ulli out of the office | |
| Moving the contract to UCD
Review of MSR draft for May 2022 | |
| Discussed contract issues, move to University of Colorado | |
|
| |
| Updated status of SOW tasks Ben is supporting challenge organizers & participants MIDRC test challenge will start soon including Docker submissions. Supporting Codalab open source project. | |
| SIRPRISE - active, users can download data. Challenge organizers will establish the scores and Ben will enter them at the end of the challenge. Scores will not be updated in real-time. No submissions yet. Truth-based CT reconstruction challenge - is openSubmitters will upload the data to Dropbox Organizers approve people in Codalab. After approval, the organizers will grant the participants access to a dropbox folder for the participants to upload the results. DL Sparse - version 2 will go live soon. | |
Action items |
| |
Meeting with Codalab developers | Ben is attending meetings. He is helping with fixing bugs. Ben to review the federated learning tasks and discuss with Codalab team. Python3 | |
Responses to question about keeping challenges open |
Unknown User (bearceba) - investigate options on how to keep challenges open, add phase or copy challenge, avoid having users add to the previous submissions SPIE-AAPM-NCI DAIR Digital Breast Tomosynthesis Lesion Detection Challenge (DBTex) - Phase 1 - |
Unknown User (bearceba) verify with submitters that decision of phase 2 applies to phase 1 SPIE-AAPM-NCI-DAIR & SPIE-AAPM-NCI-DAIR Phase 2 - TBD - (Nick Konz, Wed 11/24/2021 11:36 AM) "Maciej and I discussed this and an idea that we’ve been floating is to have some sort of ongoing validation phase, that would allow anyone to get validation results on their model. We would also want to have a periodic, occasional test phase for people to get test results. BMMR2 - close on as previously planned. (David Newitt, Thu 10/28/2021 7:10 PM) - link to the challenge data will be de-activated on 12/1/2021. DL Sparse - keep open (Emil Sidky, Thu 10/28/2021 11:42 PM) MICCAI 2020 - has already an open test phase ISBI 2018 - has already an open test phase AAPM RT-MAC Challenge - is open-ended | ||
Upcoming challenges | MIDRC classification challenge (frontal/lateral views) AAPM - SIPRISE challenge - Diane Alvares, Baptist Health. She filled out the submission form and sent instructions. TrueCT - Truth-based CT reconstruction challenge - is being setup. Submitters need to upload images (30 GB per submission). Ben is investigating what the best method is e.g. submission to the cloud. Several options were discussed. Google workspace. | |
Information sharing | Ben established connection with developers of Codalab/Codabench. He will join their meeting and potentially take a more active role in product development. These activities can be charged to the contract. | |
Monthly status reports | due on the 10th day of the following month | |
Contract / invoice status / SOW |
| |
Paper | The draft will be submitted by September 15, 2020. - Not received | |
Videos |
| |
Federated learning challenge - proposal for MICCAI, ACR confirmed that the data can be used Wagner, Ulrike (NIH/NCI) [C] Change SOW, take cloud out, put federated learning modifications for Codalab in, send to Medici for review |