NIH | National Cancer Institute | NCI Wiki  

Date

Attendees

Name

Attended

Benjamin BearceYes
Carolyn KlingerYes
Jayashree Kalpathy-CramerNo
Keyvan FarahaniNo
Linmin PeiYes
Ulli WagnerNo


Currently supported challenges

Name of Challenge

Organization / Meeting

Medici instance
used

Release
Training data

Release Validation data

Release
Test data

Number of
Participants

Closing Date

Will the challenge remain open?

Comments

DL-sparse challenge #2

Ben is talking with Emil about using MedICI again for another DL-Sparse challenge








SIRPRISE Reconstruction Challenge

https://www.aapm.org/GrandChallenge/SIRPRISE/https://medici-codalab-main.eastus.cloudapp.azure.com/







Submitters will upload images to Medici. Did decide not to use FTP transfer but upload zipped images directly to Codalab. It was found out that the data size is not as large as anticipated.



Truth-based CT reconstruction challenge


https://medici-codalab-main.eastus.cloudapp.azure.com/







submitters will upload images to Dropbox, organizers will provide the scoring. Ben will create entries for each participants and enter the scores so that they can be shown on the leaderboard.

Breast density FL


https://medici-codalab-main.eastus.cloudapp.azure.com/

 


 


 


The submitter will upload the wrapped Docker image to the Codalab platform. Ben will execute the algorithms and post the performances on the leaderboard.

MIDRC COVIDx Challenge


https://midrc-medici-competitions.eastus.cloudapp.azure.com/competitions/7

or

https://www.midrc.org/

 

 

 




The submitter will upload the wrapped Docker image to the Codalab platform. Ben will execute the algorithms and post the performances on the leaderboard.


Previously supported challenges that remain open

Name of Challenge

Organization / Meeting

Medici instance
used

Release
Training data

Release Validation data

Release
Test data

Number of
Participants

Closing Date

Will the challenge remain open?

Comments











Tasks / Deliverables


Task

Due Date

Status

Supporting at least three challenges including one federated learning challenge

 

Three challenges are DBTex2, SIRPRISE, and Truth-based CT reconstruction challenge.

Support organizers and participants (help desk, FAQ, training, presentations)

 

Ongoing
Maintain and update the website

 

Ongoing

Capturing and providing metrics on the use of the MedICI website

 

Ongoing

Capturing and providing metrics on the challenges

 

Ongoing

Support at least three previously executed challenges

 

Done

MedICI on the NCI Cloud Two (GCP) and Firecloud

 

Task was suspended per discussion on  

Operations and Maintenance

 

Ongoing

Changes to MedICI to support federated learning challenges

 

See Google doc

Support Codalab / Codabench open source development

 

Ongoing

Discussion items


Date

Item

Notes

 


Hosting a challenge

  •  DGM-image
    • A task to simulate breast images using GAN
    • plans on Dec. 5, 2022

Plan a challenge

  • Tactic challenge
    • A task to estimate the time series activity variations in Radiopharmaceutical Therapy
    • plans on Dec. 15, 2022

 


  • Completed the MIDRC COVIDx Challenge
    • In the validation phase, there are 7 teams participated.
    • In the testing phase, there are 4 teams that successfully submitted the algorithms.
  • Two challenges are planning.
    • DGM-image
      • A task to simulate breast images using GAN
      • plans on Dec. 5, 2022
    • Tactic challenge
      • A task to estimate the time series activity variations in Radiopharmaceutical Therapy
      • plans on Dec. 15, 2022

 


  • Running the MIDRC COVIDx Challenge
    • Runs well
    • The issue of executing algorithm failure is fixed. The reason is that size of the uploaded container image is too large (up to 39 Gb), which exceeds the VM system.
  • Two challenges are planning.
    • DGM-image
      • A task to simulate breast images using GAN
      • plans on Dec. 5, 2022
    • Tactic challenge
      • A task to estimate the time series activity variations in Radiopharmaceutical Therapy
      • plans on Dec. 15, 2022

 


  • Running the MIDRC COVIDx Challenge
    • 35 teams registered
    • 5 teams successfully run the algorithms in the training phase, and 4 teams successfully executed algorithms in validating phase. 
    • Some participants have issues for executing the algorithms, especially when the uploaded container is too large.
  • Another three challenges are planning.

 


  • Completed the Breast Density Federated Learning
  • MIDRC COVIDx challenge
    • Selecting 362 cases for validation and testing phases.
  • Scheduled the first meeting of CVIT challenge

 


  • The Breast Density Federated Learning is completed. In the last phase, 7 teams successfully execute the algorithms.
  • PI is planning to write a manuscript based on the Breast Density FL challenge.
  • Ben is working on the monthly technical report
  • Ongoing challenge: MIDRC COVIDx challenge
  • Three potential challenges in coming winter or spring
    • CVIT challenge, similar to TrueCT reconstruction
    • Deep generative modeling for learning medical imaging statstics
    • TACTIC: Time-Activity Curve and Time-Integrated Activity Variations in Radiopharmaceutical Therapy Challenge

 


  • The submission deadline for the Breast Density Federated Learning Challenge extends to Sept. 5, 2022 (original date: Aug. 30, 2022)
  • Ben is working on the monthly technical report
  • New challenge plan: TrueCT
  • Ongoing challenge: MIDRC COVIDx challenge (deadline: Dec. 1, 2022)

 


  • Participants start to use the new website. Most of them are able to upload docker image, and execute algorithms successfully. However, a few cases are failed because of unknown reasons. Ben believes that output path may be the reason. He can manually handle it but takes time.
  • Just clarify that Aug. 31 is the starting date for testing submission.
  • Around 40 teams had trained their models.
  • CodaBench is a new platform. It is risky to run new challenges. However, using CodaLab is quite difficult because of the old version of python. In future, TrueCT and MIDRC Challenge 1 are potential challenges. 
  • Send the monthly report of July via email.

 


  • Asking participants to use the real site after Aug. 7th
  • For security reasons, the website restricts IP to public.
  • There are 37 groups approved in the Breast Density FL Challenge.
  • For longterm service, we need to consider the CodaBench for hosting challenges in the future.
  • Question about date for monthly report

 


https://www.medici-challenges.org/

https://medici-codalab-main.eastus.cloudapp.azure.com

  • Introduce the new TPM (Linmin Pei) to the group.
  • Some participants are able to register with WiFI connect. However, they still have access issues. Ben works around to solve the issue. 
  • All participants use Docker template to create the Docker image and their algorithms use GPU version. 
  • Transition to UColorado

 


  • Transition to UColorado
    • contract
    • Emails
  • Monthly status report May needs to be finalized
  • Monthly status report June is due on 7/10/2022
  • Invoices MGH - Sept-April - was rejected
  • Ulli on vacation July 13 - August 11, 2022

 


Meeting canceled, Ulli out of the office

 


Moving the contract to UCD

  • RFP with updated SOW was sent to Sarah Carter on  (S22-077)
  • Proposal due date is 
  • Ben is starting at UCD on  
  • The agreement with MGH ended on  

Review of MSR draft for May 2022

 


Discussed contract issues, move to University of Colorado

 


  • Invoices - Finances, HR etc in charge; Ben to supply cloud expenses
  • Ben to log into the NCI cloud environment to see if something is still running. Communicate the findings to Keyvan.

 


Updated status of SOW tasks

Ben is supporting challenge organizers & participants

MIDRC test challenge will start soon including Docker submissions.

Supporting Codalab open source project.

 


SIRPRISE - active, users can download data. Challenge organizers will establish the scores and Ben will enter them at the end of the challenge. Scores will not be updated in real-time. No submissions yet.

Truth-based CT reconstruction challenge - is open

Submitters will upload the data to Dropbox

Organizers approve people in Codalab. After approval, the organizers will grant the participants access to a dropbox folder for the participants to upload the results. 


DL Sparse - version 2 will go live soon.


Action items

Meeting with Codalab developers

Ben is attending meetings. He is helping with fixing bugs. 

Ben to review the federated learning tasks and discuss with Codalab team.

Python3 


Responses to question about keeping challenges open

Bearce, Benjamin (NIH/NCI) [C]  - investigate options on how to keep challenges open, add phase or copy challenge, avoid having users add to the previous submissions

SPIE-AAPM-NCI DAIR Digital Breast Tomosynthesis Lesion Detection Challenge (DBTex) - Phase 1 - Bearce, Benjamin (NIH/NCI) [C] verify with submitters that decision of phase 2 applies to phase 1

SPIE-AAPM-NCI-DAIR & SPIE-AAPM-NCI-DAIR Phase 2  - TBD - (Nick Konz, Wed 11/24/2021 11:36 AM) "Maciej and I discussed this and an idea that we’ve been floating is to have some sort of ongoing validation phase, that would allow anyone to get validation results on their model. We would also want to have a periodic, occasional test phase for people to get test results.

BMMR2 - close on  as previously planned. (David Newitt, Thu 10/28/2021 7:10 PM) - link to the challenge data will be de-activated on 12/1/2021. 
Update (01/20/2022): David Newitt retired. They want to keep the challenge open but without data. Only participants who had previously registered may continue to work and submit. Ben to check if the data is still accessible.

DL Sparse -  keep open (Emil Sidky, Thu 10/28/2021 11:42 PM) 

MICCAI 2020 - has already an open test phase

ISBI 2018 - has already an open test phase

AAPM RT-MAC Challenge - is open-ended


Upcoming challenges

MIDRC classification challenge (frontal/lateral views)

AAPM - SIPRISE challenge - Diane Alvares, Baptist Health. She filled out the submission form and sent instructions.

TrueCT - Truth-based CT reconstruction challenge -  is being setup. Submitters need to upload images (30 GB per submission). Ben is investigating what the best method is e.g. submission to the cloud. Several options were discussed. Google workspace. 





Information sharingBen established connection with developers of Codalab/Codabench. He will join their meeting and potentially take a more active role in product development. These activities can be charged to the contract.

Monthly status reportsdue on the 10th day of the following month

Contract / invoice status / SOW
  • the last invoice was for June 2021
  • Ben sent reminder emails to Peter Hickey on 10/25/2021. Resent it today.
  • New subcontracts administrator: Connor Cigrang
  • Connor pinged Lynne Benoit 01/05/2022 regarding invoice.

PaperThe draft will be submitted by September 15, 2020. - Not received

Videos
    • Hold the work on the video for now


Federated learning challenge - proposal for MICCAI, ACR confirmed that the data can be used

Wagner, Ulrike (NIH/NCI) [C] Change SOW, take cloud out, put federated learning modifications for Codalab in, send to Medici for review

Action items

  •