NIH | National Cancer Institute | NCI Wiki  

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.


Discussion items




  • Invoices - Finances, HR etc in charge; Ben to supply cloud expenses
  • Ben to log into the NCI cloud environment to see if something is still running. Communicate the findings to Keyvan.


Updated status of SOW tasks

Ben is supporting challenge organizers & participants

MIDRC test challenge will start soon including Docker submissions.

Supporting Codalab open source project.


SIRPRISE - active, users can download data. Challenge organizers will establish the scores and Ben will enter them at the end of the challenge. Scores will not be updated in real-time. No submissions yet.

Truth-based CT reconstruction challenge - is open

Submitters will upload the data to Dropbox

Organizers approve people in Codalab. After approval, the organizers will grant the participants access to a dropbox folder for the participants to upload the results. 

DL Sparse - version 2 will go live soon.

Action items

Meeting with Codalab developers

Ben is attending meetings. He is helping with fixing bugs. 

Ben to review the federated learning tasks and discuss with Codalab team.


Responses to question about keeping challenges open

Unknown User (bearceba)  - investigate options on how to keep challenges open, add phase or copy challenge, avoid having users add to the previous submissions

SPIE-AAPM-NCI DAIR Digital Breast Tomosynthesis Lesion Detection Challenge (DBTex) - Phase 1 - Unknown User (bearceba) verify with submitters that decision of phase 2 applies to phase 1

SPIE-AAPM-NCI-DAIR & SPIE-AAPM-NCI-DAIR Phase 2  - TBD - (Nick Konz, Wed 11/24/2021 11:36 AM) "Maciej and I discussed this and an idea that we’ve been floating is to have some sort of ongoing validation phase, that would allow anyone to get validation results on their model. We would also want to have a periodic, occasional test phase for people to get test results.

BMMR2 - close on as previously planned. (David Newitt, Thu 10/28/2021 7:10 PM) - link to the challenge data will be de-activated on 12/1/2021. 
Update (01/20/2022): David Newitt retired. They want to keep the challenge open but without data. Only participants who had previously registered may continue to work and submit. Ben to check if the data is still accessible.

DL Sparse -  keep open (Emil Sidky, Thu 10/28/2021 11:42 PM) 

MICCAI 2020 - has already an open test phase

ISBI 2018 - has already an open test phase

AAPM RT-MAC Challenge - is open-ended

Upcoming challenges

MIDRC classification challenge (frontal/lateral views)

AAPM - SIPRISE challenge - Diane Alvares, Baptist Health. She filled out the submission form and sent instructions.

TrueCT - Truth-based CT reconstruction challenge - is being setup. Submitters need to upload images (30 GB per submission). Ben is investigating what the best method is e.g. submission to the cloud. Several options were discussed. Google workspace. 

Information sharing
Ben established connection with developers of Codalab/Codabench. He will join their meeting and potentially take a more active role in product development. These activities can be charged to the contract.

Monthly status reports
due on the 10th day of the following month

Contract / invoice status / SOW
  • the last invoice was for June 2021
  • Ben sent reminder emails to Peter Hickey on 10/25/2021. Resent it today.
  • New subcontracts administrator: Connor Cigrang
  • Connor pinged Lynne Benoit 01/05/2022 regarding invoice.

The draft will be submitted by September 15, 2020. - Not received

    • Hold the work on the video for now

Federated learning challenge - proposal for MICCAI, ACR confirmed that the data can be used

Wagner, Ulrike (NIH/NCI) [C] Change SOW, take cloud out, put federated learning modifications for Codalab in, send to Medici for review