2022-04-28 Meeting notes

Date

28 Apr 2022

Attendees

Name	Attended
Jayashree Kalpathy-Cramer
Ulli Wagner	Yes
Carolyn Klinger	No
Benjamin Bearce	Yes

Currently supported challenges

Name of Challenge

Organization / Meeting

Medici instance
used

Release
Training data

Release Validation data

Release
Test data

Number of
Participants

Closing Date

Will the challenge remain open?

Comments

SIRPRISE Reconstruction Challenge

https://www.aapm.org/GrandChallenge/SIRPRISE/

https://medici-codalab-main.eastus.cloudapp.azure.com/

Submitters will upload images to Medici. Did decide not to use FTP transfer but upload zipped images directly to Codalab. It was found out that the data size is not as large as anticipated.

Truth-based CT reconstruction challenge

https://medici-codalab-main.eastus.cloudapp.azure.com/

submitters will upload images to Dropbox, organizers will provide the scoring. Ben will create entries for each participants and enter the scores so that they can be shown on the leaderboard.

DL-sparse challenge #2

Ben is talking with Emil about using MedICI again for another DL-Sparse challenge

Previously supported challenges that remain open

Name of Challenge	Organization / Meeting	Medici instance used	Release Training data	Release Validation data	Release Test data	Number of Participants	Closing Date	Will the challenge remain open?	Comments

Tasks / Deliverables

Task	Due Date	Status
Supporting at least three challenges including one federated learning challenge	8/30/2022	Three challenges are DBTex2, SIRPRISE, and Truth-based CT reconstruction challenge.
Support organizers and participants (help desk, FAQ, training, presentations)	8/30/2022	Ongoing
Maintain and update the website	8/30/2022	Ongoing
Capturing and providing metrics on the use of the MedICI website	8/30/2022	Ongoing
Capturing and providing metrics on the challenges	8/30/2022	Ongoing
Support at least three previously executed challenges	8/30/2022	Done
MedICI on the NCI Cloud Two (GCP) and Firecloud	8/30/2022	Task was suspended per discussion on 19 Nov 2021
Operations and Maintenance	8/30/2022	Ongoing
Changes to MedICI to support federated learning challenges		See Google doc

Discussion items

Time	Item	Who	Notes
	Notes
	28 Apr 2022		Invoices - Finances, HR etc in charge; Ben to supply cloud expenses Ben to log into the NCI cloud environment to see if something is still running. Communicate the findings to Keyvan.
	17 Mar 2022		Updated status of SOW tasks Ben is supporting challenge organizers & participants MIDRC test challenge will start soon including Docker submissions. Supporting Codalab open source project.
	03 Mar 2022		SIRPRISE - active, users can download data. Challenge organizers will establish the scores and Ben will enter them at the end of the challenge. Scores will not be updated in real-time. No submissions yet. Truth-based CT reconstruction challenge - is open Submitters will upload the data to Dropbox Organizers approve people in Codalab. After approval, the organizers will grant the participants access to a dropbox folder for the participants to upload the results. DL Sparse - version 2 will go live soon.
	Action items		Close down https://qin-challenge-acrin.centralus.cloudapp.azure.com/competitions/2 and remove green border CTVIE19 http://aapmchallenges.cloudapp.net/competitions/35 - contact organizer and ask if the challenge can remain open DBTex 1&2 Mazurowski & Konz - not quite sure what to do
	Meeting with Codalab developers		Ben is attending meetings. He is helping with fixing bugs. Ben to review the federated learning tasks and discuss with Codalab team. Python3
	Responses to question about keeping challenges open		Unknown User (bearceba) - investigate options on how to keep challenges open, add phase or copy challenge, avoid having users add to the previous submissions SPIE-AAPM-NCI DAIR Digital Breast Tomosynthesis Lesion Detection Challenge (DBTex) - Phase 1 - Unknown User (bearceba) verify with submitters that decision of phase 2 applies to phase 1 SPIE-AAPM-NCI-DAIR & SPIE-AAPM-NCI-DAIR Phase 2 - TBD - (Nick Konz, Wed 11/24/2021 11:36 AM) "Maciej and I discussed this and an idea that we’ve been floating is to have some sort of ongoing validation phase, that would allow anyone to get validation results on their model. We would also want to have a periodic, occasional test phase for people to get test results. BMMR2 - close on 21 Dec 2021 as previously planned. (David Newitt, Thu 10/28/2021 7:10 PM) - link to the challenge data will be de-activated on 12/1/2021. Update (01/20/2022): David Newitt retired. They want to keep the challenge open but without data. Only participants who had previously registered may continue to work and submit. Ben to check if the data is still accessible. DL Sparse - keep open (Emil Sidky, Thu 10/28/2021 11:42 PM) MICCAI 2020 - has already an open test phase ISBI 2018 - has already an open test phase AAPM RT-MAC Challenge - is open-ended
	Upcoming challenges		MIDRC classification challenge (frontal/lateral views) AAPM - SIPRISE challenge - Diane Alvares, Baptist Health. She filled out the submission form and sent instructions. TrueCT - Truth-based CT reconstruction challenge - 03 Feb 2022 is being setup. Submitters need to upload images (30 GB per submission). Ben is investigating what the best method is e.g. submission to the cloud. Several options were discussed. Google workspace.

	Information sharing		Ben established connection with developers of Codalab/Codabench. He will join their meeting and potentially take a more active role in product development. These activities can be charged to the contract.
	Monthly status reports		due on the 10th day of the following month
	Contract / invoice status / SOW		the last invoice was for June 2021 Ben sent reminder emails to Peter Hickey on 10/25/2021. Resent it today. New subcontracts administrator: Connor Cigrang Connor pinged Lynne Benoit 01/05/2022 regarding invoice.
	Paper		The draft will be submitted by September 15, 2020. - Not received
	Videos		Hold the work on the video for now
			Federated learning challenge - proposal for MICCAI, ACR confirmed that the data can be used Wagner, Ulrike (NIH/NCI) [C] Change SOW, take cloud out, put federated learning modifications for Codalab in, send to Medici for review

Action items

Content

Space Tools