NIH | National Cancer Institute | NCI Wiki  

Versions Compared

Old Version 2

changes.mady.by.user Wagner, Ulrike (NIH/NCI) [C]

Saved on

compared with

New Version Current

changes.mady.by.user Wagner, Ulrike (NIH/NCI) [C]

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

NameAttended
Jayashree Kalpathy-CramerYes (10 min)
Ulli WagnerYes
Carolyn KlingerYesNo
Benjamin BearceYes


Currently supported challenges

http://spie-aapm-nci-dair.westus2.cloudapp.azure.com/competitions/4

Phase 2

https://spie-aapm-nci-dair.westus2.cloudapp.azure.com/competitions/6

1/4/2021

Here's a link for the Challenge description/rules/timeline:

https://www.dropbox.com/s/ciqqbal4xlxcn8x/DL-sparse-view-CT-challenge.docx?dl=0
Challenge Location: http://dl-sparse-view-ct-challenge.eastus.cloudapp.azure.com/

Related to AAPM Annual meeting in July

Name of ChallengeOrganization / MeetingMedici instance
used

Release
Training data

Release Validation dataRelease
Test data		Number of
Participants		Closing DateWill the challenge remain open?Comments

SPIE-AAPM-NCI DAIR Digital Breast Tomosynthesis Lesion Detection Challenge (DBTex)

 

1/15/2021

132 registrants

9 submissions

1/25/2021Challenge completed. The validation phase remains open.Deep-learning for inverse problems: sparse-view X-ray Computed Tomography image reconstruction

Emil Sidky (UChicago)

3/17/2021March 31, 2021May 17, 2021123 registrants5/31/2021

2021/04/01

In validation phase

Ben & Emil are testing

Data are on an FTP server.

Ben set up an Azure container to download the data for two participants.












Previously supported challenges that remain open

Name of ChallengeOrganization / MeetingMedici instance
used

Release
Training data

Release Validation dataRelease
Test data		Number of
Participants		Closing DateWill the challenge remain open?Comments










Tasks / Deliverables


2021

Investigate Google analytics plug-in - DONE (2021/04/01)

391 page views, 107 users over the last 12 months2021

App scan found vulnerabilities, Ben fixed them, app needs to be tested with test cases that are provided by Codalab, Ben needs to familiarize himself with the test suite. Codalab needs to be app scanned again.

Codalab can connect to various storage places: Azure, AWS, GCP

2021/04/15 - NCI is hardening security which makes deployment much harder. NCI VMs are not allowed to talk to cloud environments. Kevin is investigating workarounds. The goal is to run an app scan. 

Keyvan suggested trying a Codalab deployment on Google cloud outside of the NCI network. Keyvan will investigate.

TaskDue DateDeliverableStatus

Support of Supporting at least new two challengesthree challenges including one federated learning challenge

8/30/2021

Reopen three previously executed challenges

  • process the results of new submissions
  • update the leaderboards
  • inform participants of their results
2022


Support organizers and participants (help desk, FAQ, training, presentations)

8/30/2022




8/30/2021
Maintain and update the website8/30/20212022


Capturing and providing metrics on the use of the MedICI website

8/30/2022


Capturing and providing metrics on the challenges

8/30/Investigate the use of MedICI for federated learning challenges

 

Report

https://docs.google.com/document/d/1v7Mz-diz8wSXzpdrGOGXwaDnZAOZ2t879frgTvzyt24/edit#heading=h.tqyuzeebfeoq

  • need to understand what is meant by "federated learning"
  • review report
  • have a discussion with Keyvan
  • determine if MedICI is suited for support of federated challenges
  • if yes, what needs to be done to make it suitable
  • find time for discussion during the week of November 15
Investigate the integration of MedICI with Kheops

 

Report

Presentation: NICC

The recordings are towards the bottom of the page.

URL: https://kheops.online/

Draft: https://docs.google.com/document/d/1c20ny4UNWw4BTXhdFPmc3o0YG00toOndbklXhc40Rqc/edit

The report was finalized on  

Support the deployment of MedICI in NCI cloud environments8/30/2021

Discussion items

2022


Support at least three previously executed challenges

8/30/2022


MedICI on the NCI Cloud Two (GCP) and Firecloud

8/30/2022

Task was suspended per discussion on  


Operations and Maintenance

8/30/2022







Discussion items


CodaLab on AWS, Codalab's older version is still working, and some developers check-in code. The newer version (Python 3) had the last commit in August 2020. Deploying on a different cloud platform (AWS) and within the NCI environment is difficult. 
  • Authorization to Proceed was issued, covers the month of September
  • Contract mod will follow
  • Invoice ending 8/30/2020 - in progress
  • Invoice 8/31/2020
  • Cost proposal for the new contract year: FNLCR will request a modification to the cost proposal -  Done.
  • review paper
  • open challenges back up (contract / funding)
    • website
    TimeItemWhoNotes

    		Notes 


    		Action items




    		Responses to question about keeping challenges open

    Unknown User (bearceba)  - investigate options on how to keep challenges open, add phase or copy challenge, avoid having users add to the previous submissions

    SPIE-AAPM-NCI DAIR Digital Breast Tomosynthesis Lesion Detection Challenge (DBTex) - Phase 1 - Unknown User (bearceba) verify with submitters that decision of phase 2 applies to phase 1

    SPIE-AAPM-NCI-DAIR & SPIE-AAPM-NCI-DAIR Phase 2  - TBD - (Nick Konz, Wed 11/24/2021 11:36 AM) "Maciej and I discussed this and an idea that we’ve been floating is to have some sort of ongoing validation phase, that would allow anyone to get validation results on their model. We would also want to have a periodic, occasional test phase for people to get test results.

    BMMR2 - close on as previously planned. (David Newitt, Thu 10/28/2021 7:10 PM) - link to the challenge data will be de-activated on 12/1/2021

    DL Sparse -  keep open (Emil Sidky, Thu 10/28/2021 11:42 PM) 

    MICCAI 2020 - has already an open test phase

    ISBI 2018 - has already an open test phase

    AAPM RT-MAC Challenge - is open-ended


    		Upcoming challenges






    		Information sharing


    		Monthly status reports
    		due on the 10th day of the following month
    TimeItemWhoNotes
    Notes Action itemsUpcoming challenges
    • COVID challenge (Randy Gollub, Jayashree Kalpathy-Cramer)
    		Information sharingMonthly status reportsadd Monday NCI cloud email to the monthly status reportNCI MedICI installation

    		Contract / invoice status / SOWPIV card / GFE laptop / VPN

    VPN access


    • the last invoice was for June 2021
    • need invoices at least up to 08/2021 to close out the contract year that ended  
    • need invoices for the current contract year
    • Ben sent reminder emails to Peter Hickey on 10/25/2021. Resent it today.

    		Paper
    		The draft will be submitted by September 15, 2020. - Not received

    		Videos
      • Hold the work on the video for now



    Federated learning challenge - proposal for MICCAI, ACR confirmed that the data can be used

    Wagner, Ulrike (NIH/NCI) [C] Change SOW, take cloud out, put federated learning modifications for Codalab in, send to Medici for review

    		Strategy meeting





    Action items