Page History

...

Some of the key advantages of challenges over conventional methods include 1) scientific rigor (sequestering the test data), 2) comparing methods on the same datasets with the same, agreed-upon metrics, 3) allowing computer scientists without access to medical data to test their methods on large clinical datasets, 4) making available resources such as source code, and 5) bringing together diverse communities (that may traditionally not work together) of imaging and computer scientists, machine learning algorithm developer, software developers, clinicians and biologists. CKK: Reviewer comment "Review wording of # 4 and 5". CKK: This paragraph also ended with a stray (4) which I just deleted. See the Word document to know what I mean.

However, despite this potential, there are a number of challenges. Medical data is usually governed by privacy and security policies such as HIPPA that make it difficult to share patient data. Patient health records can be very difficult to completely deidentify. Medical imaging data, especially brain MRIs can be particularly challenging as once could easily reconstruct a recognizable 3D model of the subject.

...

Challenge Post has been used to organize hackathons, online challenges and other software collaborative activities. In person hackathons are free while the online challenges cost $1500/month (plus other optional charges). CKK: Reviewer comment: "Where does the section for commercial challenge management systems end?" and separate comment: "Could the suitability of each platform for imaging challenges be discussed?"

Open Source

Synapse is both an open source platform and a hosted solution for challenges and collaborative activities created by Sage bionetworks. It has been used for a number of challenges including the DREAM challenge. Synapse allows the sharing of code as well as data. However, the code typically is in R, Python and similar languages. Synapse also has a nice programmatic interface and methods to upload/download data, submit results, create annotations and provenance through R, Python, command line and Java. These options can be configured for the different challenges. Content in Synapse is referenced by unique Synapse IDs. The three basic types of Synapse objects include projects, folders and files. These can be accessed through the web interface or through programmatic APIs. Experience and support for running image analysis code within Synapse is limited.

...

Matrix of Features and Frameworks (1 -5)

CKK: Reviewer comment: "Set this section apart from the review of the challenge management system."

CKK: Reviewer comment: "Some of the systems mentioned earlier are not listed in the matrix: Innocentive, TopCoder, ChallengePost, Midas/COVALIC"

Below is a table that rates the relative merits of the most relevant frameworks that we evaluated. The scale is 1-5 where 1 indicates excellent support for the feature while 5 indicates that that feature is not currently part of the system or there is limited support.

CKK: Reviewer comment located in the CodaLab ease of setting up new challenge cell: "Explain the scoring system"

...