Page History

...

Figure 2 Dice coefficients of inter-rater agreement and of rater vs. fused label maps

Figure 3 Dice coefficients of individual algorithms and fused results indicating improvement with label fusion
More recently, the medical imaging community has begun organizing cloud-based challenges. The VISCERAL project (EU-funded effort) has organized a series of challenges at ISBI, MIICAI and other conferences where the participants in the challenge also share the algorithms and code, in addition to the results.
Below is a workflow diagram that describes the various stakeholders in the challenge and their tasks.
Image Removed Image Added
Figure 4 Challenge stakeholders and their tasks
Existing challenge infrastructure
A number of platforms exist for conducting challenges and crowdsourced efforts. Many of the popular platforms are commercial products, typically offering hosting and organizing services. Challenge organizers work with the company to set up the challenge. In some cases, the challenges are fairly trivial to set up and can be set up with the organizer without much support from the challenge platform company.
Commercial/hosted
We begin with a brief review a number of popular platforms used for challenges.
These platforms typically charge a hosting fee and offering monetary rewards is pretty common. They have large communities (hundreds of thousands) of registered users and coders and can be a way to introduce the problem to communities outside the core domain expert academic researchers and get solutions that are novel in the domain. The
Kaggle (https://www.kaggle.com/) is a very popular platform for data science competitions. It is a commercial platform used by companies to pose problems for monetary rewards, jobs and knowledge advancement. There are public and private leaderboards with the test data also being withheld from the participant. Typical hosting costs are reported to be $15,000-20,000 plus additional costs for prizes. However, Kaggle does have a free hosting option to organize challenges for educational purposes. https://inclass.kaggle.com/Questions:
How are educational challenges defined? Would NCI-run challenges fall under this category? What limitations do educational challenges vs. paid challenge hosting have? This option is primarily meant to be used by instructors as part of the class curriculum. Kaggle does not provide any support for organizers of Kaggle In Class. There is a 100GB limit on file size. There also appears to be very simple options for scoring. Almost all challenges hosted here appear to be prediction type challenges where results can be submitted as a csv file and the "truth" is also a csv file. It does not appear that imaging-based challenges (such as segmentation challenges) would lend themselves to being hosted on Kaggle In Class without significant effort.
The metrics that Kaggle supports include the following _GoBack" class="external-link" rel="nofollow">https://www.kaggle.com/wiki/Metrics

...