 |
Page History
...
Challenges are being increasingly viewed as a mechanism to foster advances in a number of domains including healthcare and medicine. The US United States Federal governmentGovernment, as part of the open-government initiative, has underscored the role of challenges as a way to "promote innovation through collaboration and (to) harness the ingenuity of the American Public." Large quantities of publicly available data and cultural changes in the openness of science have now made it possible to use these challenges and crowdsourcing efforts to propel the field forward.
Sites such as Kaggle
| Multiexcerpt include | ||||||
|---|---|---|---|---|---|---|
|
| Multiexcerpt include | ||||||
|---|---|---|---|---|---|---|
|
| Multiexcerpt include | ||||||
|---|---|---|---|---|---|---|
|
In the biomedical domain, challenges have been used effectively in bioinformatics as seen by recent crowd-sourced efforts such as Critical Assessment of Protein Structure Prediction (CASP), the CLARITY Challenge for standardizing clinical genome sequencing analysis and reporting and the cancer Genome atlas Pan-cancer analysis Working Group, DREAM Challenges
| Multiexcerpt include | ||||||
|---|---|---|---|---|---|---|
|
...
Challenges have been popular in a number of scientific communities since the 1990s. In the text retrieval community, the Text REtrieval Conference (TREC), co-sponsored by NIST, is an early example of evaluation campaigns where participants work on a common task using data provided by the organizers and evaluated with a common set of metrics. ChaLearn
| Multiexcerpt include | ||||||
|---|---|---|---|---|---|---|
|
...
Grand Challenges in Biomedical Image Analysis
| Multiexcerpt include | ||||||
|---|---|---|---|---|---|---|
|
| Multiexcerpt include | ||||||
|---|---|---|---|---|---|---|
|
- A task is defined (the output). In our context, this could be segmentation of a lesion or organ, classification of an imaging study as being benign or malignant, prediction of survival, classification of a patients as being a responder or non-responder, pixel/voxel level classification of tissue or tumor grading.
- A set of images are provided (the input). These images are chosen to be of a sufficient size and diversity to reflect the challenges of the clinical problem. Data is typically split up into training and test datasets. The "truth" is made available to the participants for the training data but not the test data. This reduces the risk of overfitting the data and ensures the integrity of the results.
- An evaluation procedure is clearly defined; given the output of an algorithm on a the test images, one or more metrics are computed that measure the performance, usually a reference output is used in this process, but it could also be a visual evaluation of the results by human experts)
- Participants apply their algorithm to all data in the public test dataset provided. They can estimate their performance on the training test.
- Some challenges have an optional leaderboard phase where a subset of the test images is made available to the participants ahead of the final test. Participants can submit their results to the challenge system and have them evaluated or ranked but these are not considered the final standing.
- The reference standard or "ground truth" is defined using methodology clearly described to the participants but is not made publicly available in order to ensure that algorithm results are submitted to the organizers for publication rather than retained privately.
- Final valuation is carried out by the challenge organizers on the test set where the ground truth is sequestered from the participants.
...
These platforms typically charge a hosting fee and offering monetary rewards is pretty common. They have large communities (hundreds of thousands) of registered users and coders and can be a way to introduce the problem to communities outside the core domain expert academic researchers and get solutions that are novel in the domain.
| Multiexcerpt include | ||||||
|---|---|---|---|---|---|---|
|
The metrics that Kaggle supports
| Multiexcerpt include | ||||||
|---|---|---|---|---|---|---|
|
Error Metrics for Regression Problems
...
Topcoder is a similar popular website for software developers, graphic designers and data scientists. In this case, participants typically share their code or designs. They use the Appirio proprietary crowdsourcing development platform, built on Amazon Web Services, Cloud Foundry, Heroku, HTML5, Ruby and Java. A recent computational biology challenge run on Topcoder demonstrated that this crowdsourcing approach produced algorithmic solutions that greatly outperform commonly used algorithms such as BLAST for sequence annotation {Lakhani, 2013 #3789}. This competition was run with a $6000 prize and drew 733 participants (17% of whom submitted code) and the prize-winning algorithms were made available with an open source license.
| Multiexcerpt include | ||||||
|---|---|---|---|---|---|---|
|
...
The main steps to create a new challenge are:
- Create a project
Multiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include - Add pages
Multiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include - Making uploaded files available for download
- Allowing others to register for your project
- Make your project appear in the projects overview
- Allow file uploads
- Including content from files on a page
- Allow others to edit the project
- Changing colors and other styling
- Project data folder
- Page permissions
...
