This document is for potential competition organizers. It provides an overview of how competitions work, and describes the processes involved in creating and running CodaLab competitions. Beginners might want to use the ChaLab wizard.
In a typical CodaLab competition, participants compete to find the best approach for a particular problem. Competitions may be conducted in multiple phases, e.g. development, feedback, and final. The appropriate data is made available to participants at each phase of the competition. During the development and feedback phases, participants have access to training data to develop and refine their algorithms. During the final competition phase, participants are provided with final test data to generate results, which they can then submit to the competition. Results are calculated at the end of each phase, at which point participants can see the competition results on the leaderboard.
Competitions consist of a set of files collectively known as a "bundle". Although technically CodaLab considers any zipped archive to be a bundle, competition bundles generally contain a specific assortment of files:
This listing gives a general idea of the files which make up a competition, although this can vary depending on the type of competition. Beginners may want to use ChaLab, a wizard that guides you step-by-step to build a competition bundle.
CodaLab competitions can be set up in several different ways. We provide several templates, which differ in the type of data and complexity of the organization. These three templates are the most basic:
Yellow World: The simplest template with RESULT submission. Competitors submit an answer which is compared with the known correct answer.
Compute Pi: The simplest template with CODE submission. Competitors submit an answer which is compared with the known correct answer.
Iris: A two-phase competition similar to what the ChaLab wizard can produce, with either result or code submission.
More complex examples are also provided for developers and advanced users.
In this section we'll walk through the major segments of the competition creation process.
The very first step in creating a competition is planning. For help, see the ChaLearn tips. It is advisable to first start preparing data, a scoring program, and a sample solution. This will be the basis for the "starting kit", which you can distribute to participants to get started. New in Codalab 1.5: the "ingestion program", which allows you to parse the submission of participants.
The diagram below illustrates how submissions made by participants are processed. They are received by and ingestion program, which decides whether to treat them as "result" submission of "code" submission. Result submissions are directly forwarded to the scoring program whereas code submissions are executed on the platform to produce results. The scoring program compares the results with the solution (known only to the organizers).
There are several kinds of data an organizer can provide in each phase:
sample data
in a code submission competition or labeled training data and unlabeled test data
in a result submission competition.The data format can freely be chosen by the organizers.
Codalab lets you either include data in your competition bundle (as zip files) or upload data in My Competitions>My Datasets, then reference the datasets from your YAML configuration file. You can also switch dataset on-the-fly while your competition is running by re-uploading a dataset and select the new version via the competition editor. See My Datasets.
The ingestion program receives the submission of participants. You do not necessarily need to write one. Codalab provides a default ingestion program with the following behavior, depending on the type of submission made by the participant:
Each phase may have a different ingestion program. Scenarios in which you may want to have your own ingestion program include:
For more information see Building an Ingestion Program for a Competition.
The scoring program evaluates the participants' submissions, comparing each submission with a set of reference data (solution), and then passing the resulting scores to the competition leaderboard ("Results" tab).
Each phase may have a different scoring program. For more information see Building a Scoring Program for a Competition.
The starting kit is a "freestyle" bundle created by the organizers, containing everything the participants need to get quickly started. A typical starting kit contains:
The next step is creating a competition bundle. For detailed instructions, see Building a Competition Bundle.
Once your competition is up and running, you can manage it from your CodaLab Dashboard. For more details, see Running a Competition.