This document is for potential competition organizers. It provides an overview of how competitions work, and describes the processes involved in creating and running CodaLab competitions. Beginners might want to use the ChaLab wizard.

How Competitions Work

In a typical CodaLab competition, participants compete to find the best approach for a particular problem. Competitions may be conducted in multiple phases, e.g. development, feedback, and final. The appropriate data is made available to participants at each phase of the competition. During the development and feedback phases, participants have access to training data to develop and refine their algorithms. During the final competition phase, participants are provided with final test data to generate results, which they can then submit to the competition. Results are calculated at the end of each phase, at which point participants can see the competition results on the leaderboard.

Competition Bundle File Structure

Competitions consist of a set of files collectively known as a "bundle". Although technically CodaLab considers any zipped archive to be a bundle, competition bundles generally contain a specific assortment of files:

...

This listing gives a general idea of the files which make up a competition, although this can vary depending on the type of competition. Beginners may want to use ChaLab, a wizard that guides you step-by-step to build a competition bundle.

Types of Competition

CodaLab competitions can be set up in several different ways. We provide several templates, which differ in the type of data and complexity of organization. These three templates are the most basic:

...

More complex examples are also provided for developers and advanced users.

Competition End to End Process

In this section we'll walk through the major segments of the competition creation process.

Planning a Competition

The very first step in creating a competition is planning. For help, see the ChaLearn tips. It is advisable to first start preparing data, a scoring program, and a sample solution. This will be the basis for the "starting kit", which you can distribute to participants to get started. New in Codalab 1.5: the "ingestion program", which allows you to parse the submission of participants.

How submissions are processed

The diagram below illustrates how submissions made by participants are processed. They are received by and ingestion program, which decides whether to treat them as "result" submission of "code" submission. Result submissions are directly forwarded to the scoring program whereas code submissions are executed on the platform to produce results. The scoring program compares the results with the solution (known only to the organizers).

Making Data Available

There are several kinds of data an organizer can provide in each phase:

...

Codalab lets you either include data in your competition bundle (as zip files) or upload data in My Competitions>My Datasets, then reference the datasets from your YAML configuration file. You can also switch dataset on-the-fly while your competition is running by re-uploading a dataset and select the new version via the competition editor. See My Datasets.

Creating an Ingestion Program

The ingestion program receives the submission of participants. You do not necessarily need to write one. Codalab provides a default ingestion program with the following behavior, depending on the type of submission made by the participant:

...

For more information see Building an Ingestion Program for a Competition.

Creating a Scoring Program

The scoring program evaluates the participants' submissions, comparing each submission with a set of reference data (solution), and then passing the resulting scores to the competition leaderboard ("Results" tab).

Each phase may have a different scoring program. For more information see Building a Scoring Program for a Competition.

Creating a Starting Kit

The starting kit is a "free style" bundle created by the organizers, containing everything the participants need to get quickly started. A typical starting kit contains:

• A README file (use e.g. README.md or README.ipynb to facilitate sharing the starting kit on Github).
• A sample result submission.
• A sample code submission. The readme file contains instruction and/or sample code to read/display data, and prepare an sample submission. We provide a template starting kit -- also downloadable as zip -- for the Iris challenge.

Creating a Competition Bundle

The next step is creating a competition bundle. For detailed instructions, see Building a Competition Bundle.

Running a Competition

Once your competition is up and running, you can manage it from your CodaLab Dashboard. For more details, see Running a Competition.