Hello folks. My name is Artem Mamanov and I'm a software developer
for Massachusetts General Hospital working with Jayashree Kalpathy and today I will demo
our cbibop platform which is used for hosting, evaluation, and optimization
of algorithms for medical image processes
it's available of the web address cbibop.cloud.net it's based on the
CodaLab project developed by Microsoft and also for evaluation and optimization of
algorithms for machine learning. The cbibop platform consists of two main
apps, the competitions app and the worksheets app. The competitions
app contains all the information related to the competition
such as hosting, organization, evaluation and presentation of results
The worksheets app is used as a digital notebook for running
computational experiments in the cloud.
It's a very useful app because it allows you to save all of your input data, code,
program results and allows you to make graphs and tables and present it all in
the one digital notebook. It can also be used for digital publications.
Ok, let me show you around the website.
One of the features of the platform is that it allows user registration.
Actually, in order for participants to take part in the competition, he or
she should first register with the platform so important to register you
can follow the sign in button located in the top right corner of the screen
this will take you to
login screen so if you have an account you can sign in here
if you don't have an account you can follow the same up link that will take
you to sign up page.
so here you can provide your username, email, password and click the sign up
button, This will send an email to your email account with the link to
confirm registration for the platform.
You can follow this link and complete the registration process. I already have
an account so I will sign in in the system. Now let's look look at the main
competitions page.
The main competitions page contains several different tabs that help to
organize competitions and data sets hosted on the website. So the first step
shows all publicly available competitions.
The second type of competitions I'm in shows all competitions in which the
logged-in user is participating in.
So notice that it contains more competitions than the first one because
not all of them are publicly available. The next step competitions I'm running
shows are all competitions in which logged in user is
organizing or is a co-organizer. So there are different buttons that
allow to make changes to the competition; for example, to delete the competition,
to edit the competition, publish a competition to make it a publicly
available. Also allows to show the list of all participants registered in a
particular competition, allows to see list of submissions, and allows to see
the leaderboard all the results submitted to the leaderboard.
also notice that there is a create competition bundle which allows to
submit a new competition. The last stop shows all datasets used in the
competition.
also this data sets for example can be used by organizer to provide data set
provide data for competition or for example to upload a new evaluation
program or ground truth data set using competition
so the new bundle can be uploaded by clicking and Create Dataset.
Okay, now let's look in more detail how competitions look inside so we can
click on the particular competition which takes us to a page with different
tabs that have access to different features of competition.
so the first step on the details gives the overall overview of a competition and
provides a description of the competition. On the left side
there is a list of different pages that describe different aspects of the
competition. The first one overview the second one evaluation that maybe
describes a different criteria used by the evaluation script, terms and
conditions that maybe provides some disclosures of the data and the rules of
use of the data and there might be other pages that describe different aspects of
the competition. The next step
describes different phases used in the competition.
So in this example, there is only one phase, evaluate.
The next stop is the most important tab because it provides access to the data
used in the computation and also allows participants to make submissions so in
this example there is a description of the data that is used in the competition
as well as we provide the link to the TCIA archive where the data can be
downloaded
so the next page submit and view results provides access to make
submissions. There is a submit bottom
so if you click on it it opens a window where you can choose your submission and
then there is a table that provides a description of different submissions
so it has different useful information such as the filenames, submission date,
status whether it's finished or failed. If it failed
you can click on the plus sign and at the end it will give a detailed explanation
of the error so that the participant can fix the problem and resubmit the
results.
It also gives access to several log files such as a log files from
the evaluation script as well as the results of the score of the
evaluation script.
Also there is a button that allows participants to submit results to the
leaderboard which takes me to the next step which provides access to the
leaderboard. So leaderboard shows a list of users that uploaded the submissions
to the leaderboard so the submissions can be anonymous.
Participants don't have to submit the results if they don't want to.
It also gives a list different metrics
which are calculated by the evaluation script. Also the last stop
provides access to the forum where participants can ask questions as well
as organizers of the competition can make announcements.
Now let's talk about how to create the competition in the cbibop platform.
The competition submission starts with a creating competition bundle. A competition
bundle consists of several files compressed into a single zip file.
Here is a list of different files used to make a competition.
There is a main configuration file in the yaml format that describes the
main parameters to make a competition. Let's examine the context of this file. So
it has different parameters that describes different features of the
competition such as title, description of the competition, image uses logo,
what the participants have to register for the competition,
the end date of the competition as an HTML field that describes different HTML
files that they used to describe different aspects of the competition.
There is the phases field that list and describe different phases of the
competition
so there might be more than one phase; for example, they might be a training phase and
a test phase. Each phase should have a description
such as the phase number, the start date
the maximum number of submissions, the scoring program used to evaluate that
phase and the reference date to the ground truth data are used by the
evaluation script
it also has a leaderboard field that describes the table with the results,
so there might be more than one table in this case is just a single
table with multiple columns that the calculated by the evaluation script
ok so there are two more files that are necessary to prepare a competition
bundle.
One of them is the evaluation bundle which should be in a zip file that
contains the evaluation script.
Tis is the main script that is used to evaluate the results and there is a
ground truth or reference bundle
also in the zip file that contains the ground truth data used by the evaluation
script
so when your files are ready you can zip them,
archive them in a single zip file, and then we'll upload it to the
competition
We can navigate the Competitions I'm Running page of
the cbibop platform and click on Create Competition Bundle
so we can follow the upload definition link and then find our data, find our
competition bundle, competition zip file, click OK and now we have to wait
for the competition to finish uploading to the website. It may take a few minutes
depending on the size of the competition.
Some files, for example the ground truth files, might take a lot of
space and that may increase the time necessary to upload the
competition.
Ok so the competition successfully was uploaded. Now it's processing and
parsing the different files to build the competition
inside the platform. That might also take a few minutes depending on the
complexity and different files used in the competition.
Ok the competition has been successfully created so we can view our new
competition. It looks good. Actually a nice feature of the CodaLab platform is
that it provides a secret URL which can be shared by the competition organizers
without exposing it to the public.
So, for example, I can copy and paste this link and send it to other competition
organizers for them to check if everything looks good and they can also
edit the competition and make changes to the
competition.
Let's look at how to submit results for evaluation to complication in the cbibop
platform.
First you have to prepare a submission bottle. A submission bundle
should contain
one of several files that will be evaluated by the evaluation script and
should be compressed in the single zip file. In this case I have a single nifty
file that contains a tumor segmentation generated by segmentation algorithm so I
archived my file in submission to the zip file. Now let's look at how to make
a submission.
Let's navigate to the Participate tab of the competition,
click on the submit button, choose our zip file,
click ok, and this will initiate the submission process so the
submission table shows the status of the submission
so now chose submitting I can refresh the status
it will change to Submitted and then Running and then Finished. If everything is
ok the final status should be Finished.
It may take several minutes to complete the submission process and also the
evaluation script depending on the complexity of the calculations may take
several minutes to complete.
Ok so the status has just changed from Running to Finished,
so it means that our submission was successfully evaluated.
So here the the platform provides links to several log files, for example we can
download the files that the output from scoring step.
Let's check that.
Ok the scores are located in the txt file.
Let's open this file. Ok, it contains key value pairs the metric evaluated
in the value of the metric. So these values will be will be posted to the
leaderboard. So we can submit our results to the leaderboard and then let's check
how it looks.
So here we have different codes corresponding to different matrix
and we have our results.
So now we can also look at the detailed results for each submitted file. In this
case, we have only one file but if we had several files then we'd have a
graph with plots corresponding with different parts corresponding to
different files. Also we have a nice table shows the results.
Okay, that's about it.
Thank you for listening.