NIH | National Cancer Institute | NCI Wiki  

This document explains how to install CodaLab and is an alternative to the more detailed user manual on the CodaLab Wiki. CodaLab is under active development, so the procedures here are subject to change.

Your first step is to decide which virtual machine or computer you want to install MedICI.

Creating a Virtual Machine on Microsoft Azure\Ubuntu Server 18.04 LTS

This tutorial focuses on Ubuntu-based installations on Azure, although instructions on how to do this on Microsoft and Apple operating systems and in AWS are also included.

Create a VM in Azure or use a computer with Ubuntu Server 18.04 installed. Once you have an Azure account, go to the Azure home page. A sidebar with many options or categories appears. You can create either a resource or a resource group and then add resources into it. This tutorial uses the resource group method to demonstrate good organizational practices.

  1. Select Resource groups > add.
    Microsoft Azure, Resource groups screen, highlighting the Add button
  2. Name the resource group and then decide how you will pay for this service. 
  3. Click Review + create. 
  4. Go to the resource group and add a VM:
    Microsoft Azure, Tutorials screen

  5. Choose Ubuntu Server 18.04 LTS.
    The VM creation page appears. 

  6. Choose the following settings.

    Microsoft Azure, Create a virtual machine screen

  7. Now click Review + create. Review the options and then click create. Azure starts creating the VM and then finishes, as shown in the following two images.
    Microsoft Azure, Your deployment is underway screen

    Microsoft Azure, Your deployment is complete page

IP Address Config

  1. Navigate to the VM by clicking Go to resource. Go to the resources home page to get the public IP address so that you can access the machine.

    MedICI-CodaLab-Master page
  2. Set the IP address setting to Static. Make a note of the IP address. Click Configure to edit the DNS settings. Click Save.
    MedICI-CodaLab-Master-ip Configuration page

SSH Port

We need to configure the machine to be ssh accessible (port 22) and open http (port 80) (https://medium.com/techinpieces/practical-azure-how-to-enable-ssh-on-azure-vm-84d8fba8103e). Go to the VM and click Networking. On the right find the button Add inbound port rule. Change Destination port ranges to 22, Protocol to TCP, Priority to 330, and Name to anything you’d like:

Enable_22 screen for MedICI-CodaLab-Master nsg

Once you have that let’s open a terminal and get inside the machine by typing $ ssh <user>@<public IP address>.

If you get a message regarding encryption keys, enter y\yes as this is the first time signing into this machine.

HTTP Port

On the right find the button Add inbound port rule. Change the Destination port ranges to 80, Protocol to TCP, Priority to 300, and Name to anything you like.

Enable HTTP screen

This step is necessary to access the site once you deploy it.

Installing CodaLab

CodaLab has documentation regarding their preferred installation configuration. The two best sources are the following:

The base code for CodaLab can be set up on different cloud providers (Google Cloud, AWS, Azure). The CodaLab team has made Docker images for AWS and Google cloud that in theory should just run once some settings are established, but require a good working knowledge of Docker. For the purposes of MedICI, it is recommended to follow the “Configure CodaLab from scratch (harder documentation)” procedure found in the first link above.

Step 1 - Install Docker and Docker-Compose

Before we clone GitHub, we must install docker and docker-compose. Follow the Docker installation instructions to install on Ubuntu.

Docker

The following are the commands for a basic installation of Docker on Ubuntu.

$ sudo apt-get remove docker docker-engine docker.io containerd runc
$ sudo apt-get update
$ sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo apt-key fingerprint 0EBFCD88

$ sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io
$ sudo docker run hello-world

This creates the following result.

Now if we run the docker command, we see the options list that tells us how to use the command, which verifies that Docker is installed and we can use it.

Docker-Compose

Check to see if docker-compose is installed from the terminal on your new virtual machine. If not, rather than use sudo apt install docker-compose, the CodaLab documentation from Configure CodaLab from scratch (harder documentation) says to install it from Docker's website. Use the following command.

sudo curl -L "https://github.com/docker/compose/releases/download/1.25.5/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose


Now change the permissions in /usr/local/bin/docker-compose to use docker-compose:

sudo chmod +x /usr/local/bin/docker-compose


Run docker-compose in the terminal window. The options list appears that tells us how to use this command. This verifies that docker-compose is installed and ready to use.

Step 2 - Get the Source Code

The latest code is located in the codalab-competitions folder in the codalab repository (repo) in GitHub. Both sets of instructions direct you to clone this repository locally to the virtual machine. Since we will be customizing our installation, fork the repository and do the following:

  1. Pull the latest changes from the master repo https://github.com/codalab/codalab-competitions into your own repo periodically to make sure you have the latest base code. (repo: https://github.com/QTIM-Lab/MedICI/). If you need help doing this on your own, see forking a repo, but you can just clone the QTIM-Lab/MedICI repo.
  2. Store your customizations on GitHub so that you can clone from this repo for future projects.
  3. Clone the MedICI project. This follows CodaLabs instructions but with our GitHub repo instead. Below is a picture of the commands I’ve executed from “Configure CodaLab from scratch (harder documentation)”. The green circles represent once inside the “src” folder, the changes that are not in the raw CodaLab github repo. This is the start of our customizations.
  4. Push these changes into your branch. At this point the app should be ready to deploy.
  5. Run $ docker compose up -d. You may encounter the following error:

    The solution to this error is to run with sudo or add yourself to the docker group:
    $ sudo docker-compose up -d

    Or
    https://docs.docker.com/install/linux/linux-postinstall/

    • $ sudo groupadd docker (this may be done already)
    • $ sudo usermod -aG docker $USER (now restart the VM)
    • Run $ docker-compose up -d
    • The flag ‘-d’ means to run in detached mode. If you don’t use it, you will notice your command prompt is in a live feed mode telling you everything that is going on with this Docker.

Step 3 - Tweak the .env file and Azure Storage

The following documentation explains this step: https://codalab-competitions.readthedocs.io/en/latest/

Create a Storage Account

Below is the blob storage section for Azure:

You may sign up for an Azure account, then follow the directions below. You do not have to do this if you've already set up S3.

  1. Log on to the Azure Portal.
  2. From the Dashboard, click Storage accounts on the left.
  3. Click Add at the top of the page to create a new storage account.
  4. If you don't already have a subscription, create one now. The free trial requires a credit card, and deletes all your storage containers after 90 days, unless you upgrade to a different plan such as 'Pay as You Go'.
  5. Select the Classic storage account. Refer to the following image for settings.

    medici - Blobs settings page
  6. In the dashboard, click All Resources/All Subscriptions and then click  your username. Click Access Keys and copy your account name and access key to .env under AZURE_ACCOUNT_NAME and AZURE_ACCOUNT_KEY.
  7. Within that same user account, click on Containers and Add a new container.
  8. Create a new container named "bundles". Set the Access to "Private".
  9. Add another container named "public". Set the Access to "Public Blob".
  10. Make sure the DEFAULT_FILE_STORAGE .env option is set to codalab.azure_storage.AzureStorage.


The CodaLab documentation leaves one final step out. We NEED to enable CORS. Click CORS in the image in the previous procedure and add these settings (https://stackoverflow.com/questions/50785255/azure-storage-got-403-url-cors-not-enabled-or-no-matching-rule-found-for-thi):

If you don’t, uploading a competition will present a CORS errors in the browser:

Once applied, uploading works:

Adding a Custom Logo

To create a custom logo, follow the Codalab instructions but note that you must log in to the database as the super user to use the customizer module. Django doesn’t have a super user by default, so you must create it. To do so, create a secure connection to the virtual machine that hosts your site and run the following command.

$ docker exec -it django python manage.py createsuperuser


You can now return to the CodaLabs instructions and navigate to http://your-instance.com/customizer. Log in with your super user credentials and choose a new file as your logo.

  • No labels