NIH | National Cancer Institute | NCI Wiki  

Error rendering macro 'rw-search'

null

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Introduction

This document is intended to show how to install CodaLab from start to finish in a linear fashion. CodaLab is under active development, so the procedures here are subject to change.

Your first step is to decide which virtual machine or computer you want to install MedICI.

Microsoft Azure\Ubuntu Server 18.04 LTS:

This tutorial focuses on Ubuntu-based installations on Azure, although instructions on how to do this on Microsoft and Apple operating systems and in AWS are also included.

Create a Virtual Machine

First create a VM in Azure or get yourself a computer with Ubuntu Server 18.04 installed. Once you have an Azure acount go to the home page here (https://portal.azure.com/#home). You should see a side-bar with many options or categories. You have the option to create either a resource or a resource group and then add resources into it. This tutorial will demo with a resource group to demonstrate good organizational practices. First click Resource groups and then add.

Microsoft Azure, Resource groups screen, highlighting the Add button

Name the resource group and then decide how you will pay for this service. Click Review + create. Once that is complete, go to the resource group and add a VM:

Microsoft Azure, Tutorials screen

I clicked on +Add and then was redirected to a bunch of choices of which Ubuntu Server 18.04 LTS was one of the choices. When I chose that I was brought to the VM creation page and I chose these settings:

Microsoft Azure, Create a virtual machine screen

Now click Review + create. After reviewing the options you click create and Azure starts creating the VM. Azure starts creating the VM and then finishes:

Microsoft Azure, Your deployment is underway screen


Microsoft Azure, Your deployment is complete page

IP Address Config

Now navigate to the VM by clicking Go to resource. Below is the resources home page and we will need the public IP address so we can access the machine.

MedICI-CodaLab-Master page

One thing we should do before continuing is to set the IP address setting from Dynamic to Static. Grab the IP and also click Configure to edit DNS settings. Change the Dynamic to Static option and click save.

MedICI-CodaLab-Master-ip Configuration page

SSH Port

We need to configure the machine to be ssh accessible (port 22) and open http (port 80) (https://medium.com/techinpieces/practical-azure-how-to-enable-ssh-on-azure-vm-84d8fba8103e). Go to the VM and click Networking. On the right find the button Add inbound port rule. Change Destination port ranges to 22, Protocol to TCP, Priority to 330, and Name to anything you’d like:

Enable_22 screen for MedICI-CodaLab-Master nsg

Once you have that let’s open a terminal and get inside the machine by typing $ ssh <user>@<public IP address>.

If you get a message regarding encryption keys, enter y\yes as this is the first time signing into this machine.


HTTP Port

On the right find the button Add inbound port rule. Change Destination port ranges to 80, Protocol to TCP, Priority to 300, and Name to anything you’d like:



This will be important later when we deploy the site. Without this you cannot access the site.

Installing CodaLab

CodaLab has documentation regarding their preferred installation configuration. The two best sources are here:

  1. https://github.com/codalab/codalab-competitions/wiki
  2. https://codalab-competitions.readthedocs.io/en/latest/

The reason this is confusing is because the base code for codalab can be setup on different cloud providers (Google Cloud, AWS, Azure). They have made docker images for AWS and Google cloud that in theory should just run once some settings are established. If you are familiar with docker this seems reasonable. For the purposes of MedICI documentation we will go down the “Configure CodaLab from scratch (harder documentation)” found in the first of the above links.

Step 1 - Install Docker and Docker-Compose

Before we do any github cloning we need to install docker and docker-compose. First docker:

This link details installing docker for Ubuntu: https://docs.docker.com/install/linux/docker-ce/ubuntu/. For someone not familiar with linux there is a lot going on on that page. So for a basic installation this is what I ran in order:

Docker

  1. $ sudo apt-get remove docker docker-engine docker.io containerd runc
  2. $ sudo apt-get update
  3. $ sudo apt-get install \

    apt-transport-https \

    ca-certificates \

    curl \

    gnupg-agent \

    software-properties-common

  1. $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
  2. $ sudo apt-key fingerprint 0EBFCD88

$ sudo add-apt-repository \

   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \

   $(lsb_release -cs) \

   stable"

  1. $ sudo apt-get update
  2. $ sudo apt-get install docker-ce docker-ce-cli containerd.io
  3. $ sudo docker run hello-world


Now if we run ‘docker’, we should get the options list telling us how to use this command verifying it is installed and we can use it.


Docker-Compose

Next docker-compose. In the terminal on your new VM check to see if docker-compose is installed. If not which it is not for me then we need to install it. While it seems a ‘sudo apt install docker-compose’ should work, the CodaLab documentation from link “Configure CodaLab from scratch (harder documentation)” says to go here and install from dockers website. The command I’m actually running is this: sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose



Now we have to change the permissions in /usr/local/bin/docker-compose to use it: sudo chmod +x /usr/local/bin/docker-compose.


Now if we run ‘docker-compose’, we should get the options list telling us how to use this command verifying it is installed and we can use it.

Step 2 - Get the Source Code

The latest and greatest code is located here: https://github.com/codalab/codalab-competitions. Both sets of instructions want you to clone this repo locally to the VM. Before we get started I want to say since we will be making our own customizations down the road we will fork the repo CodaLab has so generously provided and do two things:

  1. Pull the latest changes from the master repo https://github.com/codalab/codalab-competitions into our own repo periodically to make sure we have the latest base code. (repo: https://github.com/QTIM-Lab/MedICI/)...If you need help doing this on your own see forking a repo. Since I’ve already done it, just follow along and clone from the QTIM-Lab/MedICI repo.
  2. Store our customizations on github so we can clone from this repo for future projects.


Clone the MedICI project. This follows CodaLabs instructions but with our github repo instead. Below is a picture of the commands I’ve executed from “Configure CodaLab from scratch (harder documentation)”. The green circles represent once inside the “src” folder, the changes that are not in the raw CodaLab github repo. This is the start of our customizations.


We can now push these changes into our branch.


At this point the app should be ready to deploy so we run ‘$ docker compose up -d’. However I have an error:



Turns out I needed to run with sudo or add myself to the docker group:

  • $ sudo docker-compose up -d

Or


The flag ‘-d’ means to run in detached mode. If you don’t use it you will notice your command prompt is in a live feed mode telling you everything that is going on with this docker.

Step 3 - Tweak The .env file and Azure Storage

Using this documentation( [2] from earlier): https://codalab-competitions.readthedocs.io/en/latest/

Create a Storage Account

Below is the blob storage section for Azure:

You may sign up for an Azure account here, then follow the directions below.

  1. You do not have to do this if you've already setup S3!
  2. Log on to the Azure Portal.
  3. From the Dashboard, click Storage accounts on the left.
  4. Click Add at the top of the page to create a new storage account.
  5. If you don't already have a subscription, create one now. The free trial requires a credit card, and deletes all your storage containers after 90 days, unless you upgrade to a different plan i.e. 'Pay as You Go'.
  6. Select the Classic storage account. Refer to the image below for settings.
  7. In the dashboard, click All Resources/All Subscriptions and then click on your username. Click Access Keys and Copy your account name and access key to .env under AZURE_ACCOUNT_NAME and AZURE_ACCOUNT_KEY.
  8. Within that same user account, click on Containers and Add a new container.
  9. Create a new container named "bundles". Set the Access to "Private".
  10. Add another container named "public". Set the Access to "Public Blob".
  11. Make sure the DEFAULT_FILE_STORAGE .env option is set to codalab.azure_storage.AzureStorage

This is the BLOB settings:

The CodaLab documentation leaves one final step out. We NEED to enable CORS. Click CORS in the above photo and add these settings (https://stackoverflow.com/questions/50785255/azure-storage-got-403-url-cors-not-enabled-or-no-matching-rule-found-for-thi):

If you don’t, uploading a competition will give you CORS errors in the browser:


Once applied uploading works:

Customizations

Codalab Instructions here: https://github.com/codalab/codalab-competitions/wiki/Customizing-your-self-hosted-instance


The only issue is that you can’t log into the customizer without being the super user. To make a superuser you need to log into the database as superuser. Django doesn’t have one by default and you need to create it. To do so ssh into your VM that hosts the site and run this command:


$ docker exec -it django python manage.py createsuperuser


Now use CodaLabs instructions and navigate to http://your-instance.com/customizer

Login with the newly made credentials. Choose a new file to be your logo.

  • No labels