NIH | National Cancer Institute | NCI Wiki  

WIKI MAINTENANCE NOTICE

Please be advised that NCI Wiki will be will be undergoing maintenance on Monday, June 24th between 1000 ET and 1100 ET.
Wiki will remain available, but users may experience screen refreshes or HTTP 502 errors during the maintenance period. If you encounter these errors, wait 1-2 minutes, then refresh your page.

If you have any questions or concerns, please contact the CBIIT Atlassian Management Team.

Versions Compared

Old Version 13

changes.mady.by.user Klinger, Carolyn (NIH/NCI) [C]

Saved on

compared with

New Version 14

changes.mady.by.user Klinger, Carolyn (NIH/NCI) [C]

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Make sure you have the dependencies (Python 2.7 and virtualenv). If you're running Ubuntu:

    Code Block
    sudo apt-get install python2.7 python2.7-dev python-virtualenv

  2. Clone the CodaLab repository:

    Code Block
       git clone https://github.com/codalab/codalab-cli
cd codalab-cli

  3. Run the setup script (will install things into a Python virtual environment):

    Code Block
      ./setup.sh

  4. Set your path to include CodaLab (add this line to your .bashrc):

    Code Block
    export PATH=$PATH:<path to codalab-cli>/codalab/bin

Optional: include some handy macros (add this line to your .bashrc):

Code Block
. <path to codalab-cli>/rc

 

Install CodaLab

  1. Fork

    Multiexcerpt include
    nopaneltrue
    MultiExcerptNameExitDisclaimer
    PageWithExcerptwikicontent:Exit Disclaimer to Include
    the CodaLab repo
    Multiexcerpt include
    nopaneltrue
    MultiExcerptNameExitDisclaimer
    PageWithExcerptwikicontent:Exit Disclaimer to Include
    from GitHub.

  2. Clone your fork:

Code Block
git clone https://github.com/<username>/codalab.git

For more details and recommended practices, see Developer Guidelines

Multiexcerpt include
nopaneltrue
MultiExcerptNameExitDisclaimer
PageWithExcerptwikicontent:Exit Disclaimer to Include
.

...

  1. Run the dev_setup script.

    Windows

     

    Code Block
          cd codalab
.\dev_setup.bat

    If you are going to use SQL Server as a database, you will need to install the Python PyODBC library

    Multiexcerpt include
    nopaneltrue
    MultiExcerptNameExitDisclaimer
    PageWithExcerptwikicontent:Exit Disclaimer to Include
    . Before running dev_setup, you can download the installer, then run the setup script as follows (assuming the installer was downloaded at the root of a D drive):

    Code Block
          cd codalab
dev_setup.bat D:\pyodbc-3.0.7.win-amd64-py2.7.exe

    Linux

     

    Code Block
         cd codalab
source ./dev_setup.sh

  2. Activate the virtual environment.

Windows

Code Block

...

 venv\Scripts\activate

...

Linux
 Code Block
 
...
source venv/bin/activate
...
Install App Schema and Default Data
Now you are ready to install the application schema and default data into the database.
...
  1. Open a Windows command prompt.
  2. Navigate to the virtual environment (venv) for CodaLab and use the following command to install MySql-Python.
    easy_install mysql-python
  3. Launch the MySQL Command Line Client.
  4. Use the following command to create a new database:
     Code Block
    create database if not exists MySQL_DevDB;
Linux
  1. Open a terminal window.
  2. Run the following command to install MySQL:sudo 
     Code Block
     sudo apt-get install mysql-server
  3. Login to MySQL as root by typing the following command:
    mysql -u root -p Enter your root password when prompted.
  4. Use the following command to create a new database:
     Code Block
    create 
     create database if not exists MySQL_DevDB;
  5. Type exit to return to the terminal prompt.
...
  1. Open a terminal window.
  2. Login to MySQL as root by typing the following command:
    mysql -u root -p Enter your root password when prompted.
  3. Use the following command to create a new database:
     Code Block
    create database if not exists MySQL_DevDB;

  4. Type exit to return to the terminal prompt.
  5. Finish up installation for Django
     Code Block
    export DYLD_LIBRARY_PATH=/usr/local/mysql/lib:$DYLD_LIBRARY_PATH
Create a Local Configuration File
  1. Open your local configuration file (local.py). If there is no local.py, save a copy of local_sample.py named local.py in the same directory.
  2. In the Azure storage section, enter your Azure account details:
     Code Block
    DEFAULT_FILE_STORAGE = 'codalab.azure_storage.AzureStorage'
AZURE_ACCOUNT_NAME = "<enter name>"
AZURE_ACCOUNT_KEY = '<enter key>'
AZURE_CONTAINER = '<enter container name>'

PRIVATE_FILE_STORAGE = 'codalab.azure_storage.AzureStorage'
PRIVATE_AZURE_ACCOUNT_NAME = "<enter name>"
PRIVATE_AZURE_ACCOUNT_KEY = "<enter key>"
PRIVATE_AZURE_CONTAINER = "<enter container name>"

BUNDLE_AZURE_CONTAINER = "<enter the name of your bundle container>"
BUNDLE_AZURE_ACCOUNT_NAME = PRIVATE_AZURE_ACCOUNT_NAME
BUNDLE_AZURE_ACCOUNT_KEY = PRIVATE_AZURE_ACCOUNT_KEY
          

    
  3. In the Service Bus section, enter your service bus connection information:
     Code Block
    SBS_NAMESPACE = '<enter the name of your service bus>'
SBS_ISSUER = 'owner'
SBS_ACCOUNT_KEY = '<enter value for 'default key'>'
     Note
    titleImportant
    Do not change the values for DEFAULT_FILE_STORAGE and PRIVATE_FILE_STORAGE, as these parameters contain the name of the Python class which implements the Azure storage back-end for Django.
  4. In the DATABASES section, enter the configuration settings for the database you want to use.
    SQL Server*
     Code Block
    DATABASES = {
    'default': {
        'ENGINE': 'sql_server.pyodbc',
        'NAME': 'somename',
        # Leaver user and password blank to use integrated security
        'USER': '',
        'PASSWORD': '',
        'HOST': '(localdb)\\v11.0', 
        'PORT': '',
        'OPTIONS': {
           'driver': 'SQL Server Native Client 11.0',
        }
    }
    MySQL
     Code Block
    DATABASES = {
    'default': {
        'ENGINE':  'django.db.backends.mysql',
        'NAME': 'MySQL_DevDB',
        'USER': 'someuser',
        'PASSWORD': 'somepassword',
        'HOST': 'someserver', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
        'PORT': '',           # Set to empty string for default.
    }
...
  1. Open codalab/codalab/codalabtools/compute/sample.config.
  2. Save a copy of sample.config named .codalabconfig in the same directory.
  3. Open .codalabconfig.
In the compute-worker section, enter the configuration settings for the storage account and the compute queue.
 Code Block
 compute-worker:
    azure-storage:
        account-name: "your account name"
        account-key: "your account key"
    azure-service-bus:
        namespace: "your namespace"
        key: "your secret key"
        issuer: "owner"
        listen-to: "name of queue"
    local-root: "D:\\Temp"
Enable CORS (Cross-Origin Resource Sharing) on Blob Storage
In order to work with competitions and bundles in your local development environment, you will need to manually enable Cross-origin resource sharing (CORS). CORS is a mechanism that allows many resources on a web page to be requested from another domain outside the domain the resource originated from. Web Browsers commonly apply same origin restriction policy to network requests. CORS relaxes these restrictions allowing domains to give each other permissions for accessing each other's resources. CORS is supported for Blob, Table and Queue services and can be enabled for each service using Azure SDK for python.
  1. Make sure you have a valid management certificate to connect to the Service Management endpoint. This tutorial explains how to create a certificate and upload it to the Azure management portal: http://azure.microsoft.com/en-us/documentation/articles/cloud-services-python-how-to-use-service-management/
     Multiexcerpt include
    nopaneltrue
    MultiExcerptNameExitDisclaimer
    PageWithExcerptwikicontent:Exit Disclaimer to Include
    .
    Open codalab/codalabtools/compute/enableCORS.py in a text editor, and update account_name and account_key with the account name and key for your blob storage account:
     Code Block
    import sys
import yaml
from os.path import dirname, abspath
# Add codalabtools to the module search path
sys.path.append(dirname(dirname(abspath(__file__))))

from codalabtools.azure_extensions import (Cors,CorsRule,set_storage_service_cors_properties)

account_name = "<your blob storage account name>"
account_key = "<your blob storage account key>"
cors_rule = CorsRule()
cors_rule.allowed_origins = '*' # this is fine for dev setup
cors_rule.allowed_methods = 'PUT'
cors_rule.exposed_headers = '*'
cors_rule.allowed_headers = '*'
cors_rule.max_age_in_seconds = 1800
cors_rules = Cors()
cors_rules.cors_rule.append(cors_rule)
set_storage_service_cors_properties(account_name, account_key, cors_rules)
          
    
  2. Save your changes, Save your changes, activate your virtual environment and run the script:
Windows
 Code Block
 python 
...
scripts\cors-enable.py
...
Linux
 Code Block
 python 
...
scripts/cors-enable.py
...
Initialize the Database
To initialize the database, you will need to run a few standard Django commands
 Multiexcerpt include
nopaneltrue
MultiExcerptNameExitDisclaimer
PageWithExcerptwikicontent:Exit Disclaimer to Include
, and the CodaLab database initialization script.
...
  1. Run the following commands to initialize the database.
    Windows
     Code Block
    cd codalab
python manage.py validate
python manage.py syncdb --migrate
python scripts\initialize.py
    Linux
     Code Block
    cd codalab
python manage.py validate
python manage.py syncdb --migrate
python scripts/initialize.py
  2. Run tests to verify that everything is working.
    python manage.py test
Optional: Populate the site with some sample data.
Windows
 Code Block
python scripts\users.py
python scripts\competitions.py
Linux
 Code Block
python scripts/users.py
python scripts/competitions.py
Configure the Bundle Service to Run Locally
...
  1. Use the following command to start the CodaLab server locally.
          python manage.py runserver

    
  2. Open a browser and navigate to http://127.0.0.1:8000
     Multiexcerpt include
    nopaneltrue
    MultiExcerptNameExitDisclaimer
    PageWithExcerptwikicontent:Exit Disclaimer to Include
     to preview the site.
When your next coding session comes along, remember to work in the virtual environment you created:
Windows
...

 Code Block
 venv\Scripts\activate
...
Linux
...

 Code Block
source venv/bin/activate
...
 Info
If you experience database errors try deleting the database file (\codalab\codalab\dev_db.*) and run syncdb again. After creating a new database be sure to run initialize.py in the scripts folder in order to insert initial data required by the app.
Start the Worker Roles
In order to test competitions and bundles locally, you'll need to run the compute worker roles. There are two worker.py scripts that you will need to run.
  1. Open a command prompt and activate your CodaLab virtual environment.
  2. Start the first compute worker as shown here:
          
     Code Block
    cd codalab
python worker.py
     
    
  3. Open a second command prompt and activate your CodaLab virtual environment.
  4. Start the second compute worker as shown here:
          
     Code Block
    cd codalabtools\compute
python worker.py
    

    
  5. If you plan to test competitions locally, open a third command prompt and activate the virtual environment for the CodaLab CLI, then start the bundle server:
          
     Code Block
    cl server
    
    
 
Execution Using Docker
Every execution on CodaLab (should ideally) happen in a docker 
 Multiexcerpt include
nopaneltrue
MultiExcerptNameExitDisclaimer
PageWithExcerptwikicontent:Exit Disclaimer to Include
 container, which provides a standardized Linux environment that is lighter-weight than a full virtual machine.
The current official docker image is codalab/ubuntu, which consists of Ubuntu 14.04 plus some standard packages. See the CodaLab docker registry.
To install docker on your local machine (either if you want see what's actually in the environment or to run your own local CodaLab instance), follow these instructions 
 Multiexcerpt include
nopaneltrue
MultiExcerptNameExitDisclaimer
PageWithExcerptwikicontent:Exit Disclaimer to Include
:
 Code Block
sudo sh -c "echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
sudo apt-get update
sudo apt-get install lxc-docker
sudo useradd $USER docker
 
Then, to test out your environment, open a shell (the first time you do this, it will take some time to download the image):                
 Code Block
 docker 
docker 
run -t -i codalab/ubuntu:1.8


              
Now, let us integrate docker into CodaLab. First, we need to setup a job scheduling system (that manages the deployment of runs on machines). Note that CodaLab itself doesn't do this, so that it can be easily integrated into different systems. An easy way to set this up is to use q from Percy Liang's fig package:
 Code Block
git clone https://github.com/percyliang/fig
# Add fig/bin/q to your $PATH
q -mode master   # Run in a different terminal
q -mode worker   # Run in a different terminal
Now, let us tell CodaLab to use q and run things in docker (these two things are orthogonal choices). Edit the .codalab/config.json as follows:
 Code Block
 "workers": {
    "q": {
        "verbose": 1,
        "docker_image": "codalab/ubuntu:1.8"
        "dispatch_command": "python $CODALAB_CLI/scripts/dispatch-q.py"
    }
}
 
To test it out:
 Code Block
cl work-manager -t q                 # Run in a different terminal
cl run 'cat /proc/self/cgroup' -t    # Should eventually print out lines containing the string `docker`