Page History

...

1. Log in to Windows Azure Management Portal.
2. Go to VM Screen, select "+ NEW" from the lower left corner of the screen.

3. On the pane that appears select, Compute > Virtual Machine > Quick Create.

   Code Block
   DNS Name: pick something unique and memorable Image: Ubuntu Server 13.04 New Password/Confirm: provide a good password for the azureuser Region/Affinity Group: West US

4. Click Create a virtual machine.

5. Once your virtual machine is created, select it from the VM screen.

6. Go to the Endpoints menu.

   (+ Add) an Endpoint at the bottom of the screen.

   Code Block

   Provide

   Provide the following information Name: HTTP Public Port: 80 Private Port: 8000

7. Restart the VM from the management console.

   Troubleshooting note: You may get a message that Endpoint was successfully created but restart failed. In this case, go back to Dashboard (Management Console) and click Restart. In some cases, multiple attempts may be required.

   At this point, it appears that a restart is not mandatory to continue with configuration.

8. Now you can login to it and start configuring things.

9. Login to your VM via ssh as: azureuser <password provided at vm build - Step 3)

10. Things to do once you're in Ubuntu.

    1. sudo apt-get update
       1. enter password (azureuser password from Step 3)
    2. sudo apt-get upgrade -y
    3. sudo apt-get install git python-pip -y
    4. sudo pip install virtualenv

...

1. Install azure-cli.

   In Ubuntu it can be installed using the following command:sudo

   Code Block
   sudo apt-get install nodejs-legacy

   
   

   sudo apt-get install npm

   
   

   sudo npm install -g azure-cli

    

   To login run the following command:

   azure login

   Copy the code offered to you, above, and open a browser to http://aka.ms/devicelogin. Enter the code, and then you are prompted to enter the username and password for the identity you want to use. When that process completes, the command shell completes the log in process.

2. From command line azure sb namespace create <name> <location> where <location> can be "East US"

3. Log on to the Azure Portal.
4. In the left pane, click Service Bus.
5. Select the service bus you just created.
6. At the top of the screen click Queues.
7. Click Create a new queue.
8. Click Quick Create and create a new queue named "compute".
9. Click Create A New Queue.
10. At the bottom of the screen, click New, and create another queue named "response".
11. In the left pane, click Service Bus.
12. At the bottom of the page, click Connection Information.
13. Copy the following connection information:
    • Namespace name
    • Default issuer
    • Default key

...

1. Open your local configuration file (local.py). If there is no local.py, save a copy of local_sample.py named local.py in the same directory.

2. In the Azure storage section, enter your Azure account details:

   Code Block

   DEFAULT_FILE_STORAGE = 'codalab.azure_storage.AzureStorage' AZURE_ACCOUNT_NAME = "<enter name>" AZURE_ACCOUNT_KEY = '<enter key>' AZURE_CONTAINER = '<enter container name>' PRIVATE_FILE_STORAGE = 'codalab.azure_storage.AzureStorage' PRIVATE_AZURE_ACCOUNT_NAME = "<enter name>" PRIVATE_AZURE_ACCOUNT_KEY = "<enter key>" PRIVATE_AZURE_CONTAINER = "<enter container name>" BUNDLE_AZURE_CONTAINER = "<enter the name of your bundle container>" BUNDLE_AZURE_ACCOUNT_NAME = PRIVATE_AZURE_ACCOUNT_NAME BUNDLE_AZURE_ACCOUNT_KEY = PRIVATE_AZURE_ACCOUNT_KEY

3. In the Service Bus section, enter your service bus connection information:

   Code Block
   SBS_NAMESPACE = '<enter the name of your service bus>' SBS_ISSUER = 'owner' SBS_ACCOUNT_KEY = '<enter value for 'default key'>'

   Note
   title Important

   :

   Do not change the values for DEFAULT_FILE_STORAGE and PRIVATE_FILE_STORAGE, as these parameters contain the name of the Python class which implements the Azure storage back-end for Django.

4. In the DATABASES section, enter the configuration settings for the database you want to use.

   SQL Server*

   DATABASES = {
       'default': {
           'ENGINE': 'sql_server.pyodbc',
           'NAME': 'somename',
           # Leaver user and password blank to use integrated security
           'USER': '',
           'PASSWORD': '',
           'HOST': '(localdb)\\v11.0', 
           'PORT': '',
           'OPTIONS': {
              'driver': 'SQL Server Native Client 11.0',
           }
       }
   

   MySQL
   

   Code Block
   DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'MySQL_DevDB', 'USER': 'someuser', 'PASSWORD': 'somepassword', 'HOST': 'someserver', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP. 'PORT': '', # Set to empty string for default. }

...

Create a Local Compute Configuration File

1. Open codalab/codalab/codalabtools/compute/sample.config.
2. Save a copy of sample.config named .codalabconfig in the same directory.
3. Open .codalabconfig.

4. In the compute-worker section, enter the configuration settings for the storage account and the compute queue.

   compute

   Code Block
   compute-worker: azure-storage: account-name: "your account name" account-key: "your account key" azure-service-bus: namespace: "your namespace" key: "your secret key" issuer: "owner" listen-to: "name of queue" local-root: "D:\\Temp"

Enable CORS (Cross-Origin Resource Sharing) on Blob Storage

...

1. Make sure you have a valid management certificate to connect to the Service Management endpoint. This tutorial explains how to create a certificate and upload it to the Azure management portal: http://azure.microsoft.com/en-us/documentation/articles/cloud-services-python-how-to-use-service-management/.

2. Open codalab/codalabtools/compute/enableCORS.py in a text editor, and update account_name and account_key with the account name and key for your blob storage account:

   Code Block

   import sys import yaml from os.path import dirname, abspath # Add codalabtools to the module search path sys.path.append(dirname(dirname(abspath(__file__)))) from codalabtools.azure_extensions import (Cors,CorsRule,set_storage_service_cors_properties) account_name = "<your blob storage account name>" account_key = "<your blob storage account key>" cors_rule = CorsRule() cors_rule.allowed_origins = '*' # this is fine for dev setup cors_rule.allowed_methods = 'PUT' cors_rule.exposed_headers = '*' cors_rule.allowed_headers = '*' cors_rule.max_age_in_seconds = 1800 cors_rules = Cors() cors_rules.cors_rule.append(cors_rule) set_storage_service_cors_properties(account_name, account_key, cors_rules)

3. Save your changes, activate your virtual environment and run the script:

   Windows

   python scripts\cors-enable.py
   

   Linux

   python scripts/cors-enable.py
   

...

1. Run the following commands to initialize the database.

   Windows
   

   
   

   cd codalab
   python manage.py validate
   python manage.py syncdb --migrate
   python scripts\initialize.py
   

   Linux
   

   
   

   cd codalab
   python manage.py validate
   python manage.py syncdb --migrate
   python scripts/initialize.py
   

2. Run tests to verify that everything is working.

   python manage.py test

3. Optional: Populate the site with some sample data.

   Windows
   

   
   

   python scripts\users.py
   python scripts\competitions.py
   

   Linux
   

   Code Block
   python scripts/users.py python scripts/competitions.py

Configure the Bundle Service to Run Locally

...

1. Begin by enabling the Worksheet feature in your web site. In your Django settings (local.py), add:

   Code Block
   PREVIEW_WORKSHEETS = True BUNDLE_SERVICE_URL = "http://localhost:2800" # CODE_PATH points to local source code for bundles repo. Path is relative to this file. BUNDLE_SERVICE_CODE_PATH = "..\\..\\..\\..\\codalab-cli" if len(BUNDLE_SERVICE_CODE_PATH) > 0: sys.path.append(join(dirname(abspath(__file__)), BUNDLE_SERVICE_CODE_PATH)) codalab.__path__ = extend_path(codalab.__path__, codalab.__name__)

These additional elements say to enable Worksheets using the bundle service running at the given URL (in the next steps we'll cover how to start the bundle service with the cl server command). Finally, the code in apps\web\bundles.py will need to import modules from the codalab-cli project. Therefore, we extend the Python path by pointing to the CLI code on your machine. If you have repos codalab and codalab-cli checked in folders which are siblings, the relative BUNDLE_SERVICE_CODE_PATH given above should work for your setup.

The Django web site is a client of the bundle service. For example, to get a list of worksheets, the Django web site makes a call to the list_worksheets API exposed by the bundle service. However, to handle authorization, the bundle service must be able to make calls back to the web site, which hosts the OAuth server. As a result, the bundle service is a trusted OAuth client of the web site. To setup this trusted relationship:

1. a. Create a user on the web site. Typically, I call this user codalab. Treat it as an admin user even though the CodaLab site doesn't really have admin roles today.

   b. Activate your virtual environment and run the script: codalab\codalab\scripts\sample_cl_server_config.py. The script will generate an output of the form:

   Code Block
   Checking that confidential client exists for user codalab Client already exists. Add the following server block to your CLI config: "server": { "auth": { "address": "http://localhost:8000", "app_id": "5m <snip> Le", "app_key": "_b <snip> !_", "class": "OAuthHandler" }, "class": "SQLiteModel", "host": "localhost", "port": 2800 }

   Take the "server" block in the output and insert it into your CLI config file (.codalab\config.json in your home directory). Since the values for app_id and app_key can be long, make sure to remove any line breaks that resulted from copying from the command prompt.

   c. Run the Django web site. Make sure it uses port 8000. Note for Visual Studio users: with the latest Python Tools for Visual Studio, you can set the port number in the Debug tab of the project properties.

   d. Activate your virtual environment and then run the bundle service: cl server.

...

sudo sh -c "echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
sudo apt-get update
sudo apt-get install lxc-docker
sudo useradd $USER docker

docker run -t -i codalab/ubuntu:1.8

git clone https://github.com/percyliang/fig
# Add fig/bin/q to your $PATH
q -mode master   # Run in a different terminal
q -mode worker   # Run in a different terminal

 "workers": {
    "q": {
        "verbose": 1,
        "docker_image": "codalab/ubuntu:1.8"
        "dispatch_command": "python $CODALAB_CLI/scripts/dispatch-q.py"
    }
}

cl work-manager -t q                 # Run in a different terminal
cl run 'cat /proc/self/cgroup' -t    # Should eventually print out lines containing the string `docker`