Page History
Panel | ||||
---|---|---|---|---|
| ||||
|
Create a Windows Azure Ubuntu Virtual Machine
- Log in to the Microsoft Azure Portal
.Multiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include - Go to VM Screen, select "+ NEW" from the lower left corner of the screen.
On the pane that appears select, Compute > Virtual Machine > Quick Create.
Code Block DNS Name: pick something unique and memorable Image: Ubuntu Server 13.04 New Password/Confirm: provide a good password for the azureuser Region/Affinity Group: West US
Click Create a virtual machine.
Once your virtual machine is created, select it from the VM screen.
Go to the Endpoints menu.
(+ Add) an Endpoint at the bottom of the screen.
Code Block Provide the following information Name: HTTP Public Port: 80 Private Port: 8000
Restart the VM from the management console.
Troubleshooting note: You may get a message that Endpoint was successfully created but restart failed. In this case, go back to Dashboard (Management Console) and click Restart. In some cases, multiple attempts may be required.
At this point, it appears that a restart is not mandatory to continue with configuration.
Now you can login to it and start configuring things.
- Login to your VM via ssh as: azureuser <password provided at vm build - Step 3)
Things to do once you're in Ubuntu.
sudo apt-get update
enter password (azureuser password from Step 3)
sudo apt-get upgrade -y
sudo apt-get install git python-pip -y
sudo pip install virtualenv
Installation Prerequisites
Install Python 2.7. For current Debian-based Linux distributions (such as Ubuntu), BSD and Mac Python 2.7 is usually installed. However Redhat-based Linux distributions, such as RHEL and CentOS, are sometimes behind the curve and do not have Python 2.7. As of this writing, CentOS 6.4 is at Python 2.6, which is well on its way to EOL. Python 2.6 may work, but code will be written with 2.7 and 3.3+ in mind. Below are instructions for Ubuntu Linux 13.04+.
Code Block sudo apt-get install python2.7 python2.7-dev python-virtualenv
Install PIP
.Multiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include Code Block sudo apt-get install python-pip
Install virtualenv.
Code Block sudo apt-get install python-virtualenv
Install Git
.Multiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include Code Block sudo apt-get install git
Install the prerequisites for MySQL-Python
.Multiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include Code Block sudo apt-get install build-essential python-dev libmysqlclient-dev
Create Storage Containers
In order to test uploading and running bundles in CodaLab, you will need to have a Windows Azure storage account
Multiexcerpt include | ||||||
---|---|---|---|---|---|---|
|
Multiexcerpt include | ||||||
---|---|---|---|---|---|---|
|
- Log on to the Azure Portal
.Multiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include - In the left pane, click Storage.
- Select your storage account.
- At the bottom of the dashboard, click Manage Access Keys. Copy your access keys. You'll need them later.
- At the top of the dashboard page, click Containers.
- At the bottom of the Containers page click Add.
- Create a new container named "bundles". Set the Access to "Private".
- Add another container named "public". Set the Access to "Public Blob".
Add a Service Bus Namespace
Install azure-cli
.Multiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include In Ubuntu it can be installed using the following command:
Code Block sudo apt-get install nodejs-legacy sudo apt-get install npm sudo npm install -g azure-cli
To login run the following command:
Code Block azure login
Copy the code offered to you, above, and open a browser to http://aka.ms/devicelogin
. Enter the code, and then you are prompted to enter the username and password for the identity you want to use. When that process completes, the command shell completes the log in process.Multiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include From command line
azure sb namespace create <name> <location>
where<location>
can be "East US"- Log on to the Azure Portal
.Multiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include - In the left pane, click Service Bus.
- Select the service bus you just created.
- At the top of the screen click Queues.
- Click Create a new queue.
- Click Quick Create and create a new queue named "compute".
- Click Create A New Queue.
- At the bottom of the screen, click New, and create another queue named "response".
- In the left pane, click Service Bus.
- At the bottom of the page, click Connection Information.
- Copy the following connection information:
- Namespace name
- Default issuer
- Default key
Install CodaLab-CLI
Make sure you have the dependencies (Python 2.7 and virtualenv). If you're running Ubuntu:
Code Block sudo apt-get install python2.7 python2.7-dev python-virtualenv
Clone the CodaLab repository:
Code Block git clone https://github.com/codalab/codalab-cli cd codalab-cli
Run the setup script (will install things into a Python virtual environment):
Code Block ./setup.sh
Set your path to include CodaLab (add this line to your
.bashrc
):Code Block export PATH=$PATH:<path to codalab-cli>/codalab/bin
...
Code Block |
---|
. <path to codalab-cli>/rc |
Install CodaLab
the CodaLab repoMultiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include
from GitHub.Multiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include Clone your fork:
...
For more details and recommended practices, see Developer Guidelines
Multiexcerpt include | ||||||
---|---|---|---|---|---|---|
|
Configure Your Local Environment
In this segment, you will run the dev_setup script. This will install all dependencies and create a new virtual environment (venv) for CodaLab.
...
Code Block |
---|
source venv/bin/activate |
Install App Schema and Default Data
Now you are ready to install the application schema and default data into the database.
Set Up Databases
You can configure CodaLab to use either SQL Server or MySQL. Both of these require you to explicitly create a database.
Install MySQL Python
Follow these steps to install MySQL Python.
...
Open a terminal window.
Login to MySQL as root by typing the following command:
mysql -u root -p
Enter your root password when prompted.Use the following command to create a new database:
Code Block create database if not exists MySQL_DevDB;
Type
exit
to return to the terminal prompt.Finish up installation for Django
Code Block export DYLD_LIBRARY_PATH=/usr/local/mysql/lib:$DYLD_LIBRARY_PATH
Create a Local Configuration File
- Open your local configuration file (
local.py
). If there is nolocal.py
, save a copy oflocal_sample.py
namedlocal.py
in the same directory. In the
Azure storage
section, enter your Azure account details:Code Block DEFAULT_FILE_STORAGE = 'codalab.azure_storage.AzureStorage' AZURE_ACCOUNT_NAME = "<enter name>" AZURE_ACCOUNT_KEY = '<enter key>' AZURE_CONTAINER = '<enter container name>' PRIVATE_FILE_STORAGE = 'codalab.azure_storage.AzureStorage' PRIVATE_AZURE_ACCOUNT_NAME = "<enter name>" PRIVATE_AZURE_ACCOUNT_KEY = "<enter key>" PRIVATE_AZURE_CONTAINER = "<enter container name>" BUNDLE_AZURE_CONTAINER = "<enter the name of your bundle container>" BUNDLE_AZURE_ACCOUNT_NAME = PRIVATE_AZURE_ACCOUNT_NAME BUNDLE_AZURE_ACCOUNT_KEY = PRIVATE_AZURE_ACCOUNT_KEY
In the
Service Bus
section, enter your service bus connection information:Code Block SBS_NAMESPACE = '<enter the name of your service bus>' SBS_ISSUER = 'owner' SBS_ACCOUNT_KEY = '<enter value for 'default key'>'
Note title Important Do not change the values for
DEFAULT_FILE_STORAGE
andPRIVATE_FILE_STORAGE
, as these parameters contain the name of the Python class which implements the Azure storage back-end for Django.In the
DATABASES
section, enter the configuration settings for the database you want to use.SQL Server*
Code Block DATABASES = { 'default': { 'ENGINE': 'sql_server.pyodbc', 'NAME': 'somename', # Leaver user and password blank to use integrated security 'USER': '', 'PASSWORD': '', 'HOST': '(localdb)\\v11.0', 'PORT': '', 'OPTIONS': { 'driver': 'SQL Server Native Client 11.0', } }
MySQL
Code Block DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'MySQL_DevDB', 'USER': 'someuser', 'PASSWORD': 'somepassword', 'HOST': 'someserver', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP. 'PORT': '', # Set to empty string for default. }
Info | ||||||||
---|---|---|---|---|---|---|---|---|
If you want to use MySQL
|
Create a Local Compute Configuration File
- Open
codalab/codalab/codalabtools/compute/sample.config
. - Save a copy of
sample.config
named.codalabconfig
in the same directory. - Open
.codalabconfig
.
...
Code Block |
---|
compute-worker: azure-storage: account-name: "your account name" account-key: "your account key" azure-service-bus: namespace: "your namespace" key: "your secret key" issuer: "owner" listen-to: "name of queue" local-root: "D:\\Temp" |
Enable CORS (Cross-Origin Resource Sharing) on Blob Storage
In order to work with competitions and bundles in your local development environment, you will need to manually enable Cross-origin resource sharing (CORS). CORS is a mechanism that allows many resources on a web page to be requested from another domain outside the domain the resource originated from. Web Browsers commonly apply same origin restriction policy to network requests. CORS relaxes these restrictions allowing domains to give each other permissions for accessing each other's resources. CORS is supported for Blob, Table and Queue services and can be enabled for each service using Azure SDK for python.
...
Code Block |
---|
python scripts/cors-enable.py |
Initialize the Database
To initialize the database, you will need to run a few standard Django commands
Multiexcerpt include | ||||||
---|---|---|---|---|---|---|
|
...
Code Block |
---|
python scripts/users.py python scripts/competitions.py |
Configure the Bundle Service to Run Locally
This note will explain how to run the CodaLab Django web site and the CodaLab bundle service (cl server) side-by-side on a single machine. This setup is useful for people doing development, especially when working at the interface between the two systems.
...
- Take the "server" block in the output and insert it into your CLI config file (
.codalab\config.json
in your home directory). Since the values forapp_id
andapp_key
can be long, make sure to remove any line breaks that resulted from copying from the command prompt. - Run the Django web site. Make sure it uses port 8000. Note for Visual Studio users: with the latest Python Tools for Visual Studio, you can set the port number in the Debug tab of the project properties.
- Activate your virtual environment and then run the bundle service:
cl server
.
- Take the "server" block in the output and insert it into your CLI config file (
Start the Web Server
Use the following command to start the CodaLab server locally.
python manage.py runserver
Open a browser and navigate to http://127.0.0.1:8000
to preview the site.Multiexcerpt include nopanel true MultiExcerptName ExitDisclaimer PageWithExcerpt wikicontent:Exit Disclaimer to Include
...
Info |
---|
If you experience database errors try deleting the database file (\codalab\codalab\dev_db.*) and run syncdb again. After creating a new database be sure to run |
Start the Worker Roles
In order to test competitions and bundles locally, you'll need to run the compute worker roles. There are two worker.py
scripts that you will need to run.
- Open a command prompt and activate your CodaLab virtual environment.
Start the first compute worker as shown here:
Code Block cd codalab python worker.py
Open a second command prompt and activate your CodaLab virtual environment.
Start the second compute worker as shown here:
Code Block cd codalabtools\compute python worker.py
If you plan to test competitions locally, open a third command prompt and activate the virtual environment for the CodaLab CLI, then start the bundle server:
Code Block cl server