![]() |
Page History
If you decide to use the CLU to perform various activities in DME, the following steps are pre-requisites:
- If you are using Windows, install Cygwin as described in the following site: https://www.cygwin.com/
Multiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include
For best results, run Cygwin.bat from C:\cygwin64\ rather than from the Windows Start menu. Check which version of Java is currently installed:
Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true java –version
The command output should display the version of Java. DME requires Java 1.8.x.
- If Java 1.8.x is not installed, perform the following stepsPrepare Java:
Download Java 1.8.x from the following page:
https://www.java.com/en/download/
Multiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include - Install Java, as described in the following installation instructions:
https://www.java.com/en/download/help/index_installing.xmlMultiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include - Update your PATH system variable with the Java executable folder . For example:
(such as c:/jdk1.8_13/bin
bin). For details, refer to the following instructions:
https://java.com/en/download/help/path.xmlMultiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include Check whether you have performed the previous steps correctly : Open a cmd or shell window and type the following command. The command output should display the version of Java you intended to install:
Code Block java –version
by repeating the java -version command.
If the latest version of JDK is not installed, perform the following steps:
- Download JDK from the following page:
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.htmlMultiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include - Install JDK. For information, refer to the following page
- If you are using Windows, decide how you will use the Command Line Utilities. Choose one of the following options: (Question: What are the pros and cons of each?)
Install Cygwin as described in the following site:
https://www.cygwinoracle.com/ - Open a terminal cmd window. Use the following command /technetwork/java/javase/documentation/index.html
Multiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include - Download JDK from the following page:
In a cmd or shell window, type the following command in your working directory to clone DME utilities to your local machine:
codePanel | |||||
---|---|---|---|---|---|
| |||||
|
|
|
The system creates a subdirectory (HPC_DME_APIs) containing all files from that GitHub repository.
In the HPC_DME_APIs/utils/ folder, create a copy of the sample configuration file (hpcdme.properties-sample) and rename it (as hpcdme.properties). Edit your hpcdme.properties file to make the following configuration changes:
Property | Instructions | |||||
---|---|---|---|---|---|---|
Server settings | If you want to connect to the UAT tier, uncomment the UAT server settings and comment out the Production server settings. By default, the Production server settings are enabled. | |||||
hpc.ssl.keystore.password | Contact NCIDataVault@mail.nih.gov to request this password. | |||||
hpc.user | Specify your NIH user ID. | |||||
hpc.default.globus.endpoint | If you intend to transfer files using Globus, specify the UUID of the Globus endpoint that you want to use. (The following instructions may be helpful: Copying the Globus Shared Endpoint UUID.) | |||||
hpc.server.proxy.port | If you intend to use Biowulf, specify the proxy port. To determine the correct port, run the following echo command:
|
https://github.com/CBIIT/HPC_DME_APIs/blob/master/utils/README.md
...
solid Clipboard AllowLineWrap true echo $http_proxy
The output includes the port. For example, if the output of the echo command is http://dtn08-e0:3128, then your port is 3128.
hpc.server.proxy.url If you intend to use Biowulf, specify the proxy URL. To determine the correct URL, ping the output of the echo command. For example:
Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true ping dtn08-e0
The output includes an IP address. Specify that IP address as the URL.
hpc.error-log.dir
hpc.log.fileIf you are using Windows, consider specifying a non-default location for the error log file. For example:
Code Block hpc.error-log.dir=log hpc.log.file=log/hpc-cli.log
.
Configure environment variables:
Navigate to your ~/.bashrc or ~/.profile file, where ~ represents your home directory. For example:
Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true C:\cygwin64\home\<username>
If you are running CLU from Biowulf or Helix, add the following command to your ~/.bashrc or ~/.profile file:
Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true module load java/11.0.21
Edit your ~/.bashrc or ~/.profile file to append the following commands, where <client_utils_home> is the absolute path to the HPC_DME_APIs\utils\ folder. The following commands define the HPC_DM_UTILS environment variable and update the PATH environment variable to include the path to the scripts folder:
Code Block export HPC_DM_UTILS="<client_utils_home>" export PATH=$HPC_DM_UTILS/scripts/:$PATH
If you are using Cygwin on Windows, run the following commands in Cygwin to change end-of-line delimiters from Windows standard to Unix standard in any files involved in CLU setup:
Adjust the line delimiters in the file with your export commands (~/.bashrc or ~/.profile). For example:
Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true sed -i "s/\r$//" ~/.bashrc
Adjust the line delimiters in all files in the scripts folder. Use the following command:
Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true sed -i "s/\r$//" $HPC_DM_UTILS/scripts/*
To make the above changes take effect in current Bash session, source the modified file (~/.bashrc or ~/.profile). For example:
Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true source ~/.bashrc
Generate a DME API authentication token, as described in Generating a DME API Authentication Token via the CLU.
- If you want to use Globus or AWS S3, follow the instructions in the following pages:
To view a list of DME CLU commands, type dm_list_commands
. To view usage information for any DME CLU command, type the command with the help (-h) parameter. For detailed information on each of the available DME CLU commands, refer to the following pages:
Getting Started
Generating a DME API Authentication Token via the CLU — dm_generate_token
- DME CLU Quick Reference - (most frequently-used commands)
Finding Data
- Searching for Data via the CLU — dm_query_dataobject
Using Bookmarks
Adding a Bookmark via the CLU — dm_add_bookmark
- Updating a Bookmark via the CLU — dm_update_bookmark
Retrieving Metadata
Retrieving the Metadata of a Collection via the CLU — dm_get_collection
Retrieving the Metadata of a Data File via the CLU — dm_get_dataobject
Registering Data
Creating an Empty Collection via the CLU — dm_register_collection
- Registering a Data File via the CLU — dm_register_dataobject, dm_register_dataobject_presigned, dm_register_dataobject_multipart
Registering Directory Contents from Your Local System via the CLU — dm_register_directory
Updating Collection Metadata via the CLU — dm_register_collection
- Updating Data File Metadata via the CLU — dm_register_dataobject
Downloading Data
- Downloading One or More Files via the CLU — dm_download_dataobject, dm_download_dataobject_presigned, dm_download_dataobject_globus, dm_download_dataobject_s3, dm_download_dataobject_aspera
- Using dm_download_collection — dm_download_collection
- Using dm_download_collection_globus — dm_download_collection_globus
- Using dm_download_collection_s3 — dm_download_collection_s3
- Using dm_download_collection_aspera — dm_download_collection_aspera
- Retrying a Failed or Canceled Download via the CLU — dm_download_dataobject_retry, dm_download_collection_retry, dm_download_list_retry
Using DME as an Administrator
Registering a User via the CLU — dm_register_user
Adding a Bookmark and Permissions via the CLU — dm_add_bookmark
- Updating a Bookmark and Permissions via the CLU — dm_update_bookmark
- Moving or Renaming a Collection or File via the CLU — dm_rename
- Deleting Data via the CLU — dm_delete_datafile, dm_delete_collection
Show If | |||||||||
---|---|---|---|---|---|---|---|---|---|
| |||||||||
|
Important Note for Mac Version 10.12.4/10.12.6
If you want to use DME CLU on a Mac (version 10.12.4/10.12.6) and the version of curl on your machine is OpenSSL 0.9.8zh 14 Jan 2016, perform the following steps to rebuild curl.
From a terminal prompt, execute the following command:
Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)
This command downloads and installs Homebrew, a package manager to download, build, and install Mac software.
Follow the prompts. The system might request your username/password multiple times.
Execute the following command:
Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true brew install curl --with-openssl
This command invokes Homebrew to build and install curl in /usr/local/opt with the latest version of openssl. At the end of the installation, a message appears indicating that this is keg-only. It provides the command to update the PATH so that the system uses your new curl installation, rather than the Mac-supplied one.
Execute the command as provided at the end of curl installation, which should be the following command:
Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true export PATH="/usr/local/opt/curl/bin:$PATH"' >> ~/.bash_profile
Confirm the installation: Open a new terminal and check the version of curl with the following command:
Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true curl –-version
The following information should appear:
Code Block curl 7.58.0 (x86_64-apple-darwin16.7.0) libcurl/7.58.0 OpenSSL/1.0.2n zlib/1.2.8 Release-Date: <today’s date or the date you install your new curl> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy