Page History

Versions Compared

Key

This line was added.
This line was removed.
Formatting was changed.

Comment: For HPCDATAMGM-1878: Added a step.

If you decide to use the CLU to perform various activities in DME, the following steps are pre-requisites:

If you are using Windows, install Cygwin as described in the following site: https://www.cygwin.com/

Multiexcerpt include
MultiExcerptName ExitDisclaimer
nopanel true
PageWithExcerpt wikicontent:Exit Disclaimer to Include

For best results, run Cygwin.bat from C:\cygwin64\ rather than from the Windows Start menu.
Check which version of Java is currently installed:
Panel
borderColor silver
borderStyle solid
Clipboard
AllowLineWrap true
java –version
The command output should display the version of Java. DME requires Java 1.8.x.
If Java 1.8.x is not installed, perform the following stepsPrepare Java:
1. Download Java 1.8.x from the following page:
  
  https://www.java.com/en/download/
  multiexcerpt-include
  MultiExcerptName ExitDisclaimer
  nopanel true
  PageWithExcerpt
  wikicontent:Exit Disclaimer to Include
  wikicontent:Exit Disclaimer to Include
2. Install Java, as described in the following installation instructions:
  https://www.java.com/en/download/help/index_installing.xml wikicontent:
  Include Page
  Multiexcerpt include
  MultiExcerptName ExitDisclaimer
  nopanel true
  PageWithExcerpt
  Exit Disclaimer to Include
  wikicontent:Exit Disclaimer to Include
3. Update your PATH system variable with the Java executable folder . For example:
  (such as c:/jdk1.8_13/bin
  bin). For details, refer to the following instructions:
  https://java.com/en/download/help/path.xml
  multiexcerpt-include
  MultiExcerptName ExitDisclaimer
  nopanel true
  PageWithExcerpt
  wikicontent:Exit Disclaimer to Include
  wikicontent:Exit Disclaimer to Include
4. Check whether you have performed the previous steps correctly : Open a cmd or shell window and type the following command. The command output should display the version of Java you intended to install:
  Code Block
  java –version
  by repeating the java -version command.
If the latest version of JDK is not installed, perform the following steps:
1. Download JDK from the following page:
Prepare the DME CLU:
1. If you are using Windows, decide how you will use the Command Line Utilities. Choose one of the following options: (Question: What are the pros and cons of each?)Install Cygwin as described in the following site:
  https://www.cygwinoracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
  multiexcerpt-include
  MultiExcerptName ExitDisclaimer
  nopanel true
  PageWithExcerpt wikicontent:Exit Disclaimer to Include
2. Install JDK. For information, refer to the following page:
  https://www.oracle.com/technetwork/java/javase/documentation/index.html
  Multiexcerpt include
  MultiExcerptName ExitDisclaimer
  nopanel true
  PageWithExcerpt wikicontent:Exit Disclaimer to Include
  Open
In a
terminal
cmd
window.Use
or shell window, type the following
command
command in your working directory to clone DME utilities to your local machine:
code
Panel
borderColor silver
borderStyle solid
Clipboard
AllowLineWrap true
Git

clone

https://github.com/CBIIT/HPC_DME_APIs
The system creates a subdirectory (HPC_DME_APIs) containing all files from that GitHub repository.
In the HPC_DME_APIs/utils/ folder, create a copy of the sample configuration file (hpcdme.properties-sample) and rename it (as hpcdme.properties). Edit your hpcdme.properties file to make the following configuration changes:

Property Instructions
Server settings
For further instructions, refer to the following file:
https://github.com/CBIIT/HPC_DME_APIs/blob/master/utils/README.md Include Pagewikicontent:Exit Disclaimer to Includewikicontent:Exit Disclaimer to Include

Code Block
java –version

Property	Instructions
Server settings

...

If you want to connect to the UAT tier, uncomment the UAT server settings and comment out the Production server settings. By default, the Production server settings are enabled.

hpc.ssl.keystore.password Contact NCIDataVault@mail.nih.gov to request this password.

hpc.user

Specify your NIH user ID.

hpc.default.globus.endpoint If you intend to transfer files using Globus, specify the UUID of the Globus endpoint that you want to use. (The following instructions may be helpful: Copying the Globus Shared Endpoint UUID.)

hpc.server.proxy.port

If you intend to use Biowulf, specify the proxy port. To determine the correct port, run the following echo command:

Panel

borderColor	silver
borderStyle	solid

Clipboard

AllowLineWrap	true

echo $http_proxy

The output includes the port. For example, if the output of the echo command is http://dtn08-e0:3128, then your port is 3128.

hpc.server.proxy.url

If you intend to use Biowulf, specify the proxy URL. To determine the correct URL, ping the output of the echo command. For example:

Panel

borderColor	silver
borderStyle	solid

Clipboard

AllowLineWrap	true

ping dtn08-e0

The output includes an IP address. Specify that IP address as the URL.

hpc.error-log.dir
hpc.log.file

If you are using Windows, consider specifying a non-default location for the error log file. For example:

Code Block
hpc.error-log.dir=log hpc.log.file=log/hpc-cli.log

Configure environment variables:
1. Navigate to your ~/.bashrc or ~/.profile file, where ~ represents your home directory. For example:
  Panel
  borderColor silver
  borderStyle solid
  Clipboard
  AllowLineWrap true
  C:\cygwin64\home\<username>
2. If you are running CLU from Biowulf or Helix, add the following command to your ~/.bashrc or ~/.profile file:
  Panel
  borderColor silver
  borderStyle solid
  Clipboard
  AllowLineWrap true
  module load java/11.0.21
3. Edit your ~/.bashrc or ~/.profile file to append the following commands, where <client_utils_home> is the absolute path to the HPC_DME_APIs\utils\ folder. The following commands define the HPC_DM_UTILS environment variable and update the PATH environment variable to include the path to the scripts folder:
  Code Block
  export HPC_DM_UTILS="<client_utils_home>" export PATH=$HPC_DM_UTILS/scripts/:$PATH
If you are using Cygwin on Windows, run the following commands in Cygwin to change end-of-line delimiters from Windows standard to Unix standard in any files involved in CLU setup:
1. Adjust the line delimiters in the file with your export commands (~/.bashrc or ~/.profile). For example:
  Panel
  borderColor silver
  borderStyle solid
  Clipboard
  AllowLineWrap true
  sed -i "s/\r$//" ~/.bashrc
2. Adjust the line delimiters in all files in the scripts folder. Use the following command:
  Panel
  borderColor silver
  borderStyle solid
  Clipboard
  AllowLineWrap true
  sed -i "s/\r$//" $HPC_DM_UTILS/scripts/*
To make the above changes take effect in current Bash session, source the modified file (~/.bashrc or ~/.profile). For example:
Panel
borderColor silver
borderStyle solid
Clipboard
AllowLineWrap true
source ~/.bashrc
Generate a DME API authentication token, as described in Generating a DME API Authentication Token via the CLU.
If you want to use Globus or AWS S3, follow the instructions in the following pages:
- Preparing to Use Globus
- Preparing to Use AWS S3 Bucket for the CLU

To view a list of DME CLU commands, type dm_list_commands . To view usage information for any DME CLU command, type the command with the help (-h) parameter. For detailed information on each of the available DME CLU commands, refer to the following pages:

Getting Started
- Generating a DME API Authentication Token via the CLU — dm_generate_token
- DME CLU Quick Reference - (most frequently-used commands)
Finding Data
- Searching for Data via the CLU — dm_query_dataobject
Using Bookmarks
- Adding a Bookmark via the CLU — dm_add_bookmark
- Updating a Bookmark via the CLU — dm_update_bookmark
Retrieving Metadata
- Retrieving the Metadata of a Collection via the CLU — dm_get_collection
- Retrieving the Metadata of a Data File via the CLU — dm_get_dataobject
Registering Data
- Creating an Empty Collection via the CLU — dm_register_collection
- Registering a Data File via the CLU — dm_register_dataobject, dm_register_dataobject_presigned, dm_register_dataobject_multipart
- Registering Directory Contents from Your Local System via the CLU — dm_register_directory
- Updating Collection Metadata via the CLU — dm_register_collection
- Updating Data File Metadata via the CLU — dm_register_dataobject
Downloading Data
- Downloading One or More Files via the CLU — dm_download_dataobject, dm_download_dataobject_presigned, dm_download_dataobject_globus, dm_download_dataobject_s3, dm_download_dataobject_aspera
- Using dm_download_collection — dm_download_collection
- Using dm_download_collection_globus — dm_download_collection_globus
- Using dm_download_collection_s3 — dm_download_collection_s3
- Using dm_download_collection_aspera — dm_download_collection_aspera
Using DME as an Administrator
- Registering a User via the CLU — dm_register_user
- Adding a Bookmark and Permissions via the CLU — dm_add_bookmark
- Updating a Bookmark and Permissions via the CLU — dm_update_bookmark
- Moving or Renaming a Collection or File via the CLU — dm_rename
- Deleting Data via the CLU — dm_delete_datafile, dm_delete_collection

Show If

group	GP-CFW-DMEDOC-DEV, GP-CFW_ADMINS

Panel

borderColor	silver
borderStyle	solid
title	Visible to Internal Users Only

Using DME as a System Administrator
- Updating a User via the CLU — dm_update_user

Important Note for Mac Version 10.12.4/10.12.6

If you want to use DME CLU on a Mac (version 10.12.4/10.12.6) and the version of curl on your machine is OpenSSL 0.9.8zh 14 Jan 2016, perform the following steps to rebuild curl.

From a terminal prompt, execute the following command:
Panel
borderColor silver
borderStyle solid
Clipboard
AllowLineWrap true
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)
This command downloads and installs Homebrew, a package manager to download, build, and install Mac software.
Follow the prompts. The system might request your username/password multiple times.
Execute the following command:
Panel
borderColor silver
borderStyle solid
Clipboard
AllowLineWrap true
brew install curl --with-openssl
This command invokes Homebrew to build and install curl in /usr/local/opt with the latest version of openssl. At the end of the installation, a message appears indicating that this is keg-only. It provides the command to update the PATH so that the system uses your new curl installation, rather than the Mac-supplied one.
Execute the command as provided at the end of curl installation, which should be the following command:
Panel
borderColor silver
borderStyle solid
Clipboard
AllowLineWrap true
export PATH="/usr/local/opt/curl/bin:$PATH"' >> ~/.bash_profile

Confirm the installation: Open a new terminal and check the version of curl with the following command:

Panel

borderColor	silver
borderStyle	solid

Clipboard

AllowLineWrap	true

curl –-version

The following information should appear:

Code Block

curl 7.58.0 (x86_64-apple-darwin16.7.0) libcurl/7.58.0 OpenSSL/1.0.2n zlib/1.2.8
Release-Date: <today’s date or the date you install your new curl>
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy

Content

Space Tools

Versions Compared

Old Version 9

New Version Current

Key

Important Note for Mac Version 10.12.4/10.12.6