Page History

...

1. If you are using Windows, install Cygwin as described in the following site: https://www.cygwin.com/
   

   Multiexcerpt include
   MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include

   
   For best results, run Cygwin.bat from C:\cygwin64\ rather than from the Windows Start menu.

2. Check which version of Java is currently installed:

   Panel
   borderColor silver borderStyle solid
   Clipboard AllowLineWrap true java –version

   The command output should display the version of Java. DME requires Java 1.8.x.

3. If Java 1.8.x is not installed, perform the following steps:

   1. Download Java 1.8.x from the following page:  https://www.java.com/en/download/

      Multiexcerpt include
      MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include

   2. Install Java, as described in the following installation instructions:  https://www.java.com/en/download/help/index_installing.xml

      Multiexcerpt include
      MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include

   3. Update your PATH system variable with the Java executable folder (such as c:/jdk1.8_13/bin). For details, refer to the following instructions: https://java.com/en/download/help/path.xml

      Multiexcerpt include
      MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include

   4. Check whether you have performed the previous steps correctly by repeating the java -version command. 

4. If the latest version of JDK is not installed, perform the following steps:

   1. Download JDK from the following page: 
      https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

      Multiexcerpt include
      MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include

   2. Install JDK. For information, refer to the following page:
      https://www.oracle.com/technetwork/java/javase/documentation/index.html

      Multiexcerpt include
      MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include

5. In a cmd or shell window, type the following command in your working directory to clone DME utilities to your local machine:

   Panel
   borderColor silver borderStyle solid
   Clipboard AllowLineWrap true Git clone https://github.com/CBIIT/HPC_DME_APIs

   The system creates a subdirectory (HPC_DME_APIs) containing all files from that GitHub repository. 

6. In the HPC_DME_APIs/utils/ folder, create a copy of the sample configuration file (hpcdme.properties-sample) and rename it (as hpcdme.properties). Edit your hpcdme.properties file to make the following configuration changes:

   Property	Instructions
   Server settings	If you want to connect to the UAT tier, uncomment the UAT server settings and comment out the Production server settings. By default, the Production server settings are enabled.
   hpc.ssl.keystore.password	Contact NCIDataVault@mail.nih.gov to request this password.
   hpc.user	Specify your NIH user ID.
   hpc.default.globus.endpoint	If you intend to transfer files using Globus, specify the UUID of the Globus endpoint that you want to use. (The following instructions may be helpful: Copying the Globus Shared Endpoint UUID.)
   hpc.server.proxy.port	If you intend to use Biowulf, specify the proxy port. To determine the correct port, run the following echo command: Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true echo $http_proxy The output includes the port. For example, if the output of the echo command is http://dtn08-e0:3128, then your port is 3128.
   hpc.server.proxy.url	If you intend to use Biowulf, specify the proxy URL. To determine the correct URL, ping the output of the echo command. For example: Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true ping dtn08-e0 The output includes an IP address. Specify that IP address as the URL.
   hpc.error-log.dir hpc.log.file	If you are using Windows, consider specifying a non-default location for the error log file. For example: Code Block hpc.error-log.dir=log hpc.log.file=log/hpc-cli.log

   .

7. Configure environment variables:

   1. Navigate to your ~/.bashrc or ~/.profile file, where ~ represents your home directory. For example: 

      Panel
      borderColor silver borderStyle solid
      Clipboard AllowLineWrap true C:\cygwin64\home\<username>

   2. Edit your ~/.bashrc or ~/.profile file to append the following commands, where <client_utils_home> is the absolute path to the HPC_DME_APIs\utils\ folder. The following commands define the HPC_DM_UTILS environment variable and update the PATH environment variable to include the path to the scripts folder:

      Code Block
      export HPC_DM_UTILS="<client_utils_home>" export PATH=$HPC_DM_UTILS/scripts/:$PATH

8. If you are using Cygwin on Windows, run the following commands in Cygwin to change end-of-line delimiters from Windows standard to Unix standard in any files involved in CLU setup: 

   1. Adjust the line delimiters in the file with your export commands (~/.bashrc or ~/.profile). For example: 

      Panel
      borderColor silver borderStyle solid
      Clipboard AllowLineWrap true sed -i "s/\r$//" ~/.bashrc

   2. Adjust the line delimiters in all files in the scripts folder. Use the following command:

      Panel
      borderColor silver borderStyle solid
      Clipboard AllowLineWrap true sed -i "s/\r$//" $HPC_DM_UTILS/scripts/*

9. To make the above changes take effect in current Bash session, source the modified file (~/.bashrc or ~/.profile). For example:

   Panel
   borderColor silver borderStyle solid
   Clipboard AllowLineWrap true source ~/.bashrc

10. Generate a DME API authentication token, as described in Generating a DME API Authentication Token via the CLU. 

11. If you want to use Globus or AWS S3, follow the instructions in the following pages:
    • Preparing to Use Globus
    • Preparing to Use AWS S3 Bucket for the CLU

To view a list of DME CLU commands, type dm_list_commands . To view usage information for any for any DME CLU command, type the command with the help (-h) parameter.  For For detailed information on each of the available DME CLU commandsCLU commands, refer to the following pages:

• Getting Started

  • Generating a DME API Authentication Token via the CLU — dm_generate_token

  • DME CLU Quick Reference - (most frequently-used commands)

• Finding Data

  • Searching for Data via the CLU — dm_query_dataobject

• Using Bookmarks

  • Adding a Bookmark via the CLU — dm_add_bookmark

• Retrieving Metadata

  • Retrieving the Metadata of a Collection via the CLU — dm_get_collection

  • Retrieving the Metadata of a Data File via the CLU — dm_get_dataobject

• Registering Data

  • Creating an Empty Collection via the CLU — dm_register_collection

  • Registering a Data File via the CLU — dm_register_dataobject, dm_register_dataobject_presigned, dm_register_dataobject_multipart

  • Registering Directory Contents from Your File System via the CLU — dm_register_directory

  • Updating Collection Metadata via the CLU — dm_register_collection

  • Updating Data File Metadata via the CLU — dm_register_dataobject

• Downloading Data

  • Downloading One or More Files via the CLU — dm_download_dataobject, dm_download_dataobject_presigned, dm_download_dataobject_globus, dm_download_dataobject_s3
  • Using dm_download_collection — dm_download_collection
  • Using dm_download_collection_globus — dm_download_collection_globus
  • Using dm_download_collection_s3 — dm_download_collection_s3

• Using DME as an Administrator

  • Registering a User via the CLU — dm_register_user

  • Adding a Bookmark and Permissions via the CLU — dm_add_bookmark

  • Moving or Renaming a Collection or File via the CLU — dm_rename
  • Deleting Data via the CLU — dm_delete_datafile, dm_delete_collection

...