Page History
...
- If you are using Windows, install Cygwin as described in the following site: https://www.cygwin.com/
Multiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include
For best results, run Cygwin.bat from C:\cygwin64\ rather than from the Windows Start menu. Check which version of Java is currently installed:
Code Block java –version
The command output should display the version of Java. DME requires Java 1.8.x.
- If Java 1.8.x is not installed, perform the following steps:
Download Java 1.8.x from the following page: https://www.java.com/en/download/
Multiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include - Install Java, as described in the following installation instructions: https://www.java.com/en/download/help/index_installing.xml
Multiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include - Update your PATH system variable with the Java executable folder (such as c:/jdk1.8_13/bin). For details, refer to the following instructions: https://java.com/en/download/help/path.xml
Multiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include Check whether you have performed the previous steps correctly by repeating the java -version command.
If the latest version of JDK is not installed, perform the following steps:
- Download JDK from the following page:
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.htmlMultiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include - Install JDK. For information, refer to the following page:
https://www.oracle.com/technetwork/java/javase/documentation/index.htmlMultiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include
- Download JDK from the following page:
In a cmd or shell window, type the following command in your working directory to clone DME utilities to your local machine:
Code Block Git clone https://github.com/CBIIT/HPC_DME_APIs
The system creates a subdirectory (HPC_DME_APIs) containing all files from that GitHub repository.
In the HPC_DME_APIs/utils/ folder, create a copy of the sample configuration file (hpcdme.properties-sample) and rename it (as hpcdme.properties). Edit your hpcdme.properties file to make the following configuration changes:
Property Instructions Server settings If you want to connect to the Production tier, uncomment the Production server settings and comment out the UAT server settings. By default, the UAT server settings are enabled.
hpc.ssl.keystore.password Contact NCIDataVault@mail.nih.gov to request this password. hpc.user Specify your NIH user ID.
hpc.default.globus.endpoint If you intend to transfer files using Globus, specify the UUID of the Globus endpoint that you want to use. (The following instructions may be helpful: Copying the Globus Shared Endpoint UUID.) hpc.server.proxy.port If you intend to use Biowulf, specify the proxy port. To determine the correct port, run the following echo command:
Code Block echo $http_proxy
The output includes the port. For example, if the output of the echo command is http://dtn08-e0:3128, then your port is 3128.
hpc.server.proxy.url If you intend to use Biowulf, specify the proxy URL. To determine the correct URL, ping the output of the echo command. For example:
Code Block ping dtn08-e0
The output includes an IP address. Specify that IP address as the URL.
Other properties not mentioned above These properties are for special cases and internal use only. If you have questions about these properties, contact NCIDataVault@mail.nih.gov. .
Configure environment variables:
Navigate to your ~/.bashrc or ~/.profile file, where ~ represents your home directory. For example:
Code Block C:\cygwin64\home\<username>
Edit your ~/.bashrc or ~/.profile file to append the following commands, where <client_utils_home> is the absolute path to the HPC_DME_APIs\utils\ folder. The following commands define the HPC_DM_UTILS environment variable and update the PATH environment variable to include the path to the scripts folder:
Code Block export HPC_DM_UTILS="<client_utils_home>" export PATH=$HPC_DM_UTILS/scripts/:$PATH
If you are using Cygwin on Windows, run the following commands in Cygwin to change end-of-line delimiters from Windows standard to Unix standard in any files involved in CLU setup:
Adjust the line delimiters in the file with your export commands (~/.bashrc or ~/.profile). For example:
Code Block sed -i "s/\r$//" ~/.bashrc
Adjust the line delimiters in all files in the scripts folder. Use the following command:
Code Block sed -i "s/\r$//" $HPC_DM_UTILS/scripts/*
To make the above changes take effect in current Bash session, source the modified file (~/.bashrc or ~/.profile). For example:
Code Block source ~/.bashrc
Generate a DME API authentication token, as described in Generating a DME API Authentication Token via the CLU.
...
Getting Started
Generating a DME API Authentication Token via the CLU — dm_generate_token
Using Bookmarks
Adding a Bookmark via the CLU — dm_add_bookmark
Retrieving Metadata
Retrieving the Metadata of a Collection via the CLU — dm_get_collection
Retrieving the Metadata of a Data File via the CLU — dm_get_dataobject
Uploading Data
Creating an Empty Collection via the CLU — dm_register_collection
Registering a Data File via the CLU — dm_register_dataobject
- Registering a Data File with a Pre-signed URL via the CLU — dm_register_dataobject_presigned
Registering Directory Contents from Your File System via the CLU — dm_register_directory
Updating Collection Metadata via the CLU — dm_register_collection
- Updating Data File Metadata via the CLU — dm_register_dataobject
Downloading Data
Downloading a Single File to Your File System or to S3 via the CLU — dm_download_dataobject_sync
Downloading a File or Collection to a Globus Endpoint via the CLU — dm_download_async
Downloading a File or Collection to an S3 Bucket via the CLU - 20190513 — dm_download_s3
Using DME as an Administrator
Registering a User via the CLU — dm_register_user
Adding a Bookmark and Permissions via the CLU — dm_add_bookmark
- Moving or Renaming a Collection or File via the CLU — dm_rename
- Deleting Data via the CLU — dm_delete_datafile, dm_delete_collection
...