Page History
If you need to move a large volume of data, you can do so using a Globus endpoint. DME uses Globus for high performance transfer of large files. Endpoints represent locations on a file transfer service server, such as Biowulf or gridftp. In DME CLU or GUI, you can choose to use Globus for file transfer. A prerequisite is a guest collection.
For a video of narrated slides demonstrating these instructions, refer to Video of Preparing to Use Globus. to prep-Globus-slides.pptx.
To prepare a Globus endpoint:
If you do not already have a shareable Globus endpoint, contact NCIDataVault@mail.nih.gov and request one.
HTML Comment hidden true When new users send such a request, Udit always asks them:
who recommended you to us,
which data are you trying to transfer and which location are you trying to transfer your data to?. Also can you please confirm if you have a Biowulf account?people send messages to the NCI Data Vault inbox, requesting an endpoint, Udit usually asks them various questions. I asked about including those questions here, but Sunita said most of them have never seen this page. People usually see this page after the DME team has talked with them.
- Log into Globus at the following site:
http://www.globus.orgMultiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include
Use your NIH user account credentials as described in the following page:
https://docs.globus.org/how-to/get-started/Multiexcerpt include MultiExcerptName ExitDisclaimer nopanel true PageWithExcerpt wikicontent:Exit Disclaimer to Include If you do not already have a guest collection in Globus, create one from the NIH HPC Data Transfer (Biowulf) public endpoint:
Panel borderColor black borderStyle solid Expand title Instructions to Create a Guest Collection - In the Globus File Manager, open the side bar and click Endpoints. On the Endpoints page, search for and select for NIH HPC Data Transfer (Biowulf). This is a public endpoint.
- In the search results, in the row for On the NIH HPC Data Transfer page(Biowulf), click Collections. On the collection details icon (). The details page appears. On the Collections tab, click Add a Guest Collection.
- On the Create New Guest Collection new guest collection page, browse to or specify the path to the directory where you want to save your data. Click Create ShareGuest Collection.
- In the Globus File Manager, open the side bar and click Endpoints. On the Endpoints page, search for and select for NIH HPC Data Transfer (Biowulf). This is a public endpoint.
- Share your Globus endpoint:
In the Globus File Manager, navigate to the list of endpoints collections by opening the side bar, clicking Endpoints Collections, and then clicking Shareable by You.
Select the endpoint you want to useyour guest collection. Access the sharing configuration for that endpoint collection by clicking Permissions.
Add permissions by clicking Add Permissions - Share With. In the Share With field, select the option to share with a group. Click Select a Group.
Search all groups for "HPCDME-" and select the appropriate Globus group for the DME tier you intend to use:
Production: HPCDMEHPCDME-PROD-App-Accts-Pool-FNLCR group.
UAT: HPCDMEPanel borderColor black borderStyle solid Expand title Instructions to Use UAT If you intend to use the UAT tier of DME, select the HPCDME-DEV-App-Accts-Pool-FNLCR group. - Select the appropriate permission for the selected group: If you intend to download to download data from DME to the Globus endpoint, select the write permission.
- Click Add Permission. The system adds the HPCDME Globus group to the list for the current share.
Now you can register or download data in the GUI as described in the following pages:
- Registering in Bulk via the GUIDownloading to Your File System from a Globus Endpoint via the GUI
- Downloading to a Globus Endpoint via the GUI
- Downloading to an AWS S3 Bucket via the GUI
- Downloading to Google Drive via the GUI
To register or download data via CLU, refer to the additional prerequisites described in Getting Started with DME CLU.