![]() |
Page History
If your user account has the Read , Write, or Own permission level on various on various collections, you can download can download one or more of data files in those collections, from DME to an Amazon Web Services (AWS) S3 S3 bucket.
To download one or more collections to data files to S3:
Prepare an S3 bucket, as described in Preparing to use Amazon Web Services S3 with DMEUse AWS S3 Bucket for the CLU.
Consider whether you want to download a single collection or multiple collectionsdata file or multiple data files:
- To download a single collectiondata file: Plan to specify the path for that collection in data file in the command.
- To download multiple collectionsdata files: In your file local system, create a use a command line editor (such as vi editor) to create a file that lists the paths for all of the DME collections data files you want to download, delimited by newline. Plan to use the -f option to specify that file in the command.
If you are using Cygwin on Windows and downloading multiple collections, run Run the following command in Cygwin to change end-of-line delimiters from Windows standard to Unix standard in the collection list file:
Code Block sed -i "s/\r$//" <collection-list>
:
Panel borderColor silver borderStyle solid Clipboard AllowLineWrap true Run the following command:
code
collectiondm_download_
dataobject_s3 [optional
collection-logical-parameters] [
DME data path] <destination
S3
bucket>
<destination-path> [AWS
credentials
file
path]
The following table describes each parameter:
fileParameter Description [-D <REST-response>]
An optional parameter, specifying a path and filename in your
collectionlocal system. The system always creates a response file:
- If you specify this parameter, the system saves the response from the server to the specified file in the specified location.
- If you omit this parameter, the system saves the file as download-
file- dataobject-response-header.tmp in your home directory.
[-o <output-json-file>]
An optional parameter, specifying a path and filename in your
collectionlocal system. The system always creates an output file:
- If you specify this parameter, the system saves the output to the specified file in the specified location.
- If you omit this parameter, the system saves the output as download-
collection-logical-- dataobject-response-message.json.tmp in your home directory.
If the command is successful, the output file is empty.
[-f <paths-file>]
or
[
collectionDME data path]
One or more paths within DME. Select one of the following methods to specify the
collectionsdata file or
collectionsdata files that you want to download:
- To specify multiple
file- data files, use the -f parameter to specify a path and filename in your
collections- local system, of a file that lists the paths for all of the DME
collection- data files you want to download, delimited by newline.
- To specify a single
collection- data file, specify the path for the DME
use Amazon Web Services S3 with DME- data file you want to download.
<destination S3 bucket>
The name of the destination S3 bucket. <destination path>
The path to and the name of the folder in the destination bucket. Do not begin the path with a slash. If the destination folder structure you specify does not already exist, DME creates it. [AWS credentials file path]
The location of the credentials file. If your credentials file is in the default location, as noted in Preparing to
Use AWS S3 Bucket for the CLU, you can omit this parameter.
The following example downloads a Project_1 collection from the PI_Lab1 collection in DME to a bucket1 bucket in a collectionA collection.
For some examples, consider the following code specified in a credentials file:
Code Block |
---|
[default]
aws_access_key_id = SAMPLEACCESSKEY
aws_secret_access_key = SampleSecretAccessKey
region = us-east-1 |
Single File Example
The following example uses the credentials file in a non-default location to download a data file from DME.
Panel | |||||||
---|---|---|---|---|---|---|---|
| |||||||
| |||||||
Code Block | |||||||
title | Single Collection Example
|
In this example, the command performs the following:
- Locates or creates a folder1 folder in the bucket1 bucket.
- Locates or creates a subfolder1 folder within the folder1 folder.
- Downloads the data.txt file from the Project_1 collection in DME.
- Saves that file in the subfolder1 folder as file1.txt.
Multiple Files Example
For another example, consider the following command.
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
| ||||||||
Code Block | ||||||||
title | Multiple Collections Example
|
With the following code in the specified collectionfile-list.txt file, the above command downloads multiple collections command uses the credentials file in a default location to download multiple data files from various locations in DME.
Code Block | title | Contents of collection-list.txt File
---|
/Example_Archive/PI_Lab2/Project_1/data.txt /Example_Archive/PI_Lab2/Project_2/image.jpg /Example_Archive/PI_Lab3/Project_1 |
...
/image.png
|
In this example, the command performs the following:
- Locates or creates a folder1 folder in the bucket1 bucket.
- Locates or creates a subfolder1 folder within the folder1 folder.
- Downloads from DME all files listed in the specified file-list.txt file.
- Saves those files in the subfolder1 folder with the same file names they have in DME.