Preparing to use Amazon Web Services S3 with DME

If you need to move a large volume of data, you can do so using an Amazon Web Services (AWS) S3 bucket. DME uses AWS for high performance transfer of large files.

To prepare an S3 bucket:

1. In AWS, create a bucket, create an access policy, create an IAM user, get access keys for that user, and set up permissions for the bucket, as described in AWS user documentation:
    https://docs.aws.amazon.com/s3/ 

2. Gather required information: 
   • The name of the destination S3 bucket
   • The path to and the name of the file or folder to be created in the destination bucket 
   • The access key and secret key
   • The S3 region

   For instructions on locating this information, refer to AWS user documentation. 

3. In your ~/.aws/ folder, create a file with the filename credentials that specifies S3 credential information, as follows:

   [default]
   aws_access_key_id = <access key>
   aws_secret_access_key = <secret access key>
   region = <s3 region>

4. Consider placing the credentials file in the default location for CLU commands, as follows:

   ~/.aws/credentials

5. If you are using Cygwin on Windows, run the following command in Cygwin to change end-of-line delimiters from Windows standard to Unix standard in the credentials file: 

   sed -i "s/\r$//" ~/.aws/credentials

Now you can upload or download data as described in the following pages:

• Downloading a File or Collection to an S3 Bucket via the CLU
• (TBD)