Registering Directory Contents from Your File System via the CLU

This page is a work in progress.

If your user account has the Write or Own permission level on an existing collection in DME, and if that existing collection has been configured to contain another collection, you can recursively upload data: copying files within folders from your file system to that existing collection.

To register directory contents:

Make sure that your intended destination in DME is ready for your directory contents. For example, if you want to register your data in /Example_Archive/PI_Lab1/Project_New, then the Example_Archive collection, PI_Lab1 collection, and Project_New collection must already exist. For instructions, refer to the following pages:
In your file system, consider using a flat list or regular expression (regex) patterns to specify the data files you want to register. By default, the dm_register_directory command uploads all files and directories in the source directory. To register a subset of the files in the source directory, use one of the following options:
- You can create a flat list of data files. In each line of this files list file, specify a relative path to the source path. Plan to use the -l option in the command.
- You can use regex patterns to exclude files from registration, include files, or both.
  - To include files in registration, create an include file and plan to use the -i option in the command.
  - To exclude files from registration, create an exclude file and plan to use the -e option in the command.
  For information, refer to Specifying Criteria for Bulk Upload. If you proceed with a command that specifies both include and exclude files, the system first identifies files that match any of the include regex patterns. Then the system removes from the registration job the files that match any of the exclude patterns.
Consider whether the system requires metadata for the collections and data files you intend to create in DME. In your file system, for each directory and each data file that you intend to register, create a metadata file with the name <[directory,filename]>.metadata.json. Create each metadata file in the same path as its corresponding directory or data file. For example, for a file named sample.txt, name the corresponding metadata file sample.txt.metadata.json. (Keep in mind that the system uploads the metadata files along with the corresponding data, unless you specify otherwise with a flat list or regex patterns.) In the metadata file, specify the metadata that you want to upload, as follows:
```
{ 
    "metadataEntries": [
      {
        "attribute": "description",
        "value": "my-dataObject-description"
      },
      {
        "attribute": "my-second-attribute-name",
        "value": "my-second-attribute-value"
      }
    ]
}
```

Run the following command:

dm_register_directory [OPTIONS] <source-path> <destination-path>

The following table describes each option:

Option	Description
-c	If you want to turn off the checksum calculation, specify this option. By default, the system performs this calculation as a validation procedure. However, this validation can increase the time required to run the command, depending on the file size.
-d	Unable to render {include} The included page could not be found.
-e[EXCLUDE_FILE_PATH]	If you want to exclude files, specify this option. The system excludes the files that match the patterns specified in the file EXCLUDE_FILE_PATH.
-h	If you want to print a usage message for this command, specify this option.
-i[INCLUDE_FILE_PATH]	If you want to include files, specify this option. The system includes the files that match at least one of the patterns specified in INCLUDE_FILE_PATH.
-l[FILES_LIST]	If you want to register specific files and exclude all others, specify this option. The system registers the files mentioned in the FILE_LIST only. In each line of the FILES_LIST file, specify a relative path to the <source-path>.
-m	If you want to register metadata only, specify this option. The system does not register files.
-s	If you want to skip the default confirmation prompt and register directly, use this option.
-t[NUM_THREADS]	If you want to process the files more efficiently with a smoother upload, consider specifying the number of threads (from 1 to 32) that you want the system to use while uploading files to DME. Your network and the server directly impact the way the system uploads files. You can set the number of threads so the system has enough memory while processing massive files. For faster processing of your command, use more threads. For faster processing of the system as a whole, use fewer threads. The default is one thread.

The following table describes each parameter:

Parameter	Description
<source-path>	A path in your file system, specifying the data that you want to upload. The system uploads the contents of the folder you specify.
<destination-path>	A path within DME, specifying where you want the system to create the data.

If the system prompts you to confirm the registration, type Y and press Enter. The command recursively uploads files and directories from the directory specified in the <source-path> to the <destination-path>.

For example, the following command copies the contents (folders and files) of the JaneDoe folder in the file system to the Project_New collection in DME.

dm_register_directory /cygdrive/c/Users/JaneDoe /Example_Archive/PI_Lab1/Project_New

For instructions on performing similar tasks in the GUI, refer to the following pages:

Content

Space Tools

Registering Directory Contents from Your File System via the CLU