NIH | National Cancer Institute | NCI Wiki  

Error rendering macro 'rw-search'

null

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 19 Next »

Some of the features in caDSR II allow you to process content from a CSV file for bulk/batch importing to create new content. To view or use these features, log in as described in Logging In. These features need specially formatted templates. 

Contents of This Page

Conventions

Common usage details for the templates are that due to inclusion of the following Data Validations please follow these conventions: 

  • Basics
    • When the provided Template has .xls or .xlsx file extension, always start with a fresh Template for each import
      • This insures that the Data Validation and Formulas that are embedded in the template for drop-downs and auto-populate are correct for each Import and rows/valid choices have not been inadvertently deleted.
    • Save-As the completed Template as .cvs before importing
      • NOTE: This is a common step to forget and if you try to import a .xsl or .xslx file it will cause an immediate exception "error" when clicking "Import", so check the file extension if you have a problem
    • While editing the Template do not delete entire Rows or Columns
      • This is due to the Data Validation, if you delete rows some of the valid choices will be deleted
      • If you delete an entry in a cell that was auto-populated, you may lose the formula. These formulas are not essential for importing, but are helpful to avoid typos or other errors
      • You can use the Delete Key to clear the contents of individuals cell or cells but do not delete entire rows or columns
  • Templates contain Data Validation for several columns that are found on several of the templates end users avoid errors and streamline completing the spreadsheets. 
    • All templates - Context
        • Drop down of all valid Contexts
    • Import DEC and Import VD
        • Drop Down of the Standard Conceptual Domain Names, the CD ID will automatically populate
    • Import Value Domain
      • Drop Downs for Format, UoM, Datatype, and Type (Enumerated/NonEnumerated)
    • Import Designations - this is used to import the "USED_BY" designation as well as all AI types for Data Elements, Data Element Concepts, Value Meanings and Value Domains
      • Drop Downs for AI type, Alternate Name, Alternate Name Type, Alternate Name Context
      • For Alternate Name, when the Alternate Name Type is "USED_BY" the Alternate Name will automatically populate the selected Alternate Name Context
    • Import PV/VM 
      • Drop Down for VM String Type
      • Drop Down for optional VM Alternate Name Type


The Batch Owner and Batch User columns can be any text that you provide. The system does not validate these columns. 

Templates

The following table lists the latest templates.

  File Modified

Tips

Tips for All Imports

Use Data Validation1. Always start with a fresh Template with Data Validation to help avoid uncaught errors due to invalid manually entered data. We have added a lot more error handling in this release, but it's not perfect yet, in particular when invalid or retired concepts are used.
Delete/Purge2. When reimporting, it is safest to select all the rows and "Delete", then select "Purge", then select all rows and “Purge Records".
Unique Column Names3. You may add additional columns to the template for taking/keeping notes after the last template column, but the column names must be unique.
Invalid or Retired Concepts4. Invalid VM Concepts on import are not saved in the system and can create a GAP in the concept drop down. If the invalid concepts is the last qualifier concept, the system will not be able to detect it. Curators should fix all Concept Errors reported on Import BEFORE Validating. In Sprint 37, we will keep the Import Message visible after Validate to make it easier not to miss this important information.
Concept Formatting5. Concept String should not contain commas, tabs, or or other hidden characters. Always "Paste Values" in order to avoid issues.
CSV Files!!6. Template must be saved as CSV for importing. Other file types will cause an error.
Seq ID Rules7. Seq ID must be an integer.
Import Exceptions8. Missing Batch User, Batch Name, or Seq ID will cause Exception.
Except Row Number9. Multiple missing required fields may not report the correct row number in the Exception Message. Inspect the template for empty cells.
Save / Revalidate10. If making changes through front-end, Save and Revalidate. This will refresh the messages.

Tips for CDE Imports

CDE Date Created1. Newly created CDEs have Date Created that includes the timestamp (such as 09/15/2022 21:04:03.000000000). This will be fixed in Sprint 37 - 2147.
Duplicates in File2. Duplicate rows within the imported file are not caught on Import. Improve in Sprint 37 - 2151.

Tips for DEC Imports

Constrain CD Popup1. CD Popup needs to be constrained by Context in addition to workflow status. Fixed in Sprint 37 - 2114.
Validating Lots of Rows2. Sometimes, the system throws an error when attempting to validate more than 4 rows at a time. If that happens, try validating a few rows at a time.

Tips for VD Imports

Constrain CD Popup1. CD Popup needs to be constrained by Context in addition to workflow status. Fixed in Sprint 37 - 2114.

Tips for PV VM Imports

Specified VM Definition1. Optional Specified VM Definition is only used if the VM Type is "TEXT". The system will not throw an error but it won't be added to the VM. The Caption and column heading in the template will be changed in Sprint 37 - 2131.
Message for Non-Enumerated2. VD error improvement: If VD is non-enumerated, it reports "VD is either invalid, retired or non-enumerated", but if VD is retired, it reports "VD is Retired". Fixed in Sprint 37 - 2132.
Extraneous Missing Alt Name3. Extraneous warning "WARNING: Missing Alt Name, Alt Name will not be created." VM Alt Name is optional, but this message is issued for Seq IDs when there is no Alt Name details provided. Fixed in Sprint 37 - 2168.
Possible Timeout4. Validate/Create will timeout if more than 5 minutes. Try batches of 100 until we can redesign the code. Watch.
VM Reuse5. In Sprint 36, if a Text VM is specified and an existing VM with the exact name is found, it will be reused, even if it has concepts. The Validation message will indicate that existing VM is being used (need to fix spacing). Is this ok?

Tips for Designation Imports

Import Exceptions1. If AI ID, AI Version, AI Type, or Alt Name are missing, it will throw an Exception. The redesign to import without any validation will solve this problem. Fix in Sprint 38 - 2193.
Consistency Checking2. If AI Long Name does not match the specified AI Public ID Long Name, the system will show an error. We are requesting AI Long Name and AI Type to ensure Alt names are not inadvertently added to the wrong AI. There currently is no validation step. Just import and create. Public IDs are very similar, typos are easy to make. If this becomes onerous, we could add a Validate Step to display the AI Long Name and AI Type for curator to see and visually verify before selecting Create.

  • No labels