Date: Thu, 28 Mar 2024 20:04:18 -0400 (EDT) Message-ID: <1617824295.807.1711670658448@ip-10-208-26-37.ec2.internal> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_806_978065226.1711670658438" ------=_Part_806_978065226.1711670658438 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
Some of the features in caDSR II allow you to process content fr= om a CSV file for bulk/batch importing to create new content. To view or us= e these features, log in as described in Logging In. These features need specially formatted templates. = ;
Common usage details for the templates are that due to inclusion of the = following Data Validations please follow these conventions:
Form Import: "Shift" + "CMD" + "t" to transform con= tent into the correct format for importing. If you are using a REDCap data = dictionary csv file, cut and paste your content into the template starting = on Row 3, Column E "Variable / Field Name".
DO NOT USE | Batch User | Batch Name | Seq ID |
The Batch Owner and Batch User columns can be any text that you provide.= The system does not validate these columns.
The following table lists the latest templates.
Use Data Validation | Always start with a fresh Template with Data Val= idation to help avoid uncaught errors due to invalid manually entered data.= We have added a lot more error handling in this release, but it's not perf= ect yet, in particular when invalid or retired concepts are used. |
---|---|
Required Columns | All templates require a BATCH USER, BATCH NAM= E, and SEQ ID on each row. |
Delete/Purge | When reimporting, it is safest to select all = the rows and "Delete", then select "Purge", then select all rows and =E2=80= =9CPurge Records". For Forms, select "Delete Hierarchies", then selec= t "Purge", then select the rows and "Purge Hierarchies". |
Unique Column Names | You may add additional columns to the template f= or taking/keeping notes after the last template column, but the column name= s must be unique. |
Invalid or Retired Concepts | Invalid VM Concepts on import are not saved in t= he system and can create a GAP in the concept drop down. If the invalid or = missing concepts are found, the system will display and error. Curators sho= uld fix all Concept Errors reported on Import BEFORE Validating. |
Concept Formatting | Concept String should not contain commas, tabs, = or or other hidden characters. Always "Paste Values" in order to avoid issu= es. |
CSV Files!! | Template must be saved as CSV for importing. Oth= er file types will cause an error. |
Seq ID Rules | Seq ID must be a unique integer. |
Import Exceptions | Missing Batch User, Batch Name, or Seq ID will c= ause Exception. |
Selecting the "Import" Command | For imports and matching, select the type of = import/match item from the "Favorites" list, then select the "Import" comma= nd. For most imports there will be a dropdown with either one or two of the= following choices: "Conceptual Object Import" and "Data Object Import".&nb= sp; For most imports choose the first choice in the list. For VM Matc= h both choices are presented, choose the second one, "Data Object Import".<= /p> |
Exception on SEQ/Row Number | Multiple missing required fields may not report = the correct row number in the Exception Message. Inspect the template for e= mpty cells. |
Save / Revalidate | If making changes through front-end, Save and Re= validate. This will refresh the messages. |
Duplicates in File | Duplicate rows within the imported file are not = caught on Import, but duplicate CDEs will not be created. |
---|
Run Transform | CDE Match transforms the Source input into on= e row per column and permissible value. The command to transform the Source= input details is "opt"+"cmd"+"t". |
---|---|
See Instructions | Please see the CDE Match Instructions document a= ttached to this page. |
Special Features | CDE Match has a command "Run DEC Match". This wi= ll run the match algorithm on the imported files using just the CDE names, = ignoring the permissible values. Click the "Matched DEC" node to see result= s. |
Validating Lots of Rows | Sometimes, the system throws an error when attem= pting to validate more than 4 rows at a time. If that happens, try validati= ng a few rows at a time. |
---|---|
DEC Import Required Fields | DEC Context, OC Concepts, Property Concepts and = DEC CD Name/ID are required. |
DEC Update Required Fields | The DEC Context, DEC Public ID and Version are r= equired for an Update. Any other columns that are populated will be treated= as a change to the existing DEC, such as DEC CD, OC Concepts, Property Con= cepts. Retired DECs cannot be updated. DEC WFS will be set to "DRAFT = NEW". |
Known Issue | In release 1.53.4, if a duplicate DEC exists in = another Context, you cannot bulk update the DEC. This will be fixed in futu= re release. |
Description | DEC match will run matching algorithms on the na= mes in the template and display results in the "Matched DEC" node. You can = select one of the DECs as "Preferred" and then download the results using t= he Delivery Options feature. |
---|---|
Context Column | In the Context column, you can enter one or m= ore contexts to search for matching DECs. Separate each DEC with a comma. D= o not use spaces between DECs. For example: XXX,CTEP,CCR |
None at this time.
Import into Existing VD | The PV VM import is designed to import an enumer= ation into an existing Value Domain. PV/VMs can be created using existing V= Ms, or you can specify a new VM either with or without concept codes. |
---|---|
Specified VM Definition | Optional Specified VM Definition is only used if= the VM Type is "TEXT". The system will not throw an error but it won't be = added to the VM. |
Possible Timeout | Validate/Create will timeout if more than 5 minu= tes. Try batches of 100 until we can redesign the code. Watch. |
VM Reuse | In a Text VM is specified and an existing VM wit= h the exact name is found, it will be reused, even if it has concepts. The = Validation message will indicate that existing VM is being used (need to fi= x spacing). |
After Importing | The results after importing are displayed one= row per Value Meaning. All VM Match user results are kept in one database = table, and without filtering results, they are all displayed in the same gr= id. So, after selecting "Go to Data Manager" enter your batch name and clic= k "Apply Filter" to see just your results. |
---|---|
Use Cases | This feature can be used to match any terms t= o Concepts by selecting the "Concepts Only" command, including finding conc= epts for a new DEC. |
"User Tips" Column | The User Tips column is used by the system in= stead of the VM Name. Sometimes the VM Name from your form or data is not v= ery representative of the its meaning, or consists of too many terms. If a = "User Tip" is entered it is used instead of the VM Name. User Tips can be e= ntered through the front end, or starting in Sprint 47, can be entered into= the Template for each row and imported. |
Algorithm | Punctuation and spaces are removed. The "Rule= Description" explains why each match was selected. The algorithm does not = currently account for spelling errors. |
"Run Match" VM Header Command | Select the rows to run the match algorithm ag= ainst and select the "RUN MATCH" Command. Punctuation and spaces are remove= d. This command will try to find Exact or Like Matches using the VM preferr= ed name for existing VMs, Concept Preferred name and Synonyms. Click = "VM Matched Results" to review recommended matches. If you do not find a de= sired match, select "Run VM Match Unrestricted". |
"Run Match - VM Only" VM Header Comma= nd |
This match searches only for VM names that ma= tch. |
"Run VM - Match Unrestricted" = strong> VM Header Command |
This match searches VM Names on the longest t= erm in addition to more extensive matches. |
"Run Match - Concepts Only" VM Header= Command |
This match searches only for Concepts by name= and synonyms. |
"Run Match Concepts Only - Unrestrict= ed" VM Header Command |
This match searches only for Concepts by name= and synonyms the longest term in addition to more extensive matches. = |
"Run Match - Terminology" |
This matches allows the user to select a spec= ific terminology to search for Concepts by name, synonyms or Concept Code.<= /p> |
"VM Matched Results" | If there is only one exact match, the system = will automatically associate the matched result with the imported VM Name.<= /p> If there are multiple matches the user can select a row from "VM Matc= hed Results" and select the "Set Preferred" command. Multiple Concept= s can be selected one at a time to post coordinate concepts. The system wil= l appended the concept to the preferred = concept string and generate the resulting name in the order the concepts ar= e selected. This can be viewed by selecting the Node with the Concept= Name to view the selected items. The concepts and be reordered manually, b= ut be sure to reorder the concept names, or clear the results, "Save" and t= hen select concepts again from the matched results. |
Advancing through the imported VMs |
When on the "VM Match Header" you can advance to the next VM in your imp= orted file by using the "Rows x of xxx" arrows.
|
Column Order | The main column in VM Match is the VM Name. T= he processing is designed to find matching concepts based on the name and i= nsert the Concept Name and Concept Code into the table next to the VM NAME = unless "User Tips" are entered. This order of the columns makes it easier t= o populate the PV/VM Template using cut and paste. |
Exporting results | Use the "Delivery Options" command to export = the VM Match results into and Excel file. This simplifies creating the list= of PV/VMs for PV/VM Import to populate and existing Value Domain. |
Import from REDCap Data Dictionary Forma= t | The Form Import Template is an xslm filetype = and contains several Excel Macros, so please be sure to select "Enable Macr= os" when opening the template. This has been tested for use only on a Windo= ws PC platform, not Mac laptops.
|
||||||||||||
Running the Macros | There are 3 macros in the Excel Template. The= Macro run key sequence can be viewed by selecting "Tools" from the Excel m= enu bar, then "Macro".
|
||||||||||||
Filling the template with for= m data | The 1st row of headings are the REDCap DD headin= gs. The 2nd row of headings match caDSR headings. Enter a short name in the= Variable / Field Name, this can be used for matching to an existing CDE. P= aste content into the template starting in the 3rd row. Please see the temp= late for the mappings between the REDCap Data Dictionary column names and t= he caDSR form column names. | ||||||||||||
Form Context | REDCap does not have a column for Context. Conte= xt for all the forms in the templates is selected after importing the templ= ate into caDSR. |
Consistency Checking | If AI Long Name does not match the specified AI = Public ID Long Name, the system will show an error. We are requesting AI Lo= ng Name and AI Type to ensure Alt names are not inadvertently added to the = wrong AI. Public IDs are very similar, typos are easy to make. If this beco= mes onerous, we could add a Validate Step to display the AI Long Name and A= I Type for curator to see and visually verify before selecting Create. |
---|