NIH | National Cancer Institute | NCI Wiki  

Error rendering macro 'rw-search'

null

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

Contents of this Page

LexEVS Administration Links

 

 

LexEVS 6.1 Loader Scripts

Shell Script

Use and Function

LoadMedDRA

 Loads a file specified in the Medical Dictionary for Regulatory Activities (MedDRA) format.

Options:
  •    -in,--input <uri> URI or path specifying location of the source file
  •    -mf,--manifest <uri> URI or path specifying location of the manifest file
  •    -v, --validate <int> Perform validation of the candidate  resource without loading data.  If specified, the '-a' and '-t' options are ignored.  Supported levels of validation include:0 = Verify document is valid
  •    -a, --activate ActivateScheme on successful load; if unspecified the vocabulary is loaded but not activated.
  •    -t, --tag <id> An optional tag ID (e.g. 'PRODUCTION' or 'TEST') to assign.


Load Example:

LoadMedDRA -in "file:///path/to/file.asc" -a


Validation Example:

 LoadMedDRA -in "file:///path/to/file.asc" -v 0

LoadMIFVocabulary

 Loads from an the HL7 Vocabulary mif file.

 Options:
  •    -in,--input <uri> URI or path specifying location of the source file.
  •    -a, --activate ActivateScheme on successful load; if unspecified the vocabulary is loaded but not activated.
  •    -t, --tag <id> An optional tag ID (e.g. 'PRODUCTION' or 'TEST') to assign.


Load Example:

LoadMIFVocabulary -in "file:///path/to/file.xml" -a

Note

This replaces the LoadHL7RIM script and function

LoadOWL2

 Loads an OWL file. You can provide a manifest file to configure coding scheme# meta data.

 Options:

  •    -in,--input <uri> URI or path specifying location of the source file
  •    -mf,--manifest <uri> URI or path specifying location of the manifest file
  •    -lp,--loaderPrefs<uri> URI or path specifying location of the loader preference file
  •    -v, --validate <int> Perform validation of the candidate resource without loading data.  If specified, the '-a' and '-t' options are ignored.  Supported levels of validation include:
  •          0 = Verify document is well-formed
  •          1 = Verify document is valid
  •    -a, --activate ActivateScheme on successful load; if unspecified the vocabulary is loaded but not activated.
  •    -t, --tag <id> An optional tag ID (e.g. 'PRODUCTION' or 'TEST') to assign.


 Load Example: 

LoadOWL2 -in "file:///path/to/somefile.owl" -a
LoadOWL2 -in "file:///path/to/somefile.owl" -v 0

LoadResolvedValueSetDefinition

Loads Value Set Definition content, provided in LexGrid canonical xml format.


 Options:
   -u, The valueset definition URI to use
   -l, The list of coding schemes to revolve against. The format is codingschemeName::version
   -csVersionTag    The tag to use for resolving coding scheme
       
 Load Example: 

LoadResolvedValueSetDefinition -u \"Automobiles:valuesetDefinitionURI\" -l \"Automobiles::version1, GM::version2\" -csVersionTag \"production\" ""

Special Batch Loading Functions

Scheme and Metadata Removal

Pick List and Value Set Load Administration

LexEVS Validation Test Script

Command Line Scripts and Wrappers Overview

LexEVS provides a set of shell scripts and code wrappers that provide command line users standard options and automatic printouts of current coding schemes when performing administrative functions using the command line.

Displaying Current Coding Schemes

When you have coding Schemes Loaded to LexEVS many command line script calls will bring up a printout of a table of the loaded coding schemes to be used as a selective menu for other functions.

 

Using Optional Parameters

A standard parameter interface allows users to customize command line calls to LexEVS. These options control things such as loader synchronicity, supplemental file locations, and fault conditions. Shell scripts contain comments on usage for each function wrappers options.

# Loads an OWL file. You can provide a manifest file to configure coding scheme
# meta data.
#
# Options:
# -in,--input <uri> URI or path specifying location of the source file
# -mf,--manifest <uri> URI or path specifying location of the manifest file
# -lp,--loaderPrefs<uri> URI or path specifying location of the loader preference file
# -a, --activate ActivateScheme on successful load; if unspecified the
# vocabulary is loaded but not activated.
# -t, --tag <id> An optional tag ID (e.g. 'PRODUCTION' or 'TEST') to assign.
#
# Example: LoadOWL -in "file:///path/to/somefile.owl" -a
#
java -Xmx3000m -XX:MaxPermSize=256M -cp "../runtime/lbPatch.jar:../runtime/lbRuntime.jar" 

org.LexGrid.LexBIG.admin.LoadOWL $@

Setting Java Virtual Machine Options

Heap size:
Memory management for heap size is a particular issue when loading larger Terminologies such as SNOMED or the NCI Thesaurus. Users need to be prepared to increase heap size if loads crash with heap size errors.
Example:

-Xmx:4000m

Permanent Generation:
LexEVS now depends on some Permanent Generation memory management during runtime.

Typically set as follows:

-XX:MaxPermSize:256m

Installing Sample Vocabularies

This LexEVS installation provides a sample vocabulary, Automobiles.xml, that can be loaded into the database.

  1. In a Command Promptwindow, enter the following to go to the example programs.

    cd {LEXBIG_DIRECTORY}/examples
  2. To load an example vocabulary, run the LoadSampleData script, LoadSampleData.bat for Windows; LoadSampleData.sh for Linux.
  3. To load other example vocabularies, run the appropriate script for any sample vocabulary in {LEXBIG_DIRECTORY}/test/resources/testData.

Note

Vocabularies cannot be loaded until configuration of the LexEVS runtime and database server are complete.

Displays the successful load of the sample vocabulary file.

Running the sample query programs

A set of sample programs are provided in the {LEXBIG_DIRECTORY}/examples directory. To run the sample query programs successfully a vocabulary must be loaded.

  1. Enter

    cd {LEXBIG_DIRECTORY}/examples
  2. Execute one of sample programs.
    .bat for windows or .sh for Linux.
    FindPropsandAssocForCode.bat
    FindRelatedCodes

Shows the output of example programs using sample vocabulary for finding properties and associations for a given code

Installing NCI Vocabularies

NCI Thesaurus Vocabulary

This section describes the steps to download and install a full version of the NCI Thesaurus for the LexEVS Service.

  1. Using a web or ftp client go to URL: ftp://ftp1.nci.nih.gov/pub/cacore/EVS/
    screenshot showing navigating to FTP directory
  2. Select the version of NCI Thesaurus OWL you wish to download. Save the file to a directory on your machine.
  3. Extract the OWL file from the zip download and save in a directory on your machine. This directory will be referred to as NCI_THESAURUS_DIRECTORY

  4. Using the LexEVS utilities load the NCI Thesaurus

    cd {LexBIG_DIRECTORY}/admin

  • For Windows installation use the following command

    LoadOWL.bat -in "file:///{NCI_THESAURUS_DIRECTORY}/Thesaurus_10.10d.owl"

  • For Linux installation use the following command

    LoadOWL.sh -in "file:///{NCI_THESAURUS_DIRECTORY}/Thesaurus_10.10d.owl"

Note

The NCI Thesaurus has grown large enough that it can no longer be loaded on many typical desktop machines. We recommend a 64-bit operating system running on a multiprocessor computer with a minimum of 4g of memory. Server class Linux machines are the typical target for these loads. The time to load NCI Thesaurus will vary depending on machine, memory, and disk speed. Expect a couple of hours for a higher end machine.

The following code sample shows the example output from load of NCI Thesaurus 05.12f

[LexBIG] Processing TOP Node... Retired_Kind
[LexBIG] Clearing target of NCI_Thesaurus...
[LexBIG] Writing NCI_Thesaurus to target...
[LexBIG] Finished loading DB - loading transitive expansion table
[LexBIG] ComputeTransitive - Processing Anatomic_Structure_Has_Location
[LexBIG] ComputeTransitive - Processing Anatomic_Structure_is_Physical_Part_of
[LexBIG] ComputeTransitive - Processing Biological_Process_Has_Initiator_Process
[LexBIG] ComputeTransitive - Processing Biological_Process_Has_Result_Biological_Process
[LexBIG] ComputeTransitive - Processing Biological_Process_Is_Part_of_Process
[LexBIG] ComputeTransitive - Processing Conceptual_Part_Of
[LexBIG] ComputeTransitive - Processing Disease_Excludes_Finding
[LexBIG] ComputeTransitive - Processing Disease_Has_Associated_Disease
[LexBIG] ComputeTransitive - Processing Disease_Has_Finding
[LexBIG] ComputeTransitive - Processing Disease_May_Have_Associated_Disease
[LexBIG] ComputeTransitive - Processing Disease_May_Have_Finding
[LexBIG] ComputeTransitive - Processing Gene_Product_Has_Biochemical_Function
[LexBIG] ComputeTransitive - Processing Gene_Product_Has_Chemical_Classification
[LexBIG] ComputeTransitive - Processing Gene_Product_is_Physical_Part_of
[LexBIG] ComputeTransitive - Processing hasSubtype
[LexBIG] Finished building transitive expansion - building index
[LexBIG] Getting a results from sql (a page if using mysql)
[LexBIG] Indexed 0 concepts.
[LexBIG] Indexed 5000 concepts.
[LexBIG] Indexed 10000 concepts.
[LexBIG] Indexed 15000 concepts.
[LexBIG] Indexed 20000 concepts.
[LexBIG] Indexed 25000 concepts.
[LexBIG] Indexed 30000 concepts.
[LexBIG] Indexed 35000 concepts.
[LexBIG] Indexed 40000 concepts.
[LexBIG] Indexed 45000 concepts.
[LexBIG] Indexed 46000 concepts.
[LexBIG] Getting a results from sql (a page if using mysql)
[LexBIG] Closing Indexes Mon, 27 Feb 2006 01:44:22
[LexBIG] Finished indexing

NCI Metathesaurus Vocabulary

Loading the Metathesaurus

This section describes the steps to download and install a full version of the NCI Metathesaurus for the LexEVS Service.

  1. Using a web or ftp client go to URL: ftp://ftp1.nci.nih.gov/pub/cacore/EVS/
    screenshot showing navigating to ftp client in Explorer
  2. Select the version of NCI Metathesaurus RRF you wish to download. Save the file to a directory on your machine.
  3. Extract the RRF files from the zip download and save in a directory on your machine. This directory will be referred to as NCI_METATHESAURUS_DIRECTORY. RELEASE_INFO.RRF is required to be present for the load utility to work.

  4. Using the LexEVS utilities load the NCI Thesaurus

    cd {LexBIG_DIRECTORY}/admin

  • For Windows installation use the following command

    LoadMetaBatch.bat -in "file:///{NCI_METATHESAURUS_DIRECTORY}/"

    For Linux installation use the following command

    LoadMetaBatch.sh -in "file:///{NCI_THESAURUS_DIRECTORY}/"

Note

NCI Metathesaurus contains many individual vocabularies some of which are large vocabularies in and of themselves. It requires many hours to load and index. It can require 36 hours on a multiprocessor machine with 6g plus memory. The total time to load NCI MetaThesaurus will vary depending on machine, memory, and disk speed. Because this loader uses a batch loading strategy it is less dependent on memory, but some users will see 3 or 4 day load times with average multiprocessor processing power.

Resuming Loads

Since this loader is resource hungry we provide the option to restart should you find your resource settings to be inadequate. Resuming loads which have crashed or been interrupted by server problems is possible using the ResumeBatchLoad script set.

  1. Using the LexEVS utilities load the NCI Thesaurus

    cd {LexBIG_DIRECTORY}/admin

  • For Windows installation use the following command

    ResumeMetaBatch.bat -in "file:///{NCI_METATHESAURUS_DIRECTORY}/" -s "NCI Metathesaurus" -uri "urn:oid:2.16.840.1.113883.3.26.1.2" -version "200601"

  • For Linux installation use the following command

    ResumeMetaBatch.sh -in "file:///{NCI_THESAURUS_DIRECTORY}/" -s "NCI Metathesaurus" -uri "urn:oid:2.16.840.1.113883.3.26.1.2" -version "200601"

NCI History

This section describes the steps to download and install a history file for NCI Thesaurus.

  1. Using a web or ftp client go to URL:

    ftp://ftp1.nci.nih.gov/pub/cacore/EVS/
  2. Select the version of NCI History you wish to download. Save the file to a directory on your machine. Select the VersionFile download to the same directory as the history file.
  3. Extract the History files from the zip download and save in a directory on your machine. This directory will be referred to as NCI_HISTORY_DIRECTORY

  4. Using the LexEVS utilities load the NCI Thesaurus

    cd {LexBIG_DIRECTORY}/admin

  • For Windows installation use the following command:

    LoadNCIHistory.bat -nf -in "file:///{NCI_HISTORY_DIRECTORY}" -vf "[file:///NCI_HISTORY_DIRECTORY]}/VersionFile"

  • For Linux installation use the following command:

    LoadNCIHistory.sh -nf -in "file:///{NCI_HISTORY_DIRECTORY}" -vf "[file:///{NCI_HISTORY_DIRECTORY]}/VersionFile"

Note

If a 'releaseId' occurs twice in the file, the last occurrence will be stored. If LexEVS already knows about a releaseId (from a previous history load), the information is updated to match what is provided in the file. This file has to be provided to the load API on every load because you will need to maintain it in the future as each new release is made. We have created this file that should be valid as of today from the information that we found in the archive folder on your ftp server. You can find this file in the 'resources' directory of the LexEVS install.

Deactivating and Removing a Vocabulary

This section describes the steps to deactivate a coding scheme and remove coding scheme from LexEVS Service.

  1. Change directory to LexEVS administration directory. Enter:

    cd {LEXBIG_DIRECTORY}/admin

  2. Use the DeactiveScheme utility to prevent access to coding scheme. Once a coding scheme is deactivated, client programs will not be able to access the content for the specific coding scheme and version. Example:

    DeactivateScheme -u "urn:oid:2.16.840.1.113883.3.26.1.1" -v "10.10d"

  3. Use RemoveScheme utility to remove coding scheme from LexEVS service and database.
    Example:

    RemoveScheme -u "urn:oid:2.16.840.1.113883.3.26.1.1" -v "10.10d"

Tagging a Vocabulary

This section describes the steps to tag a coding scheme to be used via LexEVS API.

  1. Change directory to LexEVS administration directory. Enter:

    cd {LEXBIG_DIRECTORY}/admin

  2. Use the TagScheme to tag a coding system and version with a local tag name (e.g. PRODUCTION). This tag name can be used via LexEVS API for query restriction. Example:

    TagScheme -u "urn:oid:2.16.840.1.113883.3.26.1.1" -v "10.10d"  -t "PRODUCTION"

IndexManagement

LexEVS indexes can be manually updated, removed and optimized by users. This is a useful function particularly when metadata, authoring and post processing may have taken place.

Remove Index

  1. Change directory to LexEVS administration directory. Enter:

    cd {LEXBIG_DIRECTORY}/admin
  2. Use RemoveIndex script to remove an index for a particular coding scheme.
    Example:
    Windows:
    RemoveIndex.bat
    Linux:
    RemoveIndex.sh
  3. Interactive commmand: Enter the number of the Coding Scheme to use, then <Enter>. User responds with column number for appropriate coding scheme.
  4. Interactive command: CLEAR? ('Y' to confirm, any other key to cancel). User responds with 'Y'

  5. Command Line example:
    screenshot of the command line interface

    Note

    Concepts cannot be resolved without an index. Rebuilding the index is required.

Rebuild Index

  1. Change directory to LexEVS administration directory. Enter

    cd {LEXBIG_DIRECTORY}/admin
  2. Use RebuildIndex script to rebuild an index for a particular coding scheme.
    Example:
    Windows:
    RebuildIndex.bat
    Linux:
    RebuildIndex.sh
  3. Interactive command: Enter the number of the Coding Scheme to use, then <Enter>.

User responds with column number for appropriate coding scheme.

  1. Interactive command: REBUILD INDEX FOR URI? <uri displayed here> ('Y' to confirm, any other key to cancel). User responds with 'Y'

  2. Command Line example:
    screenshot of command line interface

    Note

    It may be useful and performance enhancing to run OptimizeLuceneIndex script after removing and rebuilding.

  • No labels