XML Loader
Overview
Loads of LexGrid XML were formerly limited by the size of the Coding Scheme model element that could be constructed in memory.
Loaded models had a single entry point at the CodingScheme element. With the intention of providing improved loading performance,
access points to other levels of LexGrid Model elements, and a convenient format for authoring, the design of the 5.1 XML Loader has been
updated for LexEVS 6.0 and will be implemented with the following considerations:
- A streaming implementation of the loading mechanism allowing larger loads in a smaller memory footprint.
- A variety of entry points to facilitate loading of Revisions, System Releases, Value Sets, Pick Lists, and Coding Schemes.
- A loading mechanism for Authoring based manipulation of LexGrid based xml files allowing entities and associations to be added to a given coding scheme in XML, then loaded into a LexGrid data repository.
Streaming XML Implementation
The latest implementation of the LexGrid XML loader provides a partitioned load of a coding scheme or other potentially large elements.
The 6.0 loader will read coding scheme meta data into memory, load it to the LexGrid database and then begin stepping through entities and associations
persisting elements to the database and then removing them from the coding scheme object as the unmarshaller reads objects from XML and into the coding scheme
object in memory. Streaming XML reads while controlling the build of the coding scheme (and potentially other objects) in memory allowing the load of
far larger terminologies constructed in LexGrid xml. This also allows users to eventually export larger coding schemes, revise them with authoring tools,
and reload them as LexGrid Revision elements. This and other authoring scenarios will be exercised upon LexGrid XML with subsequent loads to the
LexGrid database possible. (See authoring design below.)
LexGrid coding scheme meta data contains an number of required and optional elements as defined in the LexGrid schema. The last of these elements, before entity and associations
are expressed, are the coding scheme mappings and the coding scheme properties. Mappings are required elements in the coding scheme and as
such are a default stopping point of coding scheme reads. Coding scheme properties are not required and as such may not exist in a given rendering of LexGrid XML.
Before persistence of meta data to the LexGrid database takes place, a high speed pass is made over the coding scheme meta data portion of the XML with the STAX
parser api to determine coding scheme properties presence.
Loader Post Processor
In order to facilitate extra Processing at the end of an Ontology load, LexEVS 6.0 will support Loader Post Processors.
A Loader Post Processor is logic that will be executed when the actual content load to the database is complete, but before the Lucene Indexing occurs. It will be implemented as an Extension - meaning that users may place jars in the LexEVS class path and introduce Loader Post Processors without the need to recompile.
A Loader Post Processor may apply any logic that is required -- there are not constraints as to the scope. A Loader Post Processor may update database content, delete content, reference other loaded ontologies, etc.
