Overview
LexEVS provides classes to extend and interfaces to implement that help provide the framework for in memory transformations of source files into the LexGrid model. If the target source is large, batch loading may well be a better solution in which case the Loader Framework built on Spring Batch may be a better match. On the other hand users may find the standard LexEVS interface a little easier to implement.
Writing loaders requires both programming skill and content expertise. Mapping source into the LexGrid data model requires knowledge of how the source defines entities, relationships between entities and any qualifiers or properties for these elements. However some study of other presentations of that content, and documentation of the source files, can help most programmers make informed choices as to that mapping. Examples of source mapped into LexGrid are documented here (TODO) link.
Where to Start
The LexEVS API programming environment is currently available in a github repository here: LexEVS Source Code
Cloning this repository produces a large set of eclipse friendly projects that can be imported into eclipse.
Loader interfaces must extend the interface org.LexGrid.LexBIG.Extensions.Load.Loader from the lbInterfaces project and implementations of your new interface must also extend the abstract class org.LexGrid.LexBIG.Impl.loaders.BaseLoader in the lbImpl project. Let's take as an example the Medra loader interfaces and classes as it is a relatively simple implementation:
The extension of Loader is org.LexGrid.LexBIG.Extensions.Load.MedDRA_Loader.java and is found in the lbInterfaces project. This defines a loader and validation method and creates initialized class attributes name and description. The name of this interface will eventually be used by the LexEVS extension function to call the loader into existence. (Loaders are always extensions to LexEVS).
Looking at the implementation of this interface in lbImpl, org.LexGrid.LexBIG.Impl.loaders.MedDRALoaderImpl, notice that implementation is kept relatively clean thanks to much of the mechanism of loading into LexEVS is taken care of under the covers by BaseLoader. It creates a constructor that always calls the BaseLoader constructor, then prevents the use of manifests. If you wish you may choose to allow manifest use, since it allows load time manipulation of coding scheme metadata. Often source files do not provide all that needs to be known about the source. A manifest file allows values that may not be present in the source, such as copyrights, authoring information, version definition and formal coding scheme name to be added to the load.
The buildExtensionDescription method provides a background method for the registration of this loader to take place within LexEVS. It should be created the same way for each loader.
The validate method is not always implemented, but can and should be when a mechanism exists to insure that this is a correctly structured version of the source. When source is XML formatted and has a schema or dtd document commonly available for validation, this is a relatively easy process. However any free text files that do not have any associated java-based validation API, would rely on the developer to create validation functions. The validation method is not required to create a loader for LexEVS.
In order to get loading done theory the developer could at this point implement only doLoad by mapping content to a coding scheme object and persisting it before turning control over to BaseLoader which will call the doLoad method and set default load options in its load method and kick off the processes to persist a coding scheme object to the database. The other option is to implement doLoad and override the load method which sets up end user option choices for the loader. Most loaders implement the load method, customizing load options to provide to the end user. In the case of the MedDRA loader, a CUI load option is provided to the end user.
Beyond these methods, where a coding scheme object is passed to LexEVS for processing, the structuring of code is largely and necessarily left up to the developer. However a few common patterns are fairly consistent in this implementation. Generally speaking, there is a central mapping class where the coding scheme object is built. Other classes, when necessary, are supportive to this central class.
These classes are generally classified by those that are responsible for either reading a source file or accessing an API that reads the file for you and those that map objects created from that file into LexEVS coding scheme metadata, entity and relationship objects. A third category will include accessing LexEVS persistence mechanisms and passing these values to the database.
In the lbConverter project the edu.mayo.informatics.lexgrid.convert.directConversions.medDRA package contains the classes that do much of the work of the load. edu.mayo.informatics.lexgrid.convert.directConversions.medDRA.MedDRA2LGMain provides a central kickoff point with some methods that can be wrapped for load and validation responsibilities down further up the execution chain. This package also contains edu.mayo.informatics.lexgrid.convert.directConversions.medDRA.MedDRAMapToLexGrid with a readMedDRAFile that reads data from a CSV file and persists it to objects defined in package edu.mayo.informatics.lexgrid.convert.directConversions.medDRA.Data. While this data package defines some of the beans that model the content of lines read from the CSV file, it also organizes them into structures that are easier for the mapping code to user. Once done the MedDRAMapToLexGrid can map data objects derived from the MedDRA source into a complete coding scheme object.
So the tasks are to read the file into some kind of logical model or bean class object. Organize these objects or make them available to be mapped into LexEVS objects, tie all objects together as a coding scheme, and pass this potentially large coding scheme object to LexEVS to be persisted to the database. Back up the execution chaing MeDRALoaderImpl the doLoad method first calls the mapping method of the conversion class MedDRA2LGMain, then passes control of the resulting coding scheme to BaseLoader by calling this.persistCodingSchemeToDatabase method on the BaseLoader super class.
