Overview
LexEVS provides classes to extend and interfaces to implement that help provide the framework for in-memory transformations of source files into the LexGrid model. If the target source is large, batch loading may well be a better solution, in which case the Loader Framework built on Spring Batch may be a better match for source loading. On the other hand, users may find the standard LexEVS loader interface a little easier to implement.
Writing loaders requires both programming skill and content expertise. Mapping source into the LexGrid data model requires knowledge of how the source defines entities, relationships between entities and any qualifiers or properties for these elements. However, some study of other presentations of that content, and documentation of the source files, can help most programmers make informed choices as to that mapping. Examples of source mapped into LexGrid are documented here.
Where to Start
The LexEVS API programming environment is currently available in a github repository here: LexEVS Source Code
Cloning this repository produces a large set of eclipse friendly projects that can be imported into Eclipse or Intellij.
Interfaces and Abstract classes to Extend
Loader interfaces must extend the interface org.LexGrid.LexBIG.Extensions.Load.Loader from the lbInterfaces project and implementations of your new interface must also extend the abstract class org.LexGrid.LexBIG.Impl.loaders.BaseLoader in the lbImpl project. Let's take as an example the MedDRA loader interfaces and classes as it is a relatively simple implementation.
The extension of Loader is org.LexGrid.LexBIG.Extensions.Load.MedDRA_Loader.java and is found in the lbInterfaces project.
public interface MedDRA_Loader extends Loader {
public final static String name = "MedDRALoader";
public final static String description = "This loader loads MedDRA files into the LexGrid format.";
The implementation of MedDRA_Loader (and extension of BaseLoader) is org.LexGrid.LexBIG.Impl.loaders.MedDRALoaderImpl.
public class MedDRALoaderImpl extends BaseLoader implements MedDRA_Loader
This defines a load and validation methods and creates initialized class attributes name and description. The name of this interface will eventually be used by the LexEVS extension function to call the loader into existence. (Loaders are always extensions to LexEVS).
Implementation Example and Discussion
Call the Super Constructor
Looking at the implementation of this interface in lbImpl, org.LexGrid.LexBIG.Impl.loaders.MedDRALoaderImpl, notice that implementation is kept relatively clean thanks the fact that much of the mechanism of loading into LexEVS is taken care of under the covers by the BaseLoader class. MedDRALoaderImpl creates a constructor that always calls the BaseLoader constructor, then prevents the use of manifests. (If you wish you may choose to allow manifest use, since it allows load time manipulation of coding scheme metadata. Often source files do not provide all that needs to be known about the source. A manifest file allows values that may not be present in the source such as copyrights, authoring information, version definition and formal coding scheme name to be added to the load.)
Define the Extension
The buildExtensionDescription method provides a background method for the registration of this loader to take place within LexEVS. It should be created the same way for each loader.
Validate the Source (Optional)
The validate method is not always implemented, but can and should be when a mechanism exists to insure that this is a correctly structured version of the source. When source is XML formatted and has a schema or dtd document commonly available for validation, this is a relatively easy process. However any free text files that do not have any associated java-based validation API, would rely on the developer to create validation functions. The validation method is not required to create a loader for LexEVS.
Implement the Abstract Method doLoad() (and possibly override the load() method)
The developer could at this point implement only doLoad by mapping content to a coding scheme object before pass control over to BaseLoader which will call the doLoad method and set default load options in its load method and kick off the processes to persist a coding scheme object to the database. The other option is to implement doLoad and override the load method which sets up end user option choices for the loader. Most loaders implement the load method, customizing load options to provide to the end user. In the case of the MedDRA loader, a CUI load option is provided to the end user.
Less Structure Beyond the Loader, BaseLoader Implementation and Extension
Beyond these methods, the structuring of code is largely and necessarily left up to the developer. However a few common patterns are fairly consistent in this implementation. Generally speaking, there is a central mapping class where the coding scheme creation is kicked off. Other classes, when necessary, are supportive to this central class.
Tasks to be Accomplished
These classes are generally classified by those that are responsible for either reading a source file or accessing an API that reads the file for you and those that map objects created from that file into LexEVS coding scheme metadata, entity and relationship objects.
Mapping Entry Point
In the lbConverter project the edu.mayo.informatics.lexgrid.convert.directConversions.medDRA package contains the classes that do much of the work of the MedDRA load into LexGrid. edu.mayo.informatics.lexgrid.convert.directConversions.medDRA.MedDRA2LGMain provides a central kickoff point with some methods that can be wrapped for load and validation responsibilities down further up the execution chain.
Read the Data to Source Model Objects
This package also contains edu.mayo.informatics.lexgrid.convert.directConversions.medDRA.MedDRAMapToLexGrid with a readMedDRAFile method that reads from a CSV file and persists it to objects defined in package edu.mayo.informatics.lexgrid.convert.directConversions.medDRA.Data. While this data package defines beans that model the content of lines read from the CSV file, it also organizes them into structures that are easier for the mapping code to use. The individual manner of implementation is left up to the developer but this is a good example of how a third party library was used to process the file and how the resulting objects were stored in a data structure that's easy for the mapping code to consume.
Map the Model Objects to LexGrid Model Objects
Once the source is read an persisted to the appropriate model objects the MedDRAMapToLexGrid class can map these data objects derived from the MedDRA source into a complete coding scheme object.
Pass Control Back to BaseLoader
The tasks are to read the file into some kind of logical model or bean class object, organize these objects or make them available to be mapped into LexEVS objects, tie all objects together as a coding scheme, and pass this potentially large coding scheme object to the LexEVS BaseLoader to be persisted to the database. Back up the execution chain in MeDRALoaderImpl the doLoad method first calls the mapping method of the conversion class MedDRA2LGMain to get the reading and mapping done, then passes control of the resulting coding scheme to BaseLoader by calling this.persistCodingSchemeToDatabase method on the BaseLoader super class. While it's not necessary that things are done this way, many loaders have been written in this fashion.
