 |
Page History
...
Jason Lucas, Scott Bauer, Larry Wright, Cory Endle, Kim Ong, Tracy Safran, Rob Wynn, Gilberto Fragoso, Margaret Haber, Kumar, Sherri De Coronado, John Campbell, Bron, Luba, ShamineSana Din, Craig Stancl
Discussion Points:
- House keeping Items
- Reviewed agenda and approved - unless there are changes along the way.
- WebEx will be live all day.
- Goal is to record key tasks and wikis
- Goal is to set context for the rest of the meetings this week and to start identifying the issues to be addressed.
- Larry would like to start to complete the complete EVS Project Architecture (including LexEVS)
- General workflow for architecture:
- Gather terminology content → protege and meme → loaded into LexEVS terminology service → accessed via java api, rest service api, browsers
- Architecture now needs to include the triple store database and usage (REST API and native REST API). This service is to support clinical trials (CTRP). Ability to make changes into the production service was a driving factor in going to TripleStore architecture. Loading into triplestore can be done nightly if needed. Currently the loads have been nightly.
- OBO is currently being looked as as a third delivery channel.
- Expected to have expanded use of services and downloads
- Adverse events were the most downloads - and then used and built in other systems (CDISC and FDA, etc).
- Report Writer extracts value sets from LexEVS
- Current work happening to create a SPARQL based report writer. Planned for early February.
- This would be only on QA team.
- External use would require a security layer (doesn't exist today)
- Gilberto noted that report writer cannot currently take a search and return result set (with preferred names). Noted this is "simple search" and could also be part of the term browser.
- Existing templates should be able to be run without authentication.
- Current work happening to create a SPARQL based report writer. Planned for early February.
- The TripleStore still cannot provide all the terminology data needed for EVS and is stored in LexEVS.
- Mappings - need to determine how to capture and allow access to mappings. The LexEVS model and triple store model do not provide the needed flexibility today.
- Synchronization of data sources and coordination of distribution of data is an open issue. Consideration is needed to provide an umbrella API (Federated) that serves both LexEVS and TripleStore content.
- Gilberto noted that CTS2 services should be revisited and he'd like to review missing functionality and complexity (noted by CTRP).
- LexEVS historically has been based on standards since the early inception of the tooling.
- As a terminology service, all the content has been loaded into Lexgrid data model.
- Focus from standards has shifted to providing usable services to end users.
- There are possibilities for enhancing the service today that still provides interoperability.
- Thesaurus based use cases should be considered when determining goals.
- CTRP usage of TripleStore was speed in loading content. LexEVS loading of transitive table is long process. Tracy noted that LexEVS could start to use TripleStore technology to remove that bottleneck when loading content.
- Noted that existing applications may not be ready to transition to new serivces.
- Primary goal of EVS is to proviede terminolgoy content to NCI customers and users to support the sciences.
- Noted that no interest in re-desigining
- LexEVS provides a consistent model.
- Mappings
- Currently default to the LexGrid XML format. The other is RRF loader mappings.
- No plans to load mappings into SPARQL.
- ICDO3 Map from Meta - Kim looked at performace - not the best results.
- Mappings in LexEVS in coding schemes.
- FTP will be main distribution for Maps. (Tab delimited to text)
- Review of CTS2 mapping support will help decide if additional functionality can be added.
- No support for contextual mapping currently exists.
- Systems Priorities
- LexEVS has spent time this last year and utilizing Docker. This will provide efficient deployments.
- Current deployments are completed by using a documented deployment document.
- Containers will reduce the middle man needed for deployments.
- Systems team has been trying to create an "approved" NCI container so the LexEVS team can use.
- Will still need to continue running tests.
- Need to talk with systems about environment.
- Docker is operational at NCI.
- Docker usage for data
- Gilberto suggested promotion of data using docker.
- Provide the "database" container to the systems team.
- This would help as it is deployed up the tiers.
- Gilberto suggested promotion of data using docker.
- Docker distributions to be used for end users.
- ASU was working on this type of container.
- Current scripts do 90% of what is needed. Need to change configuration so it doesn't remove the services.
- LexEVS has spent time this last year and utilizing Docker. This will provide efficient deployments.
- QA
- Currently there are test scripts created by Kim and Shamime(?)Sana
- Covers the majority of the usage.
- Test scripts still being developed.
- Tin is still available for reference.
- Tests should be avaiable in Jenkins. (LexEVS team is currently doing this)
- Docker shouldn't cause concern for QA.
- Tech stack support
- Confirm that NODEJS and other technologies are supported.
- Currently there are test scripts created by Kim and Shamime(?)Sana
- Security Scans
- 508 needs to be addressed.
- Heroku and SWAGGER pages may need to be reviewed.
- 508 needs to be addressed.
- Lucene, SOLR and elastic search
- Jason noted that there might be need for discussion during the API focused discussion on Thursday.
...
Scott Bauer, Craig Stancl, Cory Endle, Gilberto Fragoso, Larry Wright, Lyubov, Tracy Safran, Rob Wynne, Kim Ong, Jason Lucas, Liz, Margaret, Bron Kiesler, Sherri De Coronado, Sana Din
Discussion Points:
- EVS REST API
- CTRP and GDC are currently using the API.
- https://evsrestapi.nci.nih.gov/evsrestapi/swagger-ui.html#/evs-controller
- Working with other groups for API usage.
- CTRP in the URL identifies the data being searched. Kim noted that in future it would be coding system and version.
- Modeled specific for the CTRP usecases. Data in the response may have specific CTRP.
- Larry would perfer to have a common API that could then be customized for specific users.
- Suggested to perform a gap analysis to look at what caDSR current uses and what is avaible available in the EVS CTRP API.
- Will need to get parameter instances so we can know exactly what is used. (restrict to code/properties, etc)
- This should be reviewed by NG, Tracy, Scott, etc and the caDSR team (Natalia, Vikram, etc.)
- Agreement to have a joint working group to help build a combined REST service to get content from both LexEVS and TripleStore environments
- Minimal testing has been completed (10 concurrent users).
- REST Service Future Direction
- The REST service should not have application specific content.
- There will be a need to consider input from users - caDSR, GDC, CDISC, CTRP
- Sherri suggested that it may be worthwhile to generalize the existing CTRP REST service and then get user feedback.
- Considerations for REST services:
- Keep the CTRP API separate and build a more common API
- Create a common API and provide specific convience APIs on top of the common API for specific customers.
- Fill in the gaps (to be identified) with CTS2 based REST service. Create local extensions and look at opportunities to update the specification.
- Customer needs will come first.
- LexEVS REST (CTS2) and LexEVS Remote Service gap analysis
- Service Discovery - could be created
- Paging and Iteration - no gap
- Entity Search - gaps in search type (LexEVS 6.4 Search Algorithm Implementation Details)
- Result Sorting - gaps exist
- Custom Result Filtering - gap exists
- CTS2 URI Read of Entiy - could be created
- Code System Operations - gap exists
- Entity Count - gap exists
- Relational Operations - gap exists
- Query Parameters - gap exists
- Result Parameters - gap exists
- Operation Functions - gap exists
- Value Set Operations - gap exists
- Value Set Search - gap exists
...
- Action Items:
- Perform a gap analysis to look at what caDSR current uses and what is avaible available in the EVS CTRP API.
- Assemble a joint working group to investigate a combined/coordinated REST service to get content from both LexEVS and TripleStore environments.
- Identify additions to the existing EVS API to make it more useful to users.
...
Tracy Safran, Jason Lucas, Rob Wynne, Larry Wright, Gilberto Fragoso, Kumar, Cory Endle, Scott Bauer, Craig Stancl, Lyubov, Margaret Haber, Bron, Sherri de Coronado, Sana Din
Discussion Points:
- Overall Impessions and themes
- Mappings
- Extract mapping from Meta efficiently
- This would save effort of creating custom maps.
- Extend the model of mappings (is it supported in CTS2) to support different types of maps.
- ie ICD-O3, Meta and the logic (OR, AND..)
- Extract mapping from Meta efficiently
- Diversity of paths through the architecture
- Addition of triple store
- Multiple APIs to address user communities.
- flow described - input of data through the tooling and delivery
- 2 views for documentation were identified:
- Focus on what users needed
- Focus on overall architecture (technical)
- Remote API Roadmap
- Determine the replacement for what of the API is needed. (based on gap analysis)
- Determine current users and identify what is required for those users.
- REST API
- Federation using SPARQL or other tooling
- Big Data will require that performance be addressed (caching, etc.)
- Will support the annotation pipelines
- LexEVS will need to provide REST services for content not available in TripleStore
- Provide documentation to better help users
- Report Writer
- Support for other terminologies. LexEVS REST services/EVS REST Services
- SWAGGER documentation
- Differentiate from the general API and CTRP specific API
- Microservices
- integration of triple store to support/enhance LexEVS functionality
- Hierarchy/Transitive Table support.
- User needs to create a unified service
- Discuss with stakeholders to gather requirements
- Determine how to move forward based on the requirements (best practice)
- Separate APIs
- Combined APIs
- Ensure the service simpilfies what the user needs to know about the technical implementation.
- This could be several months of effort (across teams)
- User Education - Enable users to use the services
- Provide better documentation for end users.
- Provide mapping of source into LexEVS or REST models so users can understand how to query the service in LexEVS
- Review and update Wiki Organization
- Provide documentation to aid in building applications that will utilize the services (REST, Java API, TripleStore/SPARQL/ftp)
- Architecture diagrams to describe the 1) flow of data and 2) technical specifics.
- Provide timeline for enhancement (REST Services), dates for deprecation, system deployments
- LexEVS REST Code Migration Guide
- Build and Deploy (Docker)
- Continued development of Docker containers with the systems team.
- Use of Node.js to be discussed with systems team.
- Investigate use of Docker for data deployments.
- Migrate CTS2 API from Heroku to NCI.
- Mappings
...
