Stories by Ayeshmantha Perera on Medium

OpenMRS-FHIR Analytics Batch Streaming Mode in Google Cloud Platform

Ayeshmantha Perera — Thu, 17 Dec 2020 16:43:45 GMT

OpenMRS-FHIR Analytics Batch Streaming Mode

OpenMRS team is currently working with the Google Cloud Platform team on an Analytical platform that provides a set of tools for transforming OpenMRS data into FHIR based data warehouse.

The platform supports both Streaming and Batch mode, focusing more on the Batch (Bulk Upload) in this blog post. Batch mode reads the whole OpenMRS MYSQL database, transforms it into FHIR resources, and uploads it to the Google Cloud Platform's target data warehouse.

Setting up the platform for Batch mode.

Currently, the repository for the FHIR analytics platform is in Google Cloud Platform Github itself. You can find it here.

Prerequisites

You can find the required docker-compose files to set up the required environment in the repository itself.

Setting up OpenMRS with dummy data.

You can find the compose file here. This compose file includes MySQL 5.7 distribution, and you can see a dump attached as a volume to the entry point, which will spin up the database with dummy data.

You can also see a MySQL.cnf file attached as a volume, which will enable the Binlog in MySQL. This is required when you are trying out streaming Binlog mode with Debezium.

Setting up the sink FHIR store.

With the current state, you can point the platform to either an FHIR store in Google Cloud Platform or to point to the local FHIR store you set up for testing purposes.

I will be focusing on setting up locally since an obvious guide on setting up the FHIR store in Google Cloud Platform is available here.

To set up for local usage again, we have provided a docker-compose file here.

Add FHIR2 module to OpenMRS.

FHIR2 module is a reimplementation of the FHIR module in OpenMRS using FHIR R4.

Different modes in the Analytics Platform get used of the FHIR2 module in different ways. Anyhow, in the end, all these modes will be publishing the data to an FHIR store. This highlights the main usage of the FHIR2 module to the Analytics platform.

Although Streaming AtomdFeed mode uses the Atom Feed module of OpenMRS to capture different types of OpenMRS resources, it uses the FHIR2 module to get the FHIR object of the relevant OpenMRS resource.

In Batch mode, we are directly using the FHIR2 module of OpenMRS to fetch the resources defined as search list parameters by the end-user.

You need to download the latest released version and install the module through the admin page in OpenMRS (or copy that file to the modules directory of your OpenMRS installation).

Suppose you made it to this point 🤞🏾🤞🏾. Now it’s time to look into the Batch Mode.

More about Batch Mode.

As of now, the core technology behind the Batch mode is Apache Beam. And based on your requirement, we have defined a few run times for you to execute the Beam ETL Pipeline.

Direct Runner

The Direct Runner executes pipelines on your machine and is designed to validate that pipelines adhere to the Apache Beam model as closely as possible. Instead of focusing on efficient pipeline execution, the Direct Runner performs additional checks to ensure that users do not rely on semantics that is not guaranteed by the model. Some of these checks include:

enforcing immutability of elements
enforcing encodability of elements
elements are processed in an arbitrary order at all points
serialization of user functions (DoFn, CombineFn, etc.)

Using the Direct Runner for testing and development helps ensure that pipelines are robust across different Beam runners. Besides, debugging failed runs can be a non-trivial task when a pipeline executes on a remote cluster. Instead, it is often faster and simpler to perform local unit testing on your pipeline code. Unit testing your pipeline locally also allows you to use your preferred local debugging tools.

2. Data Flow Runner

The Google Cloud Dataflow Runner uses the Cloud Dataflow managed service. When you run your pipeline with the Cloud Dataflow service, the runner uploads your executable code and dependencies to a Google Cloud Storage bucket and creates a Cloud Dataflow job, which executes your pipeline on managed resources in Google Cloud Platform.

The Cloud Dataflow Runner and service are suitable for large scale, continuous jobs and provide:

a fully managed service
autoscaling of the number of workers throughout the lifetime of the job
dynamic work rebalancing

3. Spark Runner

The Apache Spark Runner can be used to execute Beam pipelines using Apache Spark. The Spark Runner can execute Spark pipelines just like a native Spark application; deploying a self-contained application for local mode, running on Spark’s Standalone RM, or using YARN or Mesos.

The Spark Runner executes Beam pipelines on top of Apache Spark, providing:

Batch and streaming (and combined) pipelines.
The same fault-tolerance guarantees as provided by RDDs and DStreams.
The same security features Spark provides.
Built-in metrics reporting using Spark’s metrics system, which reports Beam Aggregators as well.
Native support for Beam side-inputs via spark’s Broadcast variables.

These are defined as separate maven profiles, and you can specify which profiles to run by their id. In this blog, I am focusing more on the direct runner defined in the default profile.

Currently, the batch mode will dump the output to Paraquest files as well as it will publish data to the FHIR store as well.

ETL Process

We are currently using Avro, a binary data structure that stands in the middle as a pass-through format. The resources queried from the OpenMRS FHIR2 module will be in JSON format, which harder to convert in the PARQUET format. But Avro files can be easily converted into PARQUET format.

The ETL process directly queries FHIR2 resource (JSON) from the OpenMRS FHIR2 module and then converted into Avro data objects using Bunsen library, a handy library that helps end-users to load, transform and analyze FHIR data with Apache Spark. We are using the FHIR resource schemas provided by Bunsen to convert openMRS FHIR resources into Avro files.

These Avro files will be converted into Parquet files using the Beam pipeline in the Batch mode of the Analytics Platform. Later on, we can spin up the FHIR data warehouse using the PARQUET files generated from the ETL Pipeline.

This is all about what we offer as a Platform for now in the Batch Mode. We will be working on adding more features in the upcoming months. Keep in touch for more updates. Cheers 🤞🏾🤞🏾🤞🏾

GSoC 2020@OpenMRS

Ayeshmantha Perera — Sat, 29 Aug 2020 12:13:00 GMT

Got selected to participate in GSoC back and with the same community OpenMRS. It’s a great community with a lot of new work going on.

About the project

Primary Mentor: Ian Bachler.

Mentors: Jude Niroshan.

Student: Ayeshmantha Perera.

Project Link:

https://summerofcode.withgoogle.com/projects/#6540918858973184.

The main idea of the project is to develop a framework that can capture System metrics as well as data metrics in a real-time way with the ability to expose to monitoring tools such as Graphite, Grapahana, Prometheus.

Currently, the workflow is to get the use of the ActiveMQ based event module which captures Hibernate interceptor based events. (Which fires after completion of transactions) and to feed them into a flat table that records details about the event time type OpenMRS class as well as the resource link to access from the OpenMRS Rest API. This will be the API workflow up to saving these records in a table. So from the client-side, we will be exposing these metrics through Dropwizard which is a great powerful toolkit of ways to measure the behavior of critical components in the production environment. The best thing is we can expose the JMX metric registry with Dropwizard with the built-in reporters. So there is a servelet in Dropwizard which was extended with HTTP servlet which we can use to expose the metrics with an endpoint.

Main Influence for the Project

What is OpenMRS & Where is it?

OpenMRS is a software platform and a reference application that enables the design of a customized medical records system with no programming knowledge (although medical and systems analysis knowledge is required). It is a common platform upon which medical informatics efforts in developing countries can be built. The system is based on a conceptual database structure that is not dependent on the actual types of medical information required to be collected or on particular data collection forms and so can be customized for different uses.

OpenMRS is now in use around the world (see the OpenMRS Atlas), including South Africa, Kenya, Rwanda, Lesotho, Zimbabwe, Mozambique, Uganda, Tanzania, Haiti, India, China, United States, Pakistan, the Philippines, and many other places. This work is supported in part by many organizations including international and government aid groups, NGO’s, as well as for-profit and nonprofit corporations.

Main Objective.

Create a framework that can capture system and server metrics for OpenMRS.

Goals

Create a New module with database configurations:- Completed.
Ability to subscribe to the ActiveMQ service bus to listen to the Hibernate events of selected resources.:- Completed.
Persist events in a flat structured table in the OpenMRS db :- Completed.
Extend DropWizard JMX instrumental library to expose System metrics requested.:- Completed.
Expose DropWizard Server metrics:- Completed.
Expose Ping Endpoint provided by DropWizard:- Completed.
Add sample UI to showcase system and server metrics:- Completed
Add the DAO level and Service level Unit tests for the project. :- Completed
Add a 3rd party plugin flow for monitoring tools like Graphite and Prometheus:- Completed
Easy to extend the module for resources required.
Easy to handle date ranges for the application:- Completed.

Project Repository

openmrs/openmrs-module-metrics

Contributions

Talk Thread links

Gsoc 2020: Expose System Metrics For Monitoring.

Blog Posts

Future Works

Extend the flow and the flat table structure to expose more information on resources.

Final Presentation

https://medium.com/media/a435000ddd1a0be741a07344b2d26ca2/href

Thoughts on GSoC

GSoC is one of the main programmes in the world which encourage and helps people to engage in opensource development. The experience it brings to developers is very valuable for there future work.

As I always mention openMRS is one of the best opensource communities in the world. My mentors were always helpful for me when proceeding on the project. And they motivated me to work towards the end of the project after going through the work I do.

Finally I would like to thank openMRS community who gave me this opportunity to work with the community and also to Google for organizing such an awesome opensource event for the students.

GSoC@OpenMRS 12thWeek

Ayeshmantha Perera — Sun, 23 Aug 2020 21:53:52 GMT

It’s the final week of the coding phase.Nothing much to add in the mean time working on below areas before the final evaluation starts.

Adding a small UI to showcase system and server metrics.
Working on the final presentation.

That’s it for the week will be publishing another blog in this week with all work done and what’s next on the project.Till then cheers.

GSoC@OpenMRS 11thWeek

Ayeshmantha Perera — Sun, 16 Aug 2020 22:51:26 GMT

It’s been 11 weeks now since the start of the coding period now.Code related work for the project has been done.It’s a great pleasure to see the community interests towards the project.The community is awesome as always they are very helpful for each other members.

After going through the second phase of GSoC my main goal is to finalizing the work on the project.

What is OpenMRS & Where is it?

OpenMRS is a software platform and a reference application that enables the design of a customized medical records system with no programming knowledge (although medical and systems analysis knowledge is required). It is a common platform upon which medical informatics efforts in developing countries can be built. The system is based on a conceptual database structure which is not dependent on the actual types of medical information required to be collected or on particular data collection forms and so can be customized for different uses.

As mentioned above openMRS is very popular among the African countries.And it’s a very well known fact that mobile devices are very popular among these countries.Without having mobile responsive UI it’s very hard for users who are using mobile devices to use the application.This is the main influence for the project to be take part on the GSoC’20.

I worked on finalizing the below or and getting it merged. last week.

METRIC-10: Add the system metric registration build flow. by Akayeshmantha · Pull Request #20 · openmrs/openmrs-module-metrics

GSoC@OpenMRS 10thWeek

Ayeshmantha Perera — Mon, 10 Aug 2020 08:33:25 GMT

It’s been 10 weeks now since the start of the coding period now. Most of the code related work for the project has been done. There are some bugs to be fixed before moving forward. It’s a great pleasure to see the community's interests in the project. The community is awesome as always they are very helpful for each other members.

After going through the second phase of GSoC my main goal is to do the documentation for the project. Up to now have not started working on the documentation since am finalizing the code yet.

Below is a summary of the work I worked in the past week.

Exposing system metrics through a servlet

Exposing server metrics through a servlet

Working on time range based filtering

Worked on dao and service layer unit tests.

Created the below PR which was not merged yet.

METRIC-10 : Add the system metric registration build flow. by Akayeshmantha · Pull Request #20 · openmrs/openmrs-module-metrics

GSoC@OpenMRS 9thWeek

Ayeshmantha Perera — Sun, 02 Aug 2020 17:30:19 GMT

It’s a great pleasure for me to get through the second phase of the project and passing the 2nd milestone. Most of the work related to the project is almost done. In the last week, I worked on tickets related to the module and got them merged. I finally got the chance to expose the metrics through a servlet that extends the default jmxservlet class of dropwizards.

Normally it is easy to expose the servlet with dropwizard with their default servlet class it has everything inbuilt. We need to configure a listener that listens to the module startup to register a default metric registry on the application context. But in openmrs we don’t support listeners on servlet context. But we support filters so came up with a class that implements the filter interface. and on the init phase we inject the registry instance which is required by the jmxservlet class.

public class MetricsInitializationFilter implements Filter {
	@Override
	public void init(FilterConfig filterConfig) {
		final ServletContext context = filterConfig.getServletContext();
        context.setAttribute(METRICS_REGISTRY, getMetricRegistry());
        context.setAttribute(METRIC_FILTER, getMetricFilter());
        if (getDurationUnit() != null) {
            context.setInitParameter(MetricsServlet.DURATION_UNIT, getDurationUnit().toString());
        }
        if (getRateUnit() != null) {
            context.setInitParameter(MetricsServlet.RATE_UNIT, getRateUnit().toString());
        }
        if (getAllowedOrigin() != null) {
            context.setInitParameter(MetricsServlet.ALLOWED_ORIGIN, getAllowedOrigin());
        }
        if (getJsonpCallbackParameter() != null) {
            context.setAttribute(CALLBACK_PARAM, getJsonpCallbackParameter());
        }
	}	
 @Override
 public void doFilter(ServletRequest req, ServletResponse res,     FilterChain chain) throws IOException, ServletException {
     chain.doFilter(req, res);
 }
}

But it was not a success path it was not injecting the registry instance during the server startup.

Then we went on to the doFilter phase of the Filter and tried to inject the registry object in that method.

public class MetricsInitializationFilter implements Filter {	@Override
	public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain) throws IOException, ServletException {
		final ServletContext context = req.getServletContext();
		if (context.getAttribute(METRICS_REGISTRY) == null) {
        	context.setAttribute(METRICS_REGISTRY, getMetricRegistry());
        }        if (context.getAttribute(METRIC_FILTER) == null) {
        	context.setAttribute(METRIC_FILTER, getMetricFilter());
        }		chain.doFilter(req, res);
	}
}

Sadly it went on the down path as well. Then thought about injecting the registry in the init method of the servlet. It was not the solution as well.

Finally, I thought to replicate the jmxservlet in a way that we can inject the metric registry with a custom servlet class so it worked. It was the long story short. Yet I didn’t push the changes.

Also, I was been able to merge below PR’s last week

https://github.com/openmrs/openmrs-module-metrics/pull/14

That’s all this week.

Cheers and hoping to work more in the next few weeks thank you.

GSoC@OpenMRS 8thWeek

Ayeshmantha Perera — Sun, 26 Jul 2020 22:15:39 GMT

The global OpenMRS community works together to build the world’s leading open-source enterprise electronic medical record system platform.

During the 8 weeks of the coding period, I got the opportunity to involve with the community on working toward the main goal to improve health care delivery in resource-constrained environments by coordinating a global community that creates a robust, scalable, user-driven, open-source medical record system platform.

As usually had the sprint call with my mentors last week and defined the work to be done during the week.

For the metrics module, we thought of creating a Jira board and maintain project-related tasks there. Below is the task board for the project.

OpenMRS Issues

So during last week worked on below tickets

Below are the PR’s which closed the issues.

Next week I will be working on finalizing the DefaultMetricServlet which is working as the API endpoint for the custom metric we exposes from the module.

GSoC@OpenMRS 7thWeek

Ayeshmantha Perera — Sun, 19 Jul 2020 19:58:58 GMT

The global OpenMRS community works together to build the world’s leading open-source enterprise electronic medical record system platform.

During the 7 weeks of the coding period, I got the opportunity to involve with the community on working toward the main goal to improve health care delivery in resource-constrained environments by coordinating a global community that creates a robust, scalable, user-driven, open-source medical record system platform.

As usually had the sprint call with my mentors last week and defined the work to be done during the week.

This week mainly worked on the efficiency of the app and resolving small issues on the activator. Since we are subscribing to all events of the required resource types. We found there is an issue in the Events module. For all the events defined it’s scanning and filtering from a for each loop which has around 5000 records which slows down the process of subscribing to a topic.

To overcome came up with a solution to filter once and keep track of the classes and subclasses related for other action types.It’s already implemented and waiting to get merged in the events module.

EVNT-40: Speed up registering multiple actions for a single class by Akayeshmantha · Pull Request #8 · openmrs/openmrs-module-event

Next week I will be working more towards the omod layer of the app.

GSoC@OpenMRS 6thWeek

Ayeshmantha Perera — Sun, 12 Jul 2020 12:19:58 GMT

It’s been 6 weeks since the start of the GSoC coding period. After fixing the issue on the activator have some issues related to beans in Spring. Currently fixed it and now the modules API is working fine with the expected workflow.

I would like to thank @dkaiwa who helped out on fixing the issue last week.

Currently, the module is capable of capturing events fired by events module and building the related resources object and saving in the database for further reference. Also, it has the capability to filter out objects from the events module as well. We followed the same way that attomfeed module follows on filtering the resource objects which are required.

Also, I got some time to work with an on fhir module and worked a little bit on it to get the flow of how it works so it will help on writing unit tests for the metrics module. Sent few PR’s which are listed below to fhir2 module.

Looking forward to finalizing the client-side of the metrics module within the upcoming weeks.

GSoC@OpenMRS 5thWeek

Ayeshmantha Perera — Mon, 06 Jul 2020 08:20:33 GMT

It’s the successful 5th week in the GSoC 20. Faced the 1st evaluation by google and successfully went through it on yesterday.And now it’s time to get going with the 2nd phase of the project.

Last week worked on exposing the data metrics that we agreed upon for the project.

1 Expose encounters grouped by type

2 No of newly registered patients.

Also worked on the basic JMX workflow.

Separated out JMX report builder and it’s the workflow to separate classes. As well as introduced the service layer which is exposing them metrics to the mood layer as well. Finally got merged the PR by Ian.

DB changes and Queries for the scope of the project. by Akayeshmantha · Pull Request #5 · openmrs/openmrs-module-metrics