Real-time streaming of data from DHIS2?

Implementation - Implémentation
1

Does DHIS2 have ability to stream data in real-time to a destination (e.g. data lake or SaaS service like Azure stream analytics / Apache Spark / Kafka)?

2

Have you looked at the new event hook feature: Event Hooks - DHIS2 Documentation ?

2 Likes
3

Thanks for the response @claude.mamo it looks like it only supports metadata and scheduled tasks at the moment. I’m hoping to have CRUD functionality streamed in real-time if possible.

1 Like
4

Hi Ed

DHIS2 doesn’t have built in functionality for this. Other than event hooks which have some limitations. But the underlying postgresql database certainly has rich functionality to facilitate this type of requirement.

At a lower level, logical replication slots provide a mature, robust and powerful mechanism for streaming changes in DHIS2. For example, there is a nice article here which describes setting up and using logical replication with wal2json : Unlocking the Power of Wal2Json in Ubuntu with PostgreSQL

A consumer of this type of service could easily enough push that data into something like kafka.

At a higher level of abstraction, debezium can be used to manage the link between dhis2 and kafka.

Because this wal streaming is happening at the database level, there will most often be a requirement for enrichment, For example if you were wanting to provide a streaming orgunit service, you would have to have some middleware which sees the database change as a trigger and go back to the api to pull the enriched orgunt resource, its attributes, groups etc. This is the type of workflow which tools like apache camel are well suited for. Camel has components for dhis2, postgresql, debezium, kafka, FHIR etc.

FWIW we are currently looking at use cases around these mechanisms. Creating a streaming orgunit service like described above being one such example. But there are many possibilities. For example streaming trackedentities as FHIR patients on a kafka topic ,

Regards
Bob

3 Likes
5

Using ActiveMQ Artemis you can achieve this

2 Likes
6

Thanks for the replies @bobj @joseph_man, very helpful. In a nutshell, the answer is yes with some effort. I just wanted to get a feel for the kind of budget we would suggest for this kind of solution. Great architecture suggestions @bobj - they’ll come in handy.
Ed

1 Like