Integration between Kobo and dhis2

Resources - Ressources
1

In this post I would like to share our (@solidlines) experience connecting Kobo tool box and dhis2. This experience was already shared in a weekly call of the dhis2 integration working group.

Context

An organization plans to collect surveys in Kobo / KoboToolbox (a free toolkit for collecting data). However, they want to have the surveys on a DHIS2 instance as well. To avoid making the data entry twice, the idea is to transfer the survey information automatically from Kobo to dhis2. The destination in dhis2 is an single event program that was already implemented in the dhis2 instance.

The Kobo data is available using the Kobo API, but some curation data process was needed.

As a requirement, the integration process should be running every 2 hours

In addition, notification emails would sent for different purposes (the process has started, there was a problem in the dhis2 payload…), in order to monitor the automatic process.

Approach/Solution

Architecture
Architecture1049Ă—443 34.6 KB

In order to accomplish this request, an ETL (Extract, Transform, and Load) process was set up using Apache NiFi (an easy to use, powerful, and reliable system to process and distribute data). The main steps of that process are:

  1. Retrieve data from Kobo using the API (CSV) every 2 hours. Each row is a survey submitted in Kobo.
  2. Internal pre-processing (data curation)
  3. Select only the latest surveys to be processed. There was an agreement to process only the surveys of the last 10 days.
  4. Organisation Unit mapping between Kobo OU code system and dhis2 (using an external CSV that needs to be up-to-date)
  5. Internal: Split one CSV file (that contains multiple surveys) into multiple JSON files (future payloads).
  6. Generate the payload to be sent to dhis2 (using JOLT, JSON to JSON transform library). In this step there is also a value mapping process (mainly for dhis2 optionSets)
  7. Double check if the survey was already uploaded (using the dhis2 API).
  8. If the survey was not previously uploaded, send data (event payload) to dhis2 (remember that the event program was already implemented in the dhis2 instance).

In this screenshot you can see the ETL process configured in Apache NiFi

NiFi screenshot
NiFi screenshot1536Ă—657 249 KB

2 Likes
2

Hi @ctejo

Thank you for sharing with the community about your experience using this integration as well as the knowledge and resources. :pray:

If I understand correctly the integration is successful because of a mapping done in Apache, are these steps automated?

What template are you using? And which API endpoint is it being sent to?

Is there any metadata configuration that needs to proceed these steps?

Thanks!

3

Hi @Gassim

If I understand correctly the integration is successful because of a mapping done in Apache, are these steps automated?

In this particular case, the mapping between OrgUnits and Data Elements is fixed (using a csv file for the mapping)

What template are you using? And which API endpoint is it being sent to?

The payload generated is sent to the events endpoint.

Is there any metadata configuration that needs to proceed these steps?

No extra dhis2 configuration is needed.

Let me know if I answer your questions :slight_smile:

1 Like
4

Thanks! I think I will need to try it step by step myself to get the complete picture. :smiley:

The events endpoint is deprecated from version 2.36 so this means this method works on earlier versions, does it work if one uses the /tracker/events endpoint?

Great, so the metadata is all being created in the process as well. Data and metadata is all being created at the same time of the mapping. :+1:

5

The events endpoint is deprecated from version 2.36 so this means this method works on earlier versions, does it work if one uses the /tracker/events endpoint?

Actually, this is still working fine in v40.6 :slight_smile:
BTW, if you want to use the new ´tracker` endpoing, it is needed only a change in the payload template generated + updated the URL of the endpoint.

Great, so the metadata is all being created in the process as well. Data and metadata is all being created at the same time of the mapping.
The program configuration was created previously in the dhis2. Let me update the initial post in order to reflect this.

1 Like
6

Thanks!

Good to know it’s easy to use the newer endpoint, and that the deprecated one is also still working.

Yes, thanks! That’s what I was wondering :pray::pray: