We need to make frequent import/export between two different DHIS2 systems. While they are similar and the format “fits”, we know some differences, mainly:
The org units concerned exist on both sides but with different ids
Same for the data elements
In some situation, the export value may be an indicator, to be imported in a data element
I can imagine making the export in CSV, then having a “matching sheet” in XLS with the link between OU in the source system and OU in the destination system.
Is someone doing those kind of operations? What’s your typical flow/way of working?
I think I remember Lars presenting some feature helping with this on one of the last DHIS2 releases, but I’m not sure.
Martin
···
**Martin Van Aken - **Freelance Enthusiast Developer
We’re currently working on something for this – a desktop app that runs on Windows / Mac / Linux that allows you to specify some parameters to ‘map’ from one side to the other and a shell app that runs on a Linux box and reads the config file created by the desktop app once a day to transfer data from the one to the other using the API. We specifically need to transfer aggregate program indicator values from one side to data elements on the other.
Things to think about:
You need to map Organisational units on both sides
You need to specify the from element + category option combo / indicator and the to element + category option combo on the other side
You need to specify the period type and optionally a start / end date so that the system knows what periods to target.
The system is still in development but will effectively enable incremental updates by recording periods already taken care of for each source / destination / ou.
I’ll keep you posted, at this stage we’re not 100% sure when it will be complete, but it should be in the next few weeks.
Ed
···
Hello DHIS2 community,
We need to make frequent import/export between two different DHIS2 systems. While they are similar and the format “fits”, we know some differences, mainly:
The org units concerned exist on both sides but with different ids
Same for the data elements
In some situation, the export value may be an indicator, to be imported in a data element
I can imagine making the export in CSV, then having a “matching sheet” in XLS with the link between OU in the source system and OU in the destination system.
Is someone doing those kind of operations? What’s your typical flow/way of working?
I think I remember Lars presenting some feature helping with this on one of the last DHIS2 releases, but I’m not sure.
Yes, basically the Idea would be to extract metadata/data from one instance (source), do some transformations and load into the other instance (destination). You can extract and load the metadata/data directly using the dhis2 import/export App or work extensively using dhis2 APIs. There are various ETL tools that can be used in the process, I prefer using R.
We’re currently working on something for this – a desktop app that runs on Windows / Mac / Linux that allows you to specify some parameters to ‘map’ from one side to the other and a shell app that runs on a Linux box and reads the config file created by the desktop app once a day to transfer data from the one to the other using the API. We specifically need to transfer aggregate program indicator values from one side to data elements on the other.
Things to think about:
You need to map Organisational units on both sides
You need to specify the from element + category option combo / indicator and the to element + category option combo on the other side
You need to specify the period type and optionally a start / end date so that the system knows what periods to target.
The system is still in development but will effectively enable incremental updates by recording periods already taken care of for each source / destination / ou.
I’ll keep you posted, at this stage we’re not 100% sure when it will be complete, but it should be in the next few weeks.
Subject: [Dhis2-users] Import/Export data between similar but distinct systems
Hello DHIS2 community,
We need to make frequent import/export between two different DHIS2 systems. While they are similar and the format “fits”, we know some differences, mainly:
The org units concerned exist on both sides but with different ids
Same for the data elements
In some situation, the export value may be an indicator, to be imported in a data element
I can imagine making the export in CSV, then having a “matching sheet” in XLS with the link between OU in the source system and OU in the destination system.
Is someone doing those kind of operations? What’s your typical flow/way of working?
I think I remember Lars presenting some feature helping with this on one of the last DHIS2 releases, but I’m not sure.
I’ll share the code and details with the group as soon as we are at release 1.0 which I anticipate in the next few weeks. Our approach is to have something that’s easy for a sysadmin to insert new mappings from one side to the other using a GUI.
Regards
Ed
···
Hi Martin,
Yes, basically the Idea would be to extract metadata/data from one instance (source), do some transformations and load into the other instance (destination). You can extract and load the metadata/data directly using the dhis2 import/export App or work extensively using dhis2 APIs. There are various ETL tools that can be used in the process, I prefer using R.
We’re currently working on something for this – a desktop app that runs on Windows / Mac / Linux that allows you to specify some parameters to ‘map’ from one side to the other and a shell app that runs on a Linux box and reads the config file created by the desktop app once a day to transfer data from the one to the other using the API. We specifically need to transfer aggregate program indicator values from one side to data elements on the other.
Things to think about:
You need to map Organisational units on both sides
You need to specify the from element + category option combo / indicator and the to element + category option combo on the other side
You need to specify the period type and optionally a start / end date so that the system knows what periods to target.
The system is still in development but will effectively enable incremental updates by recording periods already taken care of for each source / destination / ou.
I’ll keep you posted, at this stage we’re not 100% sure when it will be complete, but it should be in the next few weeks.
Subject: [Dhis2-users] Import/Export data between similar but distinct systems
Hello DHIS2 community,
We need to make frequent import/export between two different DHIS2 systems. While they are similar and the format “fits”, we know some differences, mainly:
The org units concerned exist on both sides but with different ids
Same for the data elements
In some situation, the export value may be an indicator, to be imported in a data element
I can imagine making the export in CSV, then having a “matching sheet” in XLS with the link between OU in the source system and OU in the destination system.
Is someone doing those kind of operations? What’s your typical flow/way of working?
I think I remember Lars presenting some feature helping with this on one of the last DHIS2 releases, but I’m not sure.
when you say import/export I assume data value exchange here - correct?
The approach I recommend is using the “identifier schemes”. This allows you to keep multiple sets of identifiers (id schemes) for data elements and org units. You can set it up simply by adding unique metadata attributes for data elements and/org units (do so from maintenance app > Attributes). Then enter the identifiers used by “the other side”.
Example: Add a unique metadata attribute called “Foo ID”. Then add those identifiers like “ID001”, “ID002”, for your data elements and org units.
Then during import and/or export, you can specify which id scheme to use for matching data to metadata. You can choose between UID, code and unique attributes.
We need to make frequent import/export between two different DHIS2 systems. While they are similar and the format “fits”, we know some differences, mainly:
The org units concerned exist on both sides but with different ids
Same for the data elements
In some situation, the export value may be an indicator, to be imported in a data element
I can imagine making the export in CSV, then having a “matching sheet” in XLS with the link between OU in the source system and OU in the destination system.
Is someone doing those kind of operations? What’s your typical flow/way of working?
I think I remember Lars presenting some feature helping with this on one of the last DHIS2 releases, but I’m not sure.
Martin
–
**Martin Van Aken - **Freelance Enthusiast Developer