Meta Data Export Question

Hello Hello!, Hot on the heels of my previous query, I have another one associated with CatOptionCombos

What is the best way to migrate a single dataset and its constituents from one DHIS2 system to another such that I can then export datavalues from the source to the target DHIS2 instance? I can get most of the way there with the following:

  1. Select Dataset as a detailed metadata export:

  2. Export with all dependencies
    Inline image 2

  3. Import the resulting file into a different DHIS2 instance

This process will fail initially, but with manual edits, you can work around the gotchas listed below:

  • The users are considered a dependency and exported, if the roles aren’t the same between the two systems it creates problems, so I typically delete the section. They only cause problems when updating certain bits of metadata, the system will complain that it can’t find the user, but otherwise will import just fine. Perhaps something like the following? Just throwing it out there :grin:
    Inline image 3

  • Categories and Category Options are out of order in the XML output, but the system seems to reorder them on import, so no issues here

  • CategoryOptionCombos are not exported, with all the other dependencies. Further more, there is no “CategoryOptionCombos”, (or “CategoryOptions)” listed to choose from in the metadata detailed exporter :disappointed:. Sure these can all be generated with the maintenance operation, but the result will be CatOptionCombos with completely different UIDs than the source system, which would break any ability to upload data from the source server. Furthermore, there are some references to categoryOptionCombo in the dataset XML with dependencies file if you have any , but this won’t import them, just cause the system not to find the categoryOptionCombo it is referencing.

  • I work around this by doing something like this: https://apps.dhis2.org/demo/api/categoryOptionCombos.xml?fields=name,id,displayName,categoryCombo[name,id],categoryOptions[name,id]&paging=false&filter=categoryCombo.id:eq:m2jTvAj5kkm and repeat for each category combo covered by this dataset, and cut and paste the results into the XML file generated in step 2. This is hugely time consuming if there are a lot of catcombos used by the dataset in question. Sadly the API wont let me filter on just a dataset, because there is no reverse link between category combos and data elements (one way only)

  • Defaults for Category Options, Categories, Category Combinations need to be updated (Find&Replace) with the UIDs in use by the target system. Not sure exactly how to fix this, other than making an exception in the import process for any of those three with the name “default”. Otherwise it is manual work on the export file each time.

  • ERROR 2015-11-18 09:53:57,138 ERROR: duplicate key value violates unique constraint “uk_pbj3u1nk9vnuof8f47utvowmv”
    Detail: Key (name)=(default) already exists. (SqlExceptionHelper.java [taskScheduler-1])
  • dissaggregation vs DISSAGGREGATION, in v2.20, the exporter keeps them lowercase, and the importer now expects uppercase. Why the distinction though?
  • ERROR 2015-11-18 09:45:36,496 com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not construct instance of org.hisp.dhis.common.DataDimensionType from String value ‘disaggregation’: value not one of declared Enum instance names: [DISAGGREGATION, ATTRIBUTE]

Once I contend with all the gotchas I listed, I am able to get an import from a source DHIS2 installation to the Target DHIS2 installation with the proper UIDs for the categoryOptionCombos, but it makes me think that there has to be an easier way. Am I missing a button or a step somewhere? The “default” UIDs and the catoptioncombos are the hairiest part of the whole process and probably wouldn’t be easy for most users.

The only other thing I can think of is to use the vanilla metadata exporter, but this is not feasible when there are 15 datasets in the source system and I only need to export 1.

image

···

Timothy Harding
Sr. Systems Analyst, BAO Systems

+1 202-536-1541 | tharding@baosystems.com | http://www.baosystems.com | Skype: hardingt@gmail.com | 2900 K Street, Suite 404, Washington D.C. 20007

Hello DHIS2 Devs and Community!

I’ve been doing some digging and it appears that this is about the easiest way to do this. I’ve got a question for the devs though: Is it a bug or intended behavior that when I export a dataset with dependencies it sends me a file sans CategoryOptionCombos?

image

image

image

···

On Wed, Nov 18, 2015 at 11:36 AM, Timothy Harding tharding@baosystems.com wrote:

Hello Hello!, Hot on the heels of my previous query, I have another one associated with CatOptionCombos

What is the best way to migrate a single dataset and its constituents from one DHIS2 system to another such that I can then export datavalues from the source to the target DHIS2 instance? I can get most of the way there with the following:

  1. Select Dataset as a detailed metadata export:
  1. Export with all dependencies
  1. Import the resulting file into a different DHIS2 instance

This process will fail initially, but with manual edits, you can work around the gotchas listed below:

  • The users are considered a dependency and exported, if the roles aren’t the same between the two systems it creates problems, so I typically delete the section. They only cause problems when updating certain bits of metadata, the system will complain that it can’t find the user, but otherwise will import just fine. Perhaps something like the following? Just throwing it out there :grin:
  • Categories and Category Options are out of order in the XML output, but the system seems to reorder them on import, so no issues here
  • CategoryOptionCombos are not exported, with all the other dependencies. Further more, there is no “CategoryOptionCombos”, (or “CategoryOptions)” listed to choose from in the metadata detailed exporter :disappointed:. Sure these can all be generated with the maintenance operation, but the result will be CatOptionCombos with completely different UIDs than the source system, which would break any ability to upload data from the source server. Furthermore, there are some references to categoryOptionCombo in the dataset XML with dependencies file if you have any , but this won’t import them, just cause the system not to find the categoryOptionCombo it is referencing.
  • I work around this by doing something like this: https://apps.dhis2.org/demo/api/categoryOptionCombos.xml?fields=name,id,displayName,categoryCombo[name,id],categoryOptions[name,id]&paging=false&filter=categoryCombo.id:eq:m2jTvAj5kkm and repeat for each category combo covered by this dataset, and cut and paste the results into the XML file generated in step 2. This is hugely time consuming if there are a lot of catcombos used by the dataset in question. Sadly the API wont let me filter on just a dataset, because there is no reverse link between category combos and data elements (one way only)
  • Defaults for Category Options, Categories, Category Combinations need to be updated (Find&Replace) with the UIDs in use by the target system. Not sure exactly how to fix this, other than making an exception in the import process for any of those three with the name “default”. Otherwise it is manual work on the export file each time.
  • ERROR 2015-11-18 09:53:57,138 ERROR: duplicate key value violates unique constraint “uk_pbj3u1nk9vnuof8f47utvowmv”
    Detail: Key (name)=(default) already exists. (SqlExceptionHelper.java [taskScheduler-1])
  • dissaggregation vs DISSAGGREGATION, in v2.20, the exporter keeps them lowercase, and the importer now expects uppercase. Why the distinction though?
  • ERROR 2015-11-18 09:45:36,496 com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not construct instance of org.hisp.dhis.common.DataDimensionType from String value ‘disaggregation’: value not one of declared Enum instance names: [DISAGGREGATION, ATTRIBUTE]

Once I contend with all the gotchas I listed, I am able to get an import from a source DHIS2 installation to the Target DHIS2 installation with the proper UIDs for the categoryOptionCombos, but it makes me think that there has to be an easier way. Am I missing a button or a step somewhere? The “default” UIDs and the catoptioncombos are the hairiest part of the whole process and probably wouldn’t be easy for most users.

The only other thing I can think of is to use the vanilla metadata exporter, but this is not feasible when there are 15 datasets in the source system and I only need to export 1.

Timothy Harding
Sr. Systems Analyst, BAO Systems

+1 202-536-1541 | tharding@baosystems.com | http://www.baosystems.com | Skype: hardingt@gmail.com | 2900 K Street, Suite 404, Washington D.C. 20007

Timothy Harding
Sr. Systems Analyst, BAO Systems

+1 202-536-1541 | tharding@baosystems.com | http://www.baosystems.com | Skype: hardingt@gmail.com | 2900 K Street, Suite 404, Washington D.C. 20007

Hi Tim,

yes I agree we must make it simpler to export a full data set including category option combos. It is a common use-case. I have started on a blueprint for it for 2.23 here:

https://blueprints.launchpad.net/dhis2/+spec/data-set-exchange

regards,

Lars

image

image

image

···

On Tue, Dec 15, 2015 at 4:10 PM, Timothy Harding tharding@baosystems.com wrote:

Hello DHIS2 Devs and Community!

I’ve been doing some digging and it appears that this is about the easiest way to do this. I’ve got a question for the devs though: Is it a bug or intended behavior that when I export a dataset with dependencies it sends me a file sans CategoryOptionCombos?


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Timothy Harding
Sr. Systems Analyst, BAO Systems

+1 202-536-1541 | tharding@baosystems.com | http://www.baosystems.com | Skype: hardingt@gmail.com | 2900 K Street, Suite 404, Washington D.C. 20007

On Wed, Nov 18, 2015 at 11:36 AM, Timothy Harding tharding@baosystems.com wrote:

Hello Hello!, Hot on the heels of my previous query, I have another one associated with CatOptionCombos

What is the best way to migrate a single dataset and its constituents from one DHIS2 system to another such that I can then export datavalues from the source to the target DHIS2 instance? I can get most of the way there with the following:

  1. Select Dataset as a detailed metadata export:
  1. Export with all dependencies
  1. Import the resulting file into a different DHIS2 instance

This process will fail initially, but with manual edits, you can work around the gotchas listed below:

  • The users are considered a dependency and exported, if the roles aren’t the same between the two systems it creates problems, so I typically delete the section. They only cause problems when updating certain bits of metadata, the system will complain that it can’t find the user, but otherwise will import just fine. Perhaps something like the following? Just throwing it out there :grin:
  • Categories and Category Options are out of order in the XML output, but the system seems to reorder them on import, so no issues here
  • CategoryOptionCombos are not exported, with all the other dependencies. Further more, there is no “CategoryOptionCombos”, (or “CategoryOptions)” listed to choose from in the metadata detailed exporter :disappointed:. Sure these can all be generated with the maintenance operation, but the result will be CatOptionCombos with completely different UIDs than the source system, which would break any ability to upload data from the source server. Furthermore, there are some references to categoryOptionCombo in the dataset XML with dependencies file if you have any , but this won’t import them, just cause the system not to find the categoryOptionCombo it is referencing.
  • I work around this by doing something like this: https://apps.dhis2.org/demo/api/categoryOptionCombos.xml?fields=name,id,displayName,categoryCombo[name,id],categoryOptions[name,id]&paging=false&filter=categoryCombo.id:eq:m2jTvAj5kkm and repeat for each category combo covered by this dataset, and cut and paste the results into the XML file generated in step 2. This is hugely time consuming if there are a lot of catcombos used by the dataset in question. Sadly the API wont let me filter on just a dataset, because there is no reverse link between category combos and data elements (one way only)
  • Defaults for Category Options, Categories, Category Combinations need to be updated (Find&Replace) with the UIDs in use by the target system. Not sure exactly how to fix this, other than making an exception in the import process for any of those three with the name “default”. Otherwise it is manual work on the export file each time.
  • ERROR 2015-11-18 09:53:57,138 ERROR: duplicate key value violates unique constraint “uk_pbj3u1nk9vnuof8f47utvowmv”
    Detail: Key (name)=(default) already exists. (SqlExceptionHelper.java [taskScheduler-1])
  • dissaggregation vs DISSAGGREGATION, in v2.20, the exporter keeps them lowercase, and the importer now expects uppercase. Why the distinction though?
  • ERROR 2015-11-18 09:45:36,496 com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not construct instance of org.hisp.dhis.common.DataDimensionType from String value ‘disaggregation’: value not one of declared Enum instance names: [DISAGGREGATION, ATTRIBUTE]

Once I contend with all the gotchas I listed, I am able to get an import from a source DHIS2 installation to the Target DHIS2 installation with the proper UIDs for the categoryOptionCombos, but it makes me think that there has to be an easier way. Am I missing a button or a step somewhere? The “default” UIDs and the catoptioncombos are the hairiest part of the whole process and probably wouldn’t be easy for most users.

The only other thing I can think of is to use the vanilla metadata exporter, but this is not feasible when there are 15 datasets in the source system and I only need to export 1.

Timothy Harding
Sr. Systems Analyst, BAO Systems

+1 202-536-1541 | tharding@baosystems.com | http://www.baosystems.com | Skype: hardingt@gmail.com | 2900 K Street, Suite 404, Washington D.C. 20007

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

http://www.dhis2.org