Upgrading DHIS2 from 2.21 to 2.24 and using new exporter/importer

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

output.txt (6.63 KB)

Any thoughts please on the below approach. Let us know if any details are required from our end around this.

···

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

Hi Aamer

Probably what is happening is that the new database has some constraints that the old one doesn’t. We have had some issues with hibernate in the past, and for certain databases that means that many of the constraints was not applied…

If you look at the messages, you will see that e.g shortName must be unique for data elements… the list of errors is not big though, probably you can go through the payload and manually update the objects which has issues?

···

Another issue I see is that you are importing users, and those UIDs was only stabilized in 2.23… so you might want to also manually update those (again, should not be a big job), this is not done as a startup routine… just done when creating the admin user etc in a fresh database

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Any thoughts please on the below approach. Let us know if any details are required from our end around this.

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is huge, only few errors were shown in ImportSummary. As per errors in previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the ‘data elements’ and ‘data element group’ as well

  2. UID of admin, Super User, tracked entity was changed in payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with atomicMode=ALL.

I got exception as attached in trace file ‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has failed? and we need to resolve them one-by-one and as-and-when the importer throws them? Is there any note which has the details about the constraints being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ‘/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the importer with atomicMode=NONE

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort_trace.txt

Getting below exception in trace. Complete trace also attached.

output_created_besteffort_trace.txt (25.3 KB)

output_besteffort_trace.txt (364 KB)

···

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Probably what is happening is that the new database has some constraints that the old one doesn’t. We have had some issues with hibernate in the past, and for certain databases that means that many of the constraints was not applied…

If you look at the messages, you will see that e.g shortName must be unique for data elements… the list of errors is not big though, probably you can go through the payload and manually update the objects which has issues?

Another issue I see is that you are importing users, and those UIDs was only stabilized in 2.23… so you might want to also manually update those (again, should not be a big job), this is not done as a startup routine… just done when creating the admin user etc in a fresh database

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Any thoughts please on the below approach. Let us know if any details are required from our end around this.

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

Hi Aamer

Could you please share the payload of the object where this happen?

···

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is huge, only few errors were shown in ImportSummary. As per errors in previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the ‘data elements’ and ‘data element group’ as well
  1. UID of admin, Super User, tracked entity was changed in payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with atomicMode=ALL.

I got exception as attached in trace file ‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has failed? and we need to resolve them one-by-one and as-and-when the importer throws them? Is there any note which has the details about the constraints being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ‘/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the importer with atomicMode=NONE

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort_trace.txt

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type ReportTable (DefaultObjectBundleService.java [qtp289378424-12]) org.hibernate.TransientObjectException: object references an unsaved transient instance - save the transient instance before flushing: org.hisp.dhis.indicator.Indicator

Any suggestions please.

Thanks

Aamer.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Probably what is happening is that the new database has some constraints that the old one doesn’t. We have had some issues with hibernate in the past, and for certain databases that means that many of the constraints was not applied…

If you look at the messages, you will see that e.g shortName must be unique for data elements… the list of errors is not big though, probably you can go through the payload and manually update the objects which has issues?

Another issue I see is that you are importing users, and those UIDs was only stabilized in 2.23… so you might want to also manually update those (again, should not be a big job), this is not done as a startup routine… just done when creating the admin user etc in a fresh database

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Any thoughts please on the below approach. Let us know if any details are required from our end around this.

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

Hi Morten,

Please find the payload for below request
curl -H “Content-Type: application/json” -X GET -u username:password “http://localhost:8888/api/24/metadata?filter=created:gt:2016-07” > metadata.224.newapi.created.json

Import the payload in a fresh instance for import using

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.created.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort.txt

If the same payload is run with atomicMode=ALL, I am getting ‘Invalid references’ errors which is acceptable. But if it is run with atomicMode=NONE, it is throwing the error for which the complete stack trace is attached earlier. filename: 'output_created_besteffort_trace.txt’

Let me know if the complete payload or any other details are required from my end.

Thanks

Aamer.

metadata.224.newapi.created.json (377 KB)

···

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Could you please share the payload of the object where this happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is huge, only few errors were shown in ImportSummary. As per errors in previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the ‘data elements’ and ‘data element group’ as well
  1. UID of admin, Super User, tracked entity was changed in payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with atomicMode=ALL.

I got exception as attached in trace file ‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has failed? and we need to resolve them one-by-one and as-and-when the importer throws them? Is there any note which has the details about the constraints being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ‘/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the importer with atomicMode=NONE

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort_trace.txt

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type ReportTable (DefaultObjectBundleService.java [qtp289378424-12]) org.hibernate.TransientObjectException: object references an unsaved transient instance - save the transient instance before flushing: org.hisp.dhis.indicator.Indicator

Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Probably what is happening is that the new database has some constraints that the old one doesn’t. We have had some issues with hibernate in the past, and for certain databases that means that many of the constraints was not applied…

If you look at the messages, you will see that e.g shortName must be unique for data elements… the list of errors is not big though, probably you can go through the payload and manually update the objects which has issues?

Another issue I see is that you are importing users, and those UIDs was only stabilized in 2.23… so you might want to also manually update those (again, should not be a big job), this is not done as a startup routine… just done when creating the admin user etc in a fresh database

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Any thoughts please on the below approach. Let us know if any details are required from our end around this.

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

Thanks Aamer, I’m looking at a similar bug right now… seems some objects don’t get their deps properly attached…

···

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

Please find the payload for below request
curl -H “Content-Type: application/json” -X GET -u username:password “http://localhost:8888/api/24/metadata?filter=created:gt:2016-07” > metadata.224.newapi.created.json

Import the payload in a fresh instance for import using

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.created.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort.txt

If the same payload is run with atomicMode=ALL, I am getting ‘Invalid references’ errors which is acceptable. But if it is run with atomicMode=NONE, it is throwing the error for which the complete stack trace is attached earlier. filename: 'output_created_besteffort_trace.txt’

Let me know if the complete payload or any other details are required from my end.

Thanks

Aamer.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Could you please share the payload of the object where this happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is huge, only few errors were shown in ImportSummary. As per errors in previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the ‘data elements’ and ‘data element group’ as well
  1. UID of admin, Super User, tracked entity was changed in payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with atomicMode=ALL.

I got exception as attached in trace file ‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has failed? and we need to resolve them one-by-one and as-and-when the importer throws them? Is there any note which has the details about the constraints being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ‘/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the importer with atomicMode=NONE

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort_trace.txt

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type ReportTable (DefaultObjectBundleService.java [qtp289378424-12]) org.hibernate.TransientObjectException: object references an unsaved transient instance - save the transient instance before flushing: org.hisp.dhis.indicator.Indicator

Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Probably what is happening is that the new database has some constraints that the old one doesn’t. We have had some issues with hibernate in the past, and for certain databases that means that many of the constraints was not applied…

If you look at the messages, you will see that e.g shortName must be unique for data elements… the list of errors is not big though, probably you can go through the payload and manually update the objects which has issues?

Another issue I see is that you are importing users, and those UIDs was only stabilized in 2.23… so you might want to also manually update those (again, should not be a big job), this is not done as a startup routine… just done when creating the admin user etc in a fresh database

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Any thoughts please on the below approach. Let us know if any details are required from our end around this.

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and import.

As per the previous conversations we took care of all data migrations (as
per the new constraints on the DB- pretty much caught by the Importer
during the validation phase itself).

Having done that we run into this issue:
The 'categoryoptiongroup' import is failing with constraint violation for
the UID. We checked the data and there is no repeating UID in the database.

The error text is attached for your reference, as well the payload used for
the import.

Thanks for all the help in advance.

Regards
Vanya

output_metadata.ocba.full_removedcategoryoptiongroup.2_trace2.txt (7.25 KB)

···

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <morten@dhis2.org> wrote:

Thanks Aamer, I'm looking at a similar bug right now.. seems some objects
don't get their deps properly attached..

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <aamerm@thoughtworks.com> > wrote:

Hi Morten,

Please find the payload for below request
*curl -H "Content-Type: application/json" -X GET -u username:password
"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07
<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >
metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H "Content-Type: application/json" -X POST --data
@metadata.224.newapi.created.json -u admin:district
http://localhost:8080/api/24/metadata?atomicMode=NONE
<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting 'Invalid
references' errors which is acceptable. But if it is run with
atomicMode=NONE, it is throwing the error for which the complete stack
trace is attached earlier. filename: '
*output_created_besteffort_trace.txt'*

Let me know if the complete payload or any other details are required
from my end.

Thanks
Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <morten@dhis2.org> >> wrote:

Hi Aamer

Could you please share the payload of the object where this happen?

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <aamerm@thoughtworks.com >>> > wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is huge,
only few errors were shown in ImportSummary. As per errors in previous
ImportSummary, I have done below.
1) ensured the 'short name' field is unique for each of the 'data
elements' and 'data element group' as well
2) UID of admin, Super User, tracked entity was changed in payload to
be same to where it is getting imported.
I have manually resolved them and ran the importer again with
atomicMode=ALL.

I got exception as attached in trace file 'output_besteffort_trace.txt'
Would these kind of issues be known only after the import has failed?
and we need to resolve them one-by-one and as-and-when the importer throws
them? Is there any note which has the details about the constraints being
introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using '
/api/24/metadata?filter=lastUpdated:gt:2016-05' and run the importer
with atomicMode=NONE

*curl -H "Content-Type: application/json" -X POST --data
@metadata.224.newapi.json -u admin:district
http://localhost:8080/api/24/metadata?atomicMode=NONE
<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.
-----
* INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type
ReportTable (DefaultObjectBundleService.java [qtp289378424-12])
org.hibernate.TransientObjectException: object references an unsaved
transient instance - save the transient instance before flushing:
org.hisp.dhis.indicator.Indicator
------
Any suggestions please.

Thanks
Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <morten@dhis2.org> >>>> wrote:

Hi Aamer

Probably what is happening is that the new database has some
constraints that the old one doesn't. We have had some issues with
hibernate in the past, and for certain databases that means that many of
the constraints was not applied..

If you look at the messages, you will see that e.g `shortName` must be
unique for data elements.. the list of errors is not big though, probably
you can go through the payload and manually update the objects which has
issues?

Another issue I see is that you are importing users, and those UIDs
was only stabilized in 2.23.. so you might want to also manually update
those (again, should not be a big job), this is not done as a startup
routine.. just done when creating the admin user etc in a fresh database

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed < >>>>> aamerm@thoughtworks.com> wrote:

Any thoughts please on the below approach. Let us know if any details
are required from our end around this.

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed < >>>>>> aamerm@thoughtworks.com> wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have
sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the
upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and
import it to a fresh instance which is already on 2.24. We followed the
below steps.
1) Export the metadata from the upgraded 2.24 instance using the new
exporter
*curl -H "Content-Type: application/json" -X GET -u
username:password http://xxxx:8080/api/24/metadata.json
<http://xxxx:8080/api/24/metadata.json> > metadata.224.newapi.json *

2) Import the metadata using the new importer to a fresh instance
which is already on 2.24
*curl -H "Content-Type: application/json" -X POST --data
@metadata.224.newapi.json -u username:password
http://yyyy:8080/api/24/metadata >
output.txt*

We see that none of the objects are getting imported. The complete
response of the importer is attached below. Any thoughts please. Also let
me know if any further details are required from our end.

Thanks
Aamer.

--
With Regards
ThoughtWorks Technologies
Hyderabad

--Stay Hungry Stay Foolish!!

Ok, thanks Vanya, I’m looking into it now

···

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and import.

As per the previous conversations we took care of all data migrations (as per the new constraints on the DB- pretty much caught by the Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint violation for the UID. We checked the data and there is no repeating UID in the database.

The error text is attached for your reference, as well the payload used for the import.

Thanks for all the help in advance.

Regards

Vanya

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen morten@dhis2.org wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some objects don’t get their deps properly attached…


With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

Please find the payload for below request
curl -H “Content-Type: application/json” -X GET -u username:password “http://localhost:8888/api/24/metadata?filter=created:gt:2016-07” > metadata.224.newapi.created.json

Import the payload in a fresh instance for import using

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.created.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort.txt

If the same payload is run with atomicMode=ALL, I am getting ‘Invalid references’ errors which is acceptable. But if it is run with atomicMode=NONE, it is throwing the error for which the complete stack trace is attached earlier. filename: 'output_created_besteffort_trace.txt’

Let me know if the complete payload or any other details are required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Could you please share the payload of the object where this happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is huge, only few errors were shown in ImportSummary. As per errors in previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the ‘data elements’ and ‘data element group’ as well
  1. UID of admin, Super User, tracked entity was changed in payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with atomicMode=ALL.

I got exception as attached in trace file ‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has failed? and we need to resolve them one-by-one and as-and-when the importer throws them? Is there any note which has the details about the constraints being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ‘/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the importer with atomicMode=NONE

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort_trace.txt

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type ReportTable (DefaultObjectBundleService.java [qtp289378424-12]) org.hibernate.TransientObjectException: object references an unsaved transient instance - save the transient instance before flushing: org.hisp.dhis.indicator.Indicator

Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Probably what is happening is that the new database has some constraints that the old one doesn’t. We have had some issues with hibernate in the past, and for certain databases that means that many of the constraints was not applied…

If you look at the messages, you will see that e.g shortName must be unique for data elements… the list of errors is not big though, probably you can go through the payload and manually update the objects which has issues?

Another issue I see is that you are importing users, and those UIDs was only stabilized in 2.23… so you might want to also manually update those (again, should not be a big job), this is not done as a startup routine… just done when creating the admin user etc in a fresh database

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Any thoughts please on the below approach. Let us know if any details are required from our end around this.

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

Hi Vanya

This should have been caught by the importer… but there are several duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at least you should know that this file have issues

···

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and import.

As per the previous conversations we took care of all data migrations (as per the new constraints on the DB- pretty much caught by the Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint violation for the UID. We checked the data and there is no repeating UID in the database.

The error text is attached for your reference, as well the payload used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen morten@dhis2.org wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some objects don’t get their deps properly attached…


With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

Please find the payload for below request
curl -H “Content-Type: application/json” -X GET -u username:password “http://localhost:8888/api/24/metadata?filter=created:gt:2016-07” > metadata.224.newapi.created.json

Import the payload in a fresh instance for import using

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.created.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort.txt

If the same payload is run with atomicMode=ALL, I am getting ‘Invalid references’ errors which is acceptable. But if it is run with atomicMode=NONE, it is throwing the error for which the complete stack trace is attached earlier. filename: 'output_created_besteffort_trace.txt’

Let me know if the complete payload or any other details are required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Could you please share the payload of the object where this happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is huge, only few errors were shown in ImportSummary. As per errors in previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the ‘data elements’ and ‘data element group’ as well
  1. UID of admin, Super User, tracked entity was changed in payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with atomicMode=ALL.

I got exception as attached in trace file ‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has failed? and we need to resolve them one-by-one and as-and-when the importer throws them? Is there any note which has the details about the constraints being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ‘/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the importer with atomicMode=NONE

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort_trace.txt

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type ReportTable (DefaultObjectBundleService.java [qtp289378424-12]) org.hibernate.TransientObjectException: object references an unsaved transient instance - save the transient instance before flushing: org.hisp.dhis.indicator.Indicator

Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Probably what is happening is that the new database has some constraints that the old one doesn’t. We have had some issues with hibernate in the past, and for certain databases that means that many of the constraints was not applied…

If you look at the messages, you will see that e.g shortName must be unique for data elements… the list of errors is not big though, probably you can go through the payload and manually update the objects which has issues?

Another issue I see is that you are importing users, and those UIDs was only stabilized in 2.23… so you might want to also manually update those (again, should not be a big job), this is not done as a startup routine… just done when creating the admin user etc in a fresh database

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Any thoughts please on the below approach. Let us know if any details are required from our end around this.

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

Hi Vanya

This should now have been fixed in master and 224. It will now do a global UID check first, and report back any duplicates, if it finds a duplicate it will remove them from the import and report back an ErrorReport (as it would do with other validation issues). So if it finds a duplicate, you will need to set atomic mode to NONE to have it import and ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that happened… maybe something needs to be cleaned in your source instance)

···

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Vanya

This should have been caught by the importer… but there are several duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and import.

As per the previous conversations we took care of all data migrations (as per the new constraints on the DB- pretty much caught by the Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint violation for the UID. We checked the data and there is no repeating UID in the database.

The error text is attached for your reference, as well the payload used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen morten@dhis2.org wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some objects don’t get their deps properly attached…


With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

Please find the payload for below request
curl -H “Content-Type: application/json” -X GET -u username:password “http://localhost:8888/api/24/metadata?filter=created:gt:2016-07” > metadata.224.newapi.created.json

Import the payload in a fresh instance for import using

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.created.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort.txt

If the same payload is run with atomicMode=ALL, I am getting ‘Invalid references’ errors which is acceptable. But if it is run with atomicMode=NONE, it is throwing the error for which the complete stack trace is attached earlier. filename: 'output_created_besteffort_trace.txt’

Let me know if the complete payload or any other details are required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Could you please share the payload of the object where this happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is huge, only few errors were shown in ImportSummary. As per errors in previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the ‘data elements’ and ‘data element group’ as well
  1. UID of admin, Super User, tracked entity was changed in payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with atomicMode=ALL.

I got exception as attached in trace file ‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has failed? and we need to resolve them one-by-one and as-and-when the importer throws them? Is there any note which has the details about the constraints being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ‘/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the importer with atomicMode=NONE

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort_trace.txt

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type ReportTable (DefaultObjectBundleService.java [qtp289378424-12]) org.hibernate.TransientObjectException: object references an unsaved transient instance - save the transient instance before flushing: org.hisp.dhis.indicator.Indicator

Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Probably what is happening is that the new database has some constraints that the old one doesn’t. We have had some issues with hibernate in the past, and for certain databases that means that many of the constraints was not applied…

If you look at the messages, you will see that e.g shortName must be unique for data elements… the list of errors is not big though, probably you can go through the payload and manually update the objects which has issues?

Another issue I see is that you are importing users, and those UIDs was only stabilized in 2.23… so you might want to also manually update those (again, should not be a big job), this is not done as a startup routine… just done when creating the admin user etc in a fresh database

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Any thoughts please on the below approach. Let us know if any details are required from our end around this.

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again. We have some interesting observations

DB snapshot:


There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch this data through the API. In Maintenance app, we are not able to see these duplicate entities. Does it also explain why we get repeating entries (with same UID’s) in the full export JSON as well? Let us know on how to proceed and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

···

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a global UID check first, and report back any duplicates, if it finds a duplicate it will remove them from the import and report back an ErrorReport (as it would do with other validation issues). So if it finds a duplicate, you will need to set atomic mode to NONE to have it import and ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Vanya

This should have been caught by the importer… but there are several duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and import.

As per the previous conversations we took care of all data migrations (as per the new constraints on the DB- pretty much caught by the Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint violation for the UID. We checked the data and there is no repeating UID in the database.

The error text is attached for your reference, as well the payload used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen morten@dhis2.org wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some objects don’t get their deps properly attached…


With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

Please find the payload for below request
curl -H “Content-Type: application/json” -X GET -u username:password “http://localhost:8888/api/24/metadata?filter=created:gt:2016-07” > metadata.224.newapi.created.json

Import the payload in a fresh instance for import using

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.created.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort.txt

If the same payload is run with atomicMode=ALL, I am getting ‘Invalid references’ errors which is acceptable. But if it is run with atomicMode=NONE, it is throwing the error for which the complete stack trace is attached earlier. filename: 'output_created_besteffort_trace.txt’

Let me know if the complete payload or any other details are required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Could you please share the payload of the object where this happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is huge, only few errors were shown in ImportSummary. As per errors in previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the ‘data elements’ and ‘data element group’ as well
  1. UID of admin, Super User, tracked entity was changed in payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with atomicMode=ALL.

I got exception as attached in trace file ‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has failed? and we need to resolve them one-by-one and as-and-when the importer throws them? Is there any note which has the details about the constraints being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ‘/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the importer with atomicMode=NONE

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort_trace.txt

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type ReportTable (DefaultObjectBundleService.java [qtp289378424-12]) org.hibernate.TransientObjectException: object references an unsaved transient instance - save the transient instance before flushing: org.hisp.dhis.indicator.Indicator

Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Probably what is happening is that the new database has some constraints that the old one doesn’t. We have had some issues with hibernate in the past, and for certain databases that means that many of the constraints was not applied…

If you look at the messages, you will see that e.g shortName must be unique for data elements… the list of errors is not big though, probably you can go through the payload and manually update the objects which has issues?

Another issue I see is that you are importing users, and those UIDs was only stabilized in 2.23… so you might want to also manually update those (again, should not be a big job), this is not done as a startup routine… just done when creating the admin user etc in a fresh database

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Any thoughts please on the below approach. Let us know if any details are required from our end around this.

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

···

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad sultanm@thoughtworks.com wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again. We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

**Entity endpoint snapshot:
**

​Interestingly we found few duplicate entities when we try to fetch this data through the API. In Maintenance app, we are not able to see these duplicate entities. Does it also explain why we get repeating entries (with same UID’s) in the full export JSON as well? Let us know on how to proceed and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a global UID check first, and report back any duplicates, if it finds a duplicate it will remove them from the import and report back an ErrorReport (as it would do with other validation issues). So if it finds a duplicate, you will need to set atomic mode to NONE to have it import and ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Vanya

This should have been caught by the importer… but there are several duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and import.

As per the previous conversations we took care of all data migrations (as per the new constraints on the DB- pretty much caught by the Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint violation for the UID. We checked the data and there is no repeating UID in the database.

The error text is attached for your reference, as well the payload used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen morten@dhis2.org wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some objects don’t get their deps properly attached…


With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

Please find the payload for below request
curl -H “Content-Type: application/json” -X GET -u username:password “http://localhost:8888/api/24/metadata?filter=created:gt:2016-07” > metadata.224.newapi.created.json

Import the payload in a fresh instance for import using

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.created.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort.txt

If the same payload is run with atomicMode=ALL, I am getting ‘Invalid references’ errors which is acceptable. But if it is run with atomicMode=NONE, it is throwing the error for which the complete stack trace is attached earlier. filename: 'output_created_besteffort_trace.txt’

Let me know if the complete payload or any other details are required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Could you please share the payload of the object where this happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is huge, only few errors were shown in ImportSummary. As per errors in previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the ‘data elements’ and ‘data element group’ as well
  1. UID of admin, Super User, tracked entity was changed in payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with atomicMode=ALL.

I got exception as attached in trace file ‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has failed? and we need to resolve them one-by-one and as-and-when the importer throws them? Is there any note which has the details about the constraints being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ‘/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the importer with atomicMode=NONE

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort_trace.txt

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type ReportTable (DefaultObjectBundleService.java [qtp289378424-12]) org.hibernate.TransientObjectException: object references an unsaved transient instance - save the transient instance before flushing: org.hisp.dhis.indicator.Indicator

Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Probably what is happening is that the new database has some constraints that the old one doesn’t. We have had some issues with hibernate in the past, and for certain databases that means that many of the constraints was not applied…

If you look at the messages, you will see that e.g shortName must be unique for data elements… the list of errors is not big though, probably you can go through the payload and manually update the objects which has issues?

Another issue I see is that you are importing users, and those UIDs was only stabilized in 2.23… so you might want to also manually update those (again, should not be a big job), this is not done as a startup routine… just done when creating the admin user etc in a fresh database

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Any thoughts please on the below approach. Let us know if any details are required from our end around this.

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

I’m not sure what is causing it, but it shouldn’t be allowed to begin with. Is there any constraint on the UID column in your database?

···

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad sultanm@thoughtworks.com wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad sultanm@thoughtworks.com wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again. We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

**Entity endpoint snapshot:
**

​Interestingly we found few duplicate entities when we try to fetch this data through the API. In Maintenance app, we are not able to see these duplicate entities. Does it also explain why we get repeating entries (with same UID’s) in the full export JSON as well? Let us know on how to proceed and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a global UID check first, and report back any duplicates, if it finds a duplicate it will remove them from the import and report back an ErrorReport (as it would do with other validation issues). So if it finds a duplicate, you will need to set atomic mode to NONE to have it import and ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Vanya

This should have been caught by the importer… but there are several duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and import.

As per the previous conversations we took care of all data migrations (as per the new constraints on the DB- pretty much caught by the Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint violation for the UID. We checked the data and there is no repeating UID in the database.

The error text is attached for your reference, as well the payload used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen morten@dhis2.org wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some objects don’t get their deps properly attached…


With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

Please find the payload for below request
curl -H “Content-Type: application/json” -X GET -u username:password “http://localhost:8888/api/24/metadata?filter=created:gt:2016-07” > metadata.224.newapi.created.json

Import the payload in a fresh instance for import using

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.created.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort.txt

If the same payload is run with atomicMode=ALL, I am getting ‘Invalid references’ errors which is acceptable. But if it is run with atomicMode=NONE, it is throwing the error for which the complete stack trace is attached earlier. filename: 'output_created_besteffort_trace.txt’

Let me know if the complete payload or any other details are required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Could you please share the payload of the object where this happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is huge, only few errors were shown in ImportSummary. As per errors in previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the ‘data elements’ and ‘data element group’ as well
  1. UID of admin, Super User, tracked entity was changed in payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with atomicMode=ALL.

I got exception as attached in trace file ‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has failed? and we need to resolve them one-by-one and as-and-when the importer throws them? Is there any note which has the details about the constraints being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ‘/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the importer with atomicMode=NONE

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort_trace.txt

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type ReportTable (DefaultObjectBundleService.java [qtp289378424-12]) org.hibernate.TransientObjectException: object references an unsaved transient instance - save the transient instance before flushing: org.hisp.dhis.indicator.Indicator

Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Probably what is happening is that the new database has some constraints that the old one doesn’t. We have had some issues with hibernate in the past, and for certain databases that means that many of the constraints was not applied…

If you look at the messages, you will see that e.g shortName must be unique for data elements… the list of errors is not big though, probably you can go through the payload and manually update the objects which has issues?

Another issue I see is that you are importing users, and those UIDs was only stabilized in 2.23… so you might want to also manually update those (again, should not be a big job), this is not done as a startup routine… just done when creating the admin user etc in a fresh database

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Any thoughts please on the below approach. Let us know if any details are required from our end around this.

On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Morten,

We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.

We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.

  1. Export the metadata from the upgraded 2.24 instance using the new exporter

***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***

  1. Import the metadata using the new importer to a fresh instance which is already on 2.24

curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt

We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.

Thanks

Aamer.

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think
there is nothing suspicious out here.


Regards,
Sultan Ahamar.

···

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <morten@dhis2.org> wrote:

I'm not sure what is causing it, but it shouldn't be allowed to begin
with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad < > sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue and
please let us know if you need any information that you might require.

Regards,
Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad < >> sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.
We have some interesting observations

*DB snapshot:*


There are no duplicate items in DB. We double checked.

*Entity endpoint snapshot:*

​Interestingly we found few duplicate entities when we try to fetch this
data through the API. In Maintenance app, we are not able to see these
duplicate entities. Does it also explain why we get repeating entries (with
same UID's) in the full export JSON as well? Let us know on how to proceed
and if you need more input.

Thanks for all the help in advance.

Regards,
Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen <morten@dhis2.org> >>> wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a
global UID check first, and report back any duplicates, if it finds a
duplicate it will remove them from the import and report back an
`ErrorReport` (as it would do with other validation issues). So if it finds
a duplicate, you will need to set atomic mode to NONE to have it import and
ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that
happened.. maybe something needs to be cleaned in your source instance)

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <morten@dhis2.org> >>>> wrote:

Hi Vanya

This should have been caught by the importer... but there are several
duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category
option groups, same with oqeVQ71LCgY..

I will look into making the validation process more robust.. but at
least you should know that this file have issues

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <morten@dhis2.org> >>>>> wrote:

Ok, thanks Vanya, I'm looking into it now

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <vanyas@thoughtworks.com> >>>>>> wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and
import.

As per the previous conversations we took care of all data
migrations (as per the new constraints on the DB- pretty much caught by the
Importer during the validation phase itself).

Having done that we run into this issue:
The 'categoryoptiongroup' import is failing with constraint
violation for the UID. We checked the data and there is no repeating UID in
the database.

The error text is attached for your reference, as well the payload
used for the import.

Thanks for all the help in advance.

Regards
Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen < >>>>>>> morten@dhis2.org> wrote:

Thanks Aamer, I'm looking at a similar bug right now.. seems some
objects don't get their deps properly attached..

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed < >>>>>>>> aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request
*curl -H "Content-Type: application/json" -X GET -u
username:password
"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07
<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >
metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H "Content-Type: application/json" -X POST --data
@metadata.224.newapi.created.json -u admin:district
http://localhost:8080/api/24/metadata?atomicMode=NONE
<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting
'Invalid references' errors which is acceptable. But if it is run with
atomicMode=NONE, it is throwing the error for which the complete stack
trace is attached earlier. filename: '
*output_created_besteffort_trace.txt'*

Let me know if the complete payload or any other details are
required from my end.

Thanks
Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen < >>>>>>>>> morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this
happen?

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed < >>>>>>>>>> aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is
huge, only few errors were shown in ImportSummary. As per errors in
previous ImportSummary, I have done below.
1) ensured the 'short name' field is unique for each of the
'data elements' and 'data element group' as well
2) UID of admin, Super User, tracked entity was changed in
payload to be same to where it is getting imported.
I have manually resolved them and ran the importer again with
atomicMode=ALL.

I got exception as attached in trace file
'output_besteffort_trace.txt'
Would these kind of issues be known only after the import has
failed? and we need to resolve them one-by-one and as-and-when the importer
throws them? Is there any note which has the details about the constraints
being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using '
/api/24/metadata?filter=lastUpdated:gt:2016-05' and run the
importer with atomicMode=NONE

*curl -H "Content-Type: application/json" -X POST --data
@metadata.224.newapi.json -u admin:district
http://localhost:8080/api/24/metadata?atomicMode=NONE
<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.
-----
* INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of
type ReportTable (DefaultObjectBundleService.java
[qtp289378424-12]) org.hibernate.TransientObjectException:
object references an unsaved transient instance - save the transient
instance before flushing: org.hisp.dhis.indicator.Indicator
------
Any suggestions please.

Thanks
Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen < >>>>>>>>>>> morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some
constraints that the old one doesn't. We have had some issues with
hibernate in the past, and for certain databases that means that many of
the constraints was not applied..

If you look at the messages, you will see that e.g `shortName`
must be unique for data elements.. the list of errors is not big though,
probably you can go through the payload and manually update the objects
which has issues?

Another issue I see is that you are importing users, and those
UIDs was only stabilized in 2.23.. so you might want to also manually
update those (again, should not be a big job), this is not done as a
startup routine.. just done when creating the admin user etc in a fresh
database

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo

[truncated for moderation]

Hi again

Would it be possible to share your database with me? I haven't seen this
issues anywhere else.

···

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad < sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think
there is nothing suspicious out here.


Regards,
Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <morten@dhis2.org> > wrote:

I'm not sure what is causing it, but it shouldn't be allowed to begin
with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad < >> sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue
and please let us know if you need any information that you might require.

Regards,
Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad < >>> sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.
We have some interesting observations

*DB snapshot:*


There are no duplicate items in DB. We double checked.

*Entity endpoint snapshot:*

​Interestingly we found few duplicate entities when we try to fetch
this data through the API. In Maintenance app, we are not able to see these
duplicate entities. Does it also explain why we get repeating entries (with
same UID's) in the full export JSON as well? Let us know on how to proceed
and if you need more input.

Thanks for all the help in advance.

Regards,
Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen <morten@dhis2.org> >>>> wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a
global UID check first, and report back any duplicates, if it finds a
duplicate it will remove them from the import and report back an
`ErrorReport` (as it would do with other validation issues). So if it finds
a duplicate, you will need to set atomic mode to NONE to have it import and
ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that
happened.. maybe something needs to be cleaned in your source instance)

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <morten@dhis2.org> >>>>> wrote:

Hi Vanya

This should have been caught by the importer... but there are several
duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category
option groups, same with oqeVQ71LCgY..

I will look into making the validation process more robust.. but at
least you should know that this file have issues

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <morten@dhis2.org> >>>>>> wrote:

Ok, thanks Vanya, I'm looking into it now

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <vanyas@thoughtworks.com> >>>>>>> wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and
import.

As per the previous conversations we took care of all data
migrations (as per the new constraints on the DB- pretty much caught by the
Importer during the validation phase itself).

Having done that we run into this issue:
The 'categoryoptiongroup' import is failing with constraint
violation for the UID. We checked the data and there is no repeating UID in
the database.

The error text is attached for your reference, as well the payload
used for the import.

Thanks for all the help in advance.

Regards
Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen < >>>>>>>> morten@dhis2.org> wrote:

Thanks Aamer, I'm looking at a similar bug right now.. seems some
objects don't get their deps properly attached..

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed < >>>>>>>>> aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request
*curl -H "Content-Type: application/json" -X GET -u
username:password
"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07
<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >
metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H "Content-Type: application/json" -X POST --data
@metadata.224.newapi.created.json -u admin:district
http://localhost:8080/api/24/metadata?atomicMode=NONE
<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting
'Invalid references' errors which is acceptable. But if it is run with
atomicMode=NONE, it is throwing the error for which the complete stack
trace is attached earlier. filename: '
*output_created_besteffort_trace.txt'*

Let me know if the complete payload or any other details are
required from my end.

Thanks
Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen < >>>>>>>>>> morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this
happen?

--
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed < >>>>>>>>>>> aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is
huge, only few errors were shown in ImportSummary. As per errors in
previous ImportSummary, I have done below.
1) ensured the 'short name' field is unique for each of the
'data elements' and 'data element group' as well
2) UID of admin, Super User, tracked entity was changed in
payload to be same to where it is getting imported.
I have manually resolved them and ran the importer again with
atomicMode=ALL.

I got exception as attached in trace file
'output_besteffort_trace.txt'
Would these kind of issues be known only after the import has
failed? and we need to resolve them one-by-one and as-and-when the importer
throws them? Is there any note which has the details about the constraints
being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using '
/api/24/metadata?filter=lastUpdated:gt:2016-05' and run the
importer with atomicMode=NONE

*curl -H "Content-Type: application/json" -X POST --data
@metadata.224.newapi.json -u admin:district
http://localhost:8080/api/24/metadata?atomicMode=NONE
<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >
output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.
-----
* INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of
type ReportTable (DefaultObjectBundleService.java
[qtp289378424-12]) org.hibernate.TransientObjectException:
object references an unsaved transient instance - save the transient
instance before flushing: org.hisp.dhis.indicator.Indicator
------
Any suggestions please.

Thanks
Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen < >>>>>>>>>>>> morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some
constraints that the old one doesn't. We have had some issues with
hibernate in the past, and for certain databases that means that many of
the constraints was not applied..

If you look at the messages, you will see that e.g `shortName`
must be unique for data elements.. the list of errors is not big though,
probably you can go through

[truncated for moderation]

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

···

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <
sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

···

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Thanks Morten. Let us know if any other details are required from our end in this regard.

Thanks

Aamer.

···

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen morten@dhis2.org wrote:

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Aamer

Lars knows what is wrong, he will provide a fix shortly :slight_smile:

···

On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Thanks Morten. Let us know if any other details are required from our end in this regard.

Thanks

Aamer.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen morten@dhis2.org wrote:

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp