I’m not sure what is causing it, but it shouldn’t be allowed to begin with. Is there any constraint on the UID column in your database?
···
On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad sultanm@thoughtworks.com wrote:
Hi Morten,
Hope you are doing good. Let us know your thoughts on the above issue and please let us know if you need any information that you might require.
Regards,
Sultan Ahamar.
–
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org
On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad sultanm@thoughtworks.com wrote:
Hi Morten,
We have pulled latest code from 2.24 and tried import / export again. We have some interesting observations
DB snapshot:
There are no duplicate items in DB. We double checked.
**Entity endpoint snapshot:
**
Interestingly we found few duplicate entities when we try to fetch this data through the API. In Maintenance app, we are not able to see these duplicate entities. Does it also explain why we get repeating entries (with same UID’s) in the full export JSON as well? Let us know on how to proceed and if you need more input.
Thanks for all the help in advance.
Regards,
Sultan Ahamar.
On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org wrote:
Hi Vanya
This should now have been fixed in master and 224. It will now do a global UID check first, and report back any duplicates, if it finds a duplicate it will remove them from the import and report back an ErrorReport (as it would do with other validation issues). So if it finds a duplicate, you will need to set atomic mode to NONE to have it import and ignore the duplicates.
(btw, there was several duplicates in your file, not sure how that happened… maybe something needs to be cleaned in your source instance)
–
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org
On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org wrote:
Hi Vanya
This should have been caught by the importer… but there are several duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category option groups, same with oqeVQ71LCgY…
I will look into making the validation process more robust… but at least you should know that this file have issues
–
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org
On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org wrote:
Ok, thanks Vanya, I’m looking into it now
–
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org
On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com wrote:
Hi Morten
Thanks for the response.
We ran into another issue while trying to do a full export and import.
As per the previous conversations we took care of all data migrations (as per the new constraints on the DB- pretty much caught by the Importer during the validation phase itself).
Having done that we run into this issue:
The ‘categoryoptiongroup’ import is failing with constraint violation for the UID. We checked the data and there is no repeating UID in the database.
The error text is attached for your reference, as well the payload used for the import.
Thanks for all the help in advance.
Regards
Vanya
On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen morten@dhis2.org wrote:
Thanks Aamer, I’m looking at a similar bug right now… seems some objects don’t get their deps properly attached…
–
With Regards
ThoughtWorks Technologies
Hyderabad
–Stay Hungry Stay Foolish!!
–
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org
On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:
Hi Morten,
Please find the payload for below request
curl -H “Content-Type: application/json” -X GET -u username:password “http://localhost:8888/api/24/metadata?filter=created:gt:2016-07” > metadata.224.newapi.created.json
Import the payload in a fresh instance for import using
curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.created.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort.txt
If the same payload is run with atomicMode=ALL, I am getting ‘Invalid references’ errors which is acceptable. But if it is run with atomicMode=NONE, it is throwing the error for which the complete stack trace is attached earlier. filename: 'output_created_besteffort_trace.txt’
Let me know if the complete payload or any other details are required from my end.
Thanks
Aamer.
On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen morten@dhis2.org wrote:
Hi Aamer
Could you please share the payload of the object where this happen?
–
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org
On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:
Hi Morten,
The Importer was run with atomicMode=ALL. Though the payload is huge, only few errors were shown in ImportSummary. As per errors in previous ImportSummary, I have done below.
- ensured the ‘short name’ field is unique for each of the ‘data elements’ and ‘data element group’ as well
- UID of admin, Super User, tracked entity was changed in payload to be same to where it is getting imported.
I have manually resolved them and ran the importer again with atomicMode=ALL.
I got exception as attached in trace file ‘output_besteffort_trace.txt’
Would these kind of issues be known only after the import has failed? and we need to resolve them one-by-one and as-and-when the importer throws them? Is there any note which has the details about the constraints being introduced in a new DHIS version?
Alternatively, If we export a filtered set of metadata using ‘/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the importer with atomicMode=NONE
curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u admin:district “http://localhost:8080/api/24/metadata?atomicMode=NONE” > output_created_besteffort_trace.txt
Getting below exception in trace. Complete trace also attached.
- INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of type ReportTable (DefaultObjectBundleService.java [qtp289378424-12]) org.hibernate.TransientObjectException: object references an unsaved transient instance - save the transient instance before flushing: org.hisp.dhis.indicator.Indicator
Any suggestions please.
Thanks
Aamer.
On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen morten@dhis2.org wrote:
Hi Aamer
Probably what is happening is that the new database has some constraints that the old one doesn’t. We have had some issues with hibernate in the past, and for certain databases that means that many of the constraints was not applied…
If you look at the messages, you will see that e.g shortName must be unique for data elements… the list of errors is not big though, probably you can go through the payload and manually update the objects which has issues?
Another issue I see is that you are importing users, and those UIDs was only stabilized in 2.23… so you might want to also manually update those (again, should not be a big job), this is not done as a startup routine… just done when creating the admin user etc in a fresh database
–
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org
On Tue, Aug 23, 2016 at 12:00 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:
Any thoughts please on the below approach. Let us know if any details are required from our end around this.
On Wed, Aug 17, 2016 at 1:18 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:
Hi Morten,
We have our metadata configured on a DHIS 2.21 instance. We have sequentially upgraded this instance to 2.22, 2.23 and 2.24 by following the upgrade notes for each of them.
We want to export the metadata from this upgraded 2.24 instance and import it to a fresh instance which is already on 2.24. We followed the below steps.
- Export the metadata from the upgraded 2.24 instance using the new exporter
***curl -H “Content-Type: application/json” -X GET -u username:password http://xxxx:8080/api/24/metadata.json > metadata.224.newapi.json ***
- Import the metadata using the new importer to a fresh instance which is already on 2.24
curl -H “Content-Type: application/json” -X POST --data @metadata.224.newapi.json -u username:password http://yyyy:8080/api/24/metadata > output.txt
We see that none of the objects are getting imported. The complete response of the importer is attached below. Any thoughts please. Also let me know if any further details are required from our end.
Thanks
Aamer.
