Upgrading DHIS2 from 2.21 to 2.24 and using new exporter/importer

Hi Aamer,

the problem here happens because the hibernate mapping between GroupSet and Group is one-to-many, ie. a Group can only be part of one GroupSet.

There is a reason for this, but it is not very good :wink: When we store favorites with group sets, we need to know which dimension (GroupSet) a dimension item (Group) is part of. If the Group is part of many GroupSets, we need another link object in between to represent which dimension it is part of. This can clearly be done and this is something we plan to fix, but it is a bit of work and it requires an upgrade routine to upgrade existing databases.

So I am pretty sure that is why you see these duplicates: You have groups which are part of multiple group sets. In these cases, Hibernate will create duplicate GroupSets as a result of the database integrity issue.

So short term solution is to change the metadata, long term we will fix this properly and allow for many-to-many between Group and GroupSet.

regards,

Lars

···

On Thu, Sep 15, 2016 at 7:28 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Lars knows what is wrong, he will provide a fix shortly :slight_smile:

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Thanks Morten. Let us know if any other details are required from our end in this regard.

Thanks

Aamer.

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen morten@dhis2.org wrote:

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Hi,

For continuation of testing, we changed the metadata to ensure that a Group can be part of only one GroupSet.

  1. We got the below issue.

{“status”:“ERROR”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“typeReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“objectReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“index”:277,“uid”:“oMbElS3lhOO”,“errorReports”:[{“message”:“Missing required property numerator.”,“mainKlass”:“org.hisp.dhis.indicator.Indicator”,“errorKlass”:“java.lang.String”,“errorCode”:“E4000”}]}]}]}

which means that Numerator is required field for an indicator. This was not the case in 2.21. This restriction was introduced in some version after 2.21. However, we modified the metadata for the indicator having this issue and proceeded further.

  1. Now we got the below issue.

{“httpStatus”:“Internal Server Error”,“httpStatusCode”:500,“status”:“ERROR”,“message”:“ERROR: duplicate key value violates unique constraint "dataelement_operand_unique_key"\n Detail: Key (dataelementid, categoryoptioncomboid)=(4497, 3576) already exists.”}

This means that you cannot have duplicate combination of (dataelementid, categoryoptioncomboid) in dataelementoperand table. This constraint was not present in 2.21 version.

When we see dataelementoperand table, there are lot of such duplicates. We would like to understand more around this constraint and why it was introduced.

Is there any note which has the details about the constraints being introduced in a DHIS version?

  1. Also this constraint (dataelement_operand_unique_key) is not present in the instance which was upgraded from 2.21 to 2.24. But it is present in the instance which is directly on 2.24. Any thoughts on this. Did the constraint not get created in upgraded instance because the data was not abiding to the constraint rules? What happens in a dhis upgrade if there is any constraint being introduced and existing data does not confirm to the constraint? Please suggest.

Thanks

Aamer.

···

On Sat, Sep 24, 2016 at 5:54 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Aamer,

the problem here happens because the hibernate mapping between GroupSet and Group is one-to-many, ie. a Group can only be part of one GroupSet.

There is a reason for this, but it is not very good :wink: When we store favorites with group sets, we need to know which dimension (GroupSet) a dimension item (Group) is part of. If the Group is part of many GroupSets, we need another link object in between to represent which dimension it is part of. This can clearly be done and this is something we plan to fix, but it is a bit of work and it requires an upgrade routine to upgrade existing databases.

So I am pretty sure that is why you see these duplicates: You have groups which are part of multiple group sets. In these cases, Hibernate will create duplicate GroupSets as a result of the database integrity issue.

So short term solution is to change the metadata, long term we will fix this properly and allow for many-to-many between Group and GroupSet.

regards,

Lars

On Thu, Sep 15, 2016 at 7:28 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Lars knows what is wrong, he will provide a fix shortly :slight_smile:

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Thanks Morten. Let us know if any other details are required from our end in this regard.

Thanks

Aamer.

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen morten@dhis2.org wrote:

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Morten and Lars

There are certain key points highlighted in the above email. Whenever a new DHIS2 version comes out there are new integrity constraints added but thats not part of the release or the upgrade notes. So, whats the right way to get information about these constraints?

When we deploy the new war we assume that the TableAlteror will take care of modifying the schema and also take care of adding the new constraints. But in case the existing data does not abide by these constraints how do we apply them?

So, principally after a successful redeploy of the war of a higher version can we say that the DHIS instance has been upgraded in its true sense.

As of now it seems the war does get updated but the underlying database does not.

Needless to say this breaks the export /import of metadata downstream.

Any insights about this would be appreciated.

Regards

Vanya

···

On Thu, Sep 29, 2016 at 5:01 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi,

For continuation of testing, we changed the metadata to ensure that a Group can be part of only one GroupSet.

  1. We got the below issue.

{“status”:“ERROR”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“typeReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“objectReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“index”:277,“uid”:“oMbElS3lhOO”,“errorReports”:[{“message”:“Missing required property numerator.”,“mainKlass”:“org.hisp.dhis.indicator.Indicator”,“errorKlass”:“java.lang.String”,“errorCode”:“E4000”}]}]}]}

which means that Numerator is required field for an indicator. This was not the case in 2.21. This restriction was introduced in some version after 2.21. However, we modified the metadata for the indicator having this issue and proceeded further.

  1. Now we got the below issue.

{“httpStatus”:“Internal Server Error”,“httpStatusCode”:500,“status”:“ERROR”,“message”:“ERROR: duplicate key value violates unique constraint "dataelement_operand_unique_key"\n Detail: Key (dataelementid, categoryoptioncomboid)=(4497, 3576) already exists.”}

This means that you cannot have duplicate combination of (dataelementid, categoryoptioncomboid) in dataelementoperand table. This constraint was not present in 2.21 version.

When we see dataelementoperand table, there are lot of such duplicates. We would like to understand more around this constraint and why it was introduced.

Is there any note which has the details about the constraints being introduced in a DHIS version?

  1. Also this constraint (dataelement_operand_unique_key) is not present in the instance which was upgraded from 2.21 to 2.24. But it is present in the instance which is directly on 2.24. Any thoughts on this. Did the constraint not get created in upgraded instance because the data was not abiding to the constraint rules? What happens in a dhis upgrade if there is any constraint being introduced and existing data does not confirm to the constraint? Please suggest.

Thanks

Aamer.

On Sat, Sep 24, 2016 at 5:54 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Aamer,

the problem here happens because the hibernate mapping between GroupSet and Group is one-to-many, ie. a Group can only be part of one GroupSet.

There is a reason for this, but it is not very good :wink: When we store favorites with group sets, we need to know which dimension (GroupSet) a dimension item (Group) is part of. If the Group is part of many GroupSets, we need another link object in between to represent which dimension it is part of. This can clearly be done and this is something we plan to fix, but it is a bit of work and it requires an upgrade routine to upgrade existing databases.

So I am pretty sure that is why you see these duplicates: You have groups which are part of multiple group sets. In these cases, Hibernate will create duplicate GroupSets as a result of the database integrity issue.

So short term solution is to change the metadata, long term we will fix this properly and allow for many-to-many between Group and GroupSet.

regards,

Lars

On Thu, Sep 15, 2016 at 7:28 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Lars knows what is wrong, he will provide a fix shortly :slight_smile:

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Thanks Morten. Let us know if any other details are required from our end in this regard.

Thanks

Aamer.

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen morten@dhis2.org wrote:

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

Hi Aamer,

···

On Thu, Sep 29, 2016 at 1:31 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi,

For continuation of testing, we changed the metadata to ensure that a Group can be part of only one GroupSet.

  1. We got the below issue.

{“status”:“ERROR”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“typeReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“objectReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“index”:277,“uid”:“oMbElS3lhOO”,“errorReports”:[{“message”:“Missing required property numerator.”,“mainKlass”:“org.hisp.dhis.indicator.Indicator”,“errorKlass”:“java.lang.String”,“errorCode”:“E4000”}]}]}]}

which means that Numerator is required field for an indicator. This was not the case in 2.21. This restriction was introduced in some version after 2.21. However, we modified the metadata for the indicator having this issue and proceeded further.

Right, we did make indicator numerator and denominator not-null in the schema. You can say that this is something we should have done from the very start. In 2.24 we tightened up the schema, which is very helpful to avoid problems in the source code, but could require some modifications in existing solutions.

  1. Now we got the below issue.

{“httpStatus”:“Internal Server Error”,“httpStatusCode”:500,“status”:“ERROR”,“message”:“ERROR: duplicate key value violates unique constraint "dataelement_operand_unique_key"\n Detail: Key (dataelementid, categoryoptioncomboid)=(4497, 3576) already exists.”}

This means that you cannot have duplicate combination of (dataelementid, categoryoptioncomboid) in dataelementoperand table. This constraint was not present in 2.21 version.

When we see dataelementoperand table, there are lot of such duplicates. We would like to understand more around this constraint and why it was introduced.

Is there any note which has the details about the constraints being introduced in a DHIS version?

This is interesting because i) it has been like this for many years and ii) I agree this constraint should not be there. Not sure why this surfaced now - likely because you have started a new db. I have removed this constraint in master and 2.24 now.

best regards,

Lars

  1. Also this constraint (dataelement_operand_unique_key) is not present in the instance which was upgraded from 2.21 to 2.24. But it is present in the instance which is directly on 2.24. Any thoughts on this. Did the constraint not get created in upgraded instance because the data was not abiding to the constraint rules? What happens in a dhis upgrade if there is any constraint being introduced and existing data does not confirm to the constraint? Please suggest.

Thanks

Aamer.

On Sat, Sep 24, 2016 at 5:54 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Aamer,

the problem here happens because the hibernate mapping between GroupSet and Group is one-to-many, ie. a Group can only be part of one GroupSet.

There is a reason for this, but it is not very good :wink: When we store favorites with group sets, we need to know which dimension (GroupSet) a dimension item (Group) is part of. If the Group is part of many GroupSets, we need another link object in between to represent which dimension it is part of. This can clearly be done and this is something we plan to fix, but it is a bit of work and it requires an upgrade routine to upgrade existing databases.

So I am pretty sure that is why you see these duplicates: You have groups which are part of multiple group sets. In these cases, Hibernate will create duplicate GroupSets as a result of the database integrity issue.

So short term solution is to change the metadata, long term we will fix this properly and allow for many-to-many between Group and GroupSet.

regards,

Lars

On Thu, Sep 15, 2016 at 7:28 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Lars knows what is wrong, he will provide a fix shortly :slight_smile:

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Thanks Morten. Let us know if any other details are required from our end in this regard.

Thanks

Aamer.

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen morten@dhis2.org wrote:

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Hi Vanya,

sure. We definitely try to avoid this. See previous email in thread. The operand constraint has been removed. We will include not-null constraints in the upgrade notes.

For 2.25 we will set the “created” and “lastupdated” columns of tables of “identifiable objects” to not-null.

best regards,

Lars

···

On Tue, Oct 4, 2016 at 9:39 AM, Vanya Seth vanyas@thoughtworks.com wrote:

Hi Morten and Lars

There are certain key points highlighted in the above email. Whenever a new DHIS2 version comes out there are new integrity constraints added but thats not part of the release or the upgrade notes. So, whats the right way to get information about these constraints?

When we deploy the new war we assume that the TableAlteror will take care of modifying the schema and also take care of adding the new constraints. But in case the existing data does not abide by these constraints how do we apply them?

So, principally after a successful redeploy of the war of a higher version can we say that the DHIS instance has been upgraded in its true sense.

As of now it seems the war does get updated but the underlying database does not.

Needless to say this breaks the export /import of metadata downstream.

Any insights about this would be appreciated.

Regards

Vanya

On Thu, Sep 29, 2016 at 5:01 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi,

For continuation of testing, we changed the metadata to ensure that a Group can be part of only one GroupSet.

  1. We got the below issue.

{“status”:“ERROR”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“typeReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“objectReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“index”:277,“uid”:“oMbElS3lhOO”,“errorReports”:[{“message”:“Missing required property numerator.”,“mainKlass”:“org.hisp.dhis.indicator.Indicator”,“errorKlass”:“java.lang.String”,“errorCode”:“E4000”}]}]}]}

which means that Numerator is required field for an indicator. This was not the case in 2.21. This restriction was introduced in some version after 2.21. However, we modified the metadata for the indicator having this issue and proceeded further.

  1. Now we got the below issue.

{“httpStatus”:“Internal Server Error”,“httpStatusCode”:500,“status”:“ERROR”,“message”:“ERROR: duplicate key value violates unique constraint "dataelement_operand_unique_key"\n Detail: Key (dataelementid, categoryoptioncomboid)=(4497, 3576) already exists.”}

This means that you cannot have duplicate combination of (dataelementid, categoryoptioncomboid) in dataelementoperand table. This constraint was not present in 2.21 version.

When we see dataelementoperand table, there are lot of such duplicates. We would like to understand more around this constraint and why it was introduced.

Is there any note which has the details about the constraints being introduced in a DHIS version?

  1. Also this constraint (dataelement_operand_unique_key) is not present in the instance which was upgraded from 2.21 to 2.24. But it is present in the instance which is directly on 2.24. Any thoughts on this. Did the constraint not get created in upgraded instance because the data was not abiding to the constraint rules? What happens in a dhis upgrade if there is any constraint being introduced and existing data does not confirm to the constraint? Please suggest.

Thanks

Aamer.


With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

On Sat, Sep 24, 2016 at 5:54 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Aamer,

the problem here happens because the hibernate mapping between GroupSet and Group is one-to-many, ie. a Group can only be part of one GroupSet.

There is a reason for this, but it is not very good :wink: When we store favorites with group sets, we need to know which dimension (GroupSet) a dimension item (Group) is part of. If the Group is part of many GroupSets, we need another link object in between to represent which dimension it is part of. This can clearly be done and this is something we plan to fix, but it is a bit of work and it requires an upgrade routine to upgrade existing databases.

So I am pretty sure that is why you see these duplicates: You have groups which are part of multiple group sets. In these cases, Hibernate will create duplicate GroupSets as a result of the database integrity issue.

So short term solution is to change the metadata, long term we will fix this properly and allow for many-to-many between Group and GroupSet.

regards,

Lars

On Thu, Sep 15, 2016 at 7:28 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Lars knows what is wrong, he will provide a fix shortly :slight_smile:

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Thanks Morten. Let us know if any other details are required from our end in this regard.

Thanks

Aamer.

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen morten@dhis2.org wrote:

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Hi Lars

Thanks for the response.

But have we only introduced not null constraints over different version updates?

But am still not sure how the constraints get applied if the data does not abide by the rules.

What looks like a reasonable process to me is :

  • Update your database data to abide by the new constraints.

  • Deploy the new war.

Thought?

Regards

Vanya

···

On Tue, Oct 4, 2016 at 1:27 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Vanya,

sure. We definitely try to avoid this. See previous email in thread. The operand constraint has been removed. We will include not-null constraints in the upgrade notes.

For 2.25 we will set the “created” and “lastupdated” columns of tables of “identifiable objects” to not-null.

best regards,

Lars

On Tue, Oct 4, 2016 at 9:39 AM, Vanya Seth vanyas@thoughtworks.com wrote:

Hi Morten and Lars

There are certain key points highlighted in the above email. Whenever a new DHIS2 version comes out there are new integrity constraints added but thats not part of the release or the upgrade notes. So, whats the right way to get information about these constraints?

When we deploy the new war we assume that the TableAlteror will take care of modifying the schema and also take care of adding the new constraints. But in case the existing data does not abide by these constraints how do we apply them?

So, principally after a successful redeploy of the war of a higher version can we say that the DHIS instance has been upgraded in its true sense.

As of now it seems the war does get updated but the underlying database does not.

Needless to say this breaks the export /import of metadata downstream.

Any insights about this would be appreciated.

Regards

Vanya


Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

On Thu, Sep 29, 2016 at 5:01 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi,

For continuation of testing, we changed the metadata to ensure that a Group can be part of only one GroupSet.

  1. We got the below issue.

{“status”:“ERROR”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“typeReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“objectReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“index”:277,“uid”:“oMbElS3lhOO”,“errorReports”:[{“message”:“Missing required property numerator.”,“mainKlass”:“org.hisp.dhis.indicator.Indicator”,“errorKlass”:“java.lang.String”,“errorCode”:“E4000”}]}]}]}

which means that Numerator is required field for an indicator. This was not the case in 2.21. This restriction was introduced in some version after 2.21. However, we modified the metadata for the indicator having this issue and proceeded further.

  1. Now we got the below issue.

{“httpStatus”:“Internal Server Error”,“httpStatusCode”:500,“status”:“ERROR”,“message”:“ERROR: duplicate key value violates unique constraint "dataelement_operand_unique_key"\n Detail: Key (dataelementid, categoryoptioncomboid)=(4497, 3576) already exists.”}

This means that you cannot have duplicate combination of (dataelementid, categoryoptioncomboid) in dataelementoperand table. This constraint was not present in 2.21 version.

When we see dataelementoperand table, there are lot of such duplicates. We would like to understand more around this constraint and why it was introduced.

Is there any note which has the details about the constraints being introduced in a DHIS version?

  1. Also this constraint (dataelement_operand_unique_key) is not present in the instance which was upgraded from 2.21 to 2.24. But it is present in the instance which is directly on 2.24. Any thoughts on this. Did the constraint not get created in upgraded instance because the data was not abiding to the constraint rules? What happens in a dhis upgrade if there is any constraint being introduced and existing data does not confirm to the constraint? Please suggest.

Thanks

Aamer.


With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

On Sat, Sep 24, 2016 at 5:54 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Aamer,

the problem here happens because the hibernate mapping between GroupSet and Group is one-to-many, ie. a Group can only be part of one GroupSet.

There is a reason for this, but it is not very good :wink: When we store favorites with group sets, we need to know which dimension (GroupSet) a dimension item (Group) is part of. If the Group is part of many GroupSets, we need another link object in between to represent which dimension it is part of. This can clearly be done and this is something we plan to fix, but it is a bit of work and it requires an upgrade routine to upgrade existing databases.

So I am pretty sure that is why you see these duplicates: You have groups which are part of multiple group sets. In these cases, Hibernate will create duplicate GroupSets as a result of the database integrity issue.

So short term solution is to change the metadata, long term we will fix this properly and allow for many-to-many between Group and GroupSet.

regards,

Lars

On Thu, Sep 15, 2016 at 7:28 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Lars knows what is wrong, he will provide a fix shortly :slight_smile:

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Thanks Morten. Let us know if any other details are required from our end in this regard.

Thanks

Aamer.

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen morten@dhis2.org wrote:

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

Hi Lars,

Thanks for the fix (removal of constraint). Yes we are trying to import the metadata into a new 2.24 db. The metadata is exported from an instance which is upgraded from 2.21 to 2.24.

  1. After your fix, there is no constraint violation error and we are able to continue further. But this time we got the below issue in the later part of the import.

stacktrace_afterconstraintremove.txt (23.6 KB)

···

On Tue, Oct 4, 2016 at 1:23 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Aamer,

On Thu, Sep 29, 2016 at 1:31 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi,

For continuation of testing, we changed the metadata to ensure that a Group can be part of only one GroupSet.

  1. We got the below issue.

{“status”:“ERROR”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“typeReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“objectReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“index”:277,“uid”:“oMbElS3lhOO”,“errorReports”:[{“message”:“Missing required property numerator.”,“mainKlass”:“org.hisp.dhis.indicator.Indicator”,“errorKlass”:“java.lang.String”,“errorCode”:“E4000”}]}]}]}

which means that Numerator is required field for an indicator. This was not the case in 2.21. This restriction was introduced in some version after 2.21. However, we modified the metadata for the indicator having this issue and proceeded further.

Right, we did make indicator numerator and denominator not-null in the schema. You can say that this is something we should have done from the very start. In 2.24 we tightened up the schema, which is very helpful to avoid problems in the source code, but could require some modifications in existing solutions.

  1. Now we got the below issue.

{“httpStatus”:“Internal Server Error”,“httpStatusCode”:500,“status”:“ERROR”,“message”:“ERROR: duplicate key value violates unique constraint "dataelement_operand_unique_key"\n Detail: Key (dataelementid, categoryoptioncomboid)=(4497, 3576) already exists.”}

This means that you cannot have duplicate combination of (dataelementid, categoryoptioncomboid) in dataelementoperand table. This constraint was not present in 2.21 version.

When we see dataelementoperand table, there are lot of such duplicates. We would like to understand more around this constraint and why it was introduced.

Is there any note which has the details about the constraints being introduced in a DHIS version?

This is interesting because i) it has been like this for many years and ii) I agree this constraint should not be there. Not sure why this surfaced now - likely because you have started a new db. I have removed this constraint in master and 2.24 now.

best regards,

Lars

  1. Also this constraint (dataelement_operand_unique_key) is not present in the instance which was upgraded from 2.21 to 2.24. But it is present in the instance which is directly on 2.24. Any thoughts on this. Did the constraint not get created in upgraded instance because the data was not abiding to the constraint rules? What happens in a dhis upgrade if there is any constraint being introduced and existing data does not confirm to the constraint? Please suggest.

Thanks

Aamer.


Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

On Sat, Sep 24, 2016 at 5:54 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Aamer,

the problem here happens because the hibernate mapping between GroupSet and Group is one-to-many, ie. a Group can only be part of one GroupSet.

There is a reason for this, but it is not very good :wink: When we store favorites with group sets, we need to know which dimension (GroupSet) a dimension item (Group) is part of. If the Group is part of many GroupSets, we need another link object in between to represent which dimension it is part of. This can clearly be done and this is something we plan to fix, but it is a bit of work and it requires an upgrade routine to upgrade existing databases.

So I am pretty sure that is why you see these duplicates: You have groups which are part of multiple group sets. In these cases, Hibernate will create duplicate GroupSets as a result of the database integrity issue.

So short term solution is to change the metadata, long term we will fix this properly and allow for many-to-many between Group and GroupSet.

regards,

Lars

On Thu, Sep 15, 2016 at 7:28 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Lars knows what is wrong, he will provide a fix shortly :slight_smile:

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Thanks Morten. Let us know if any other details are required from our end in this regard.

Thanks

Aamer.

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen morten@dhis2.org wrote:

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Vanya,

yes agree, now we typically embed the upgrade statements within DHIS 2 itself, as we have learnt that people often forget to run upgrade statements which causes all kinds of problems.

regards,

Lars

···

On Tue, Oct 4, 2016 at 11:31 AM, Vanya Seth vanyas@thoughtworks.com wrote:

Hi Lars

Thanks for the response.

But have we only introduced not null constraints over different version updates?

But am still not sure how the constraints get applied if the data does not abide by the rules.

What looks like a reasonable process to me is :

  • Update your database data to abide by the new constraints.
  • Deploy the new war.

Thought?

Regards

Vanya

On Tue, Oct 4, 2016 at 1:27 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Vanya,

sure. We definitely try to avoid this. See previous email in thread. The operand constraint has been removed. We will include not-null constraints in the upgrade notes.

For 2.25 we will set the “created” and “lastupdated” columns of tables of “identifiable objects” to not-null.

best regards,

Lars


With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

On Tue, Oct 4, 2016 at 9:39 AM, Vanya Seth vanyas@thoughtworks.com wrote:

Hi Morten and Lars

There are certain key points highlighted in the above email. Whenever a new DHIS2 version comes out there are new integrity constraints added but thats not part of the release or the upgrade notes. So, whats the right way to get information about these constraints?

When we deploy the new war we assume that the TableAlteror will take care of modifying the schema and also take care of adding the new constraints. But in case the existing data does not abide by these constraints how do we apply them?

So, principally after a successful redeploy of the war of a higher version can we say that the DHIS instance has been upgraded in its true sense.

As of now it seems the war does get updated but the underlying database does not.

Needless to say this breaks the export /import of metadata downstream.

Any insights about this would be appreciated.

Regards

Vanya


Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

On Thu, Sep 29, 2016 at 5:01 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi,

For continuation of testing, we changed the metadata to ensure that a Group can be part of only one GroupSet.

  1. We got the below issue.

{“status”:“ERROR”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“typeReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“objectReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“index”:277,“uid”:“oMbElS3lhOO”,“errorReports”:[{“message”:“Missing required property numerator.”,“mainKlass”:“org.hisp.dhis.indicator.Indicator”,“errorKlass”:“java.lang.String”,“errorCode”:“E4000”}]}]}]}

which means that Numerator is required field for an indicator. This was not the case in 2.21. This restriction was introduced in some version after 2.21. However, we modified the metadata for the indicator having this issue and proceeded further.

  1. Now we got the below issue.

{“httpStatus”:“Internal Server Error”,“httpStatusCode”:500,“status”:“ERROR”,“message”:“ERROR: duplicate key value violates unique constraint "dataelement_operand_unique_key"\n Detail: Key (dataelementid, categoryoptioncomboid)=(4497, 3576) already exists.”}

This means that you cannot have duplicate combination of (dataelementid, categoryoptioncomboid) in dataelementoperand table. This constraint was not present in 2.21 version.

When we see dataelementoperand table, there are lot of such duplicates. We would like to understand more around this constraint and why it was introduced.

Is there any note which has the details about the constraints being introduced in a DHIS version?

  1. Also this constraint (dataelement_operand_unique_key) is not present in the instance which was upgraded from 2.21 to 2.24. But it is present in the instance which is directly on 2.24. Any thoughts on this. Did the constraint not get created in upgraded instance because the data was not abiding to the constraint rules? What happens in a dhis upgrade if there is any constraint being introduced and existing data does not confirm to the constraint? Please suggest.

Thanks

Aamer.


With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

On Sat, Sep 24, 2016 at 5:54 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Aamer,

the problem here happens because the hibernate mapping between GroupSet and Group is one-to-many, ie. a Group can only be part of one GroupSet.

There is a reason for this, but it is not very good :wink: When we store favorites with group sets, we need to know which dimension (GroupSet) a dimension item (Group) is part of. If the Group is part of many GroupSets, we need another link object in between to represent which dimension it is part of. This can clearly be done and this is something we plan to fix, but it is a bit of work and it requires an upgrade routine to upgrade existing databases.

So I am pretty sure that is why you see these duplicates: You have groups which are part of multiple group sets. In these cases, Hibernate will create duplicate GroupSets as a result of the database integrity issue.

So short term solution is to change the metadata, long term we will fix this properly and allow for many-to-many between Group and GroupSet.

regards,

Lars

On Thu, Sep 15, 2016 at 7:28 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Lars knows what is wrong, he will provide a fix shortly :slight_smile:

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Thanks Morten. Let us know if any other details are required from our end in this regard.

Thanks

Aamer.

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen morten@dhis2.org wrote:

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Hi Lars/Morten,

This is reproducible on 2.25 as well. Any updates regarding this issue. Let us know if any further details are required from our end.

Thanks

Aamer.

···

On Thu, Oct 6, 2016 at 4:12 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi Lars,

Thanks for the fix (removal of constraint). Yes we are trying to import the metadata into a new 2.24 db. The metadata is exported from an instance which is upgraded from 2.21 to 2.24.

  1. After your fix, there is no constraint violation error and we are able to continue further. But this time we got the below issue in the later part of the import.

  • INFO 2016-10-06 11:46:09,906 (admin) Creating 1003 object(s) of type ReportTable (DefaultObjectBundleService.java [qtp923727115-15])

org.hibernate.TransientObjectException: object references an unsaved transient instance - save the transient instance before flushing: org.hisp.dhis.dataelement.DataElement


The complete stack trace is also attached. Any suggestions please.

Thanks

Aamer.

On Tue, Oct 4, 2016 at 1:23 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Aamer,

On Thu, Sep 29, 2016 at 1:31 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi,

For continuation of testing, we changed the metadata to ensure that a Group can be part of only one GroupSet.

  1. We got the below issue.

{“status”:“ERROR”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“typeReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“objectReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“index”:277,“uid”:“oMbElS3lhOO”,“errorReports”:[{“message”:“Missing required property numerator.”,“mainKlass”:“org.hisp.dhis.indicator.Indicator”,“errorKlass”:“java.lang.String”,“errorCode”:“E4000”}]}]}]}

which means that Numerator is required field for an indicator. This was not the case in 2.21. This restriction was introduced in some version after 2.21. However, we modified the metadata for the indicator having this issue and proceeded further.

Right, we did make indicator numerator and denominator not-null in the schema. You can say that this is something we should have done from the very start. In 2.24 we tightened up the schema, which is very helpful to avoid problems in the source code, but could require some modifications in existing solutions.

  1. Now we got the below issue.

{“httpStatus”:“Internal Server Error”,“httpStatusCode”:500,“status”:“ERROR”,“message”:“ERROR: duplicate key value violates unique constraint "dataelement_operand_unique_key"\n Detail: Key (dataelementid, categoryoptioncomboid)=(4497, 3576) already exists.”}

This means that you cannot have duplicate combination of (dataelementid, categoryoptioncomboid) in dataelementoperand table. This constraint was not present in 2.21 version.

When we see dataelementoperand table, there are lot of such duplicates. We would like to understand more around this constraint and why it was introduced.

Is there any note which has the details about the constraints being introduced in a DHIS version?

This is interesting because i) it has been like this for many years and ii) I agree this constraint should not be there. Not sure why this surfaced now - likely because you have started a new db. I have removed this constraint in master and 2.24 now.

best regards,

Lars

  1. Also this constraint (dataelement_operand_unique_key) is not present in the instance which was upgraded from 2.21 to 2.24. But it is present in the instance which is directly on 2.24. Any thoughts on this. Did the constraint not get created in upgraded instance because the data was not abiding to the constraint rules? What happens in a dhis upgrade if there is any constraint being introduced and existing data does not confirm to the constraint? Please suggest.

Thanks

Aamer.


Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

On Sat, Sep 24, 2016 at 5:54 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Aamer,

the problem here happens because the hibernate mapping between GroupSet and Group is one-to-many, ie. a Group can only be part of one GroupSet.

There is a reason for this, but it is not very good :wink: When we store favorites with group sets, we need to know which dimension (GroupSet) a dimension item (Group) is part of. If the Group is part of many GroupSets, we need another link object in between to represent which dimension it is part of. This can clearly be done and this is something we plan to fix, but it is a bit of work and it requires an upgrade routine to upgrade existing databases.

So I am pretty sure that is why you see these duplicates: You have groups which are part of multiple group sets. In these cases, Hibernate will create duplicate GroupSets as a result of the database integrity issue.

So short term solution is to change the metadata, long term we will fix this properly and allow for many-to-many between Group and GroupSet.

regards,

Lars

On Thu, Sep 15, 2016 at 7:28 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Lars knows what is wrong, he will provide a fix shortly :slight_smile:

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Thanks Morten. Let us know if any other details are required from our end in this regard.

Thanks

Aamer.

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen morten@dhis2.org wrote:

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Sure that makes sense.

But I am more curious to understand the following:

My existing Database (the data inside) does not follow the new constraints that I am trying to add. Say I decide to make “shortName” unique by altering the table to add this uniqueness constraint. But if the existing short names column has repetitions then how does this get handled?

This is just one example but we have encountered so many such use cases. The only part that I am unable to understand is “the new version war gets deployed successfully” giving a false notion of version upgrade.

Thoughts?

Regards

Vanya

···

On Thu, Oct 6, 2016 at 6:34 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Vanya,

yes agree, now we typically embed the upgrade statements within DHIS 2 itself, as we have learnt that people often forget to run upgrade statements which causes all kinds of problems.

regards,

Lars

On Tue, Oct 4, 2016 at 11:31 AM, Vanya Seth vanyas@thoughtworks.com wrote:

Hi Lars

Thanks for the response.

But have we only introduced not null constraints over different version updates?

But am still not sure how the constraints get applied if the data does not abide by the rules.

What looks like a reasonable process to me is :

  • Update your database data to abide by the new constraints.
  • Deploy the new war.

Thought?

Regards

Vanya


Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

On Tue, Oct 4, 2016 at 1:27 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Vanya,

sure. We definitely try to avoid this. See previous email in thread. The operand constraint has been removed. We will include not-null constraints in the upgrade notes.

For 2.25 we will set the “created” and “lastupdated” columns of tables of “identifiable objects” to not-null.

best regards,

Lars


With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

On Tue, Oct 4, 2016 at 9:39 AM, Vanya Seth vanyas@thoughtworks.com wrote:

Hi Morten and Lars

There are certain key points highlighted in the above email. Whenever a new DHIS2 version comes out there are new integrity constraints added but thats not part of the release or the upgrade notes. So, whats the right way to get information about these constraints?

When we deploy the new war we assume that the TableAlteror will take care of modifying the schema and also take care of adding the new constraints. But in case the existing data does not abide by these constraints how do we apply them?

So, principally after a successful redeploy of the war of a higher version can we say that the DHIS instance has been upgraded in its true sense.

As of now it seems the war does get updated but the underlying database does not.

Needless to say this breaks the export /import of metadata downstream.

Any insights about this would be appreciated.

Regards

Vanya


Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

On Thu, Sep 29, 2016 at 5:01 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Hi,

For continuation of testing, we changed the metadata to ensure that a Group can be part of only one GroupSet.

  1. We got the below issue.

{“status”:“ERROR”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“typeReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“stats”:{“total”:1,“created”:0,“updated”:0,“deleted”:0,“ignored”:1},“objectReports”:[{“klass”:“org.hisp.dhis.indicator.Indicator”,“index”:277,“uid”:“oMbElS3lhOO”,“errorReports”:[{“message”:“Missing required property numerator.”,“mainKlass”:“org.hisp.dhis.indicator.Indicator”,“errorKlass”:“java.lang.String”,“errorCode”:“E4000”}]}]}]}

which means that Numerator is required field for an indicator. This was not the case in 2.21. This restriction was introduced in some version after 2.21. However, we modified the metadata for the indicator having this issue and proceeded further.

  1. Now we got the below issue.

{“httpStatus”:“Internal Server Error”,“httpStatusCode”:500,“status”:“ERROR”,“message”:“ERROR: duplicate key value violates unique constraint "dataelement_operand_unique_key"\n Detail: Key (dataelementid, categoryoptioncomboid)=(4497, 3576) already exists.”}

This means that you cannot have duplicate combination of (dataelementid, categoryoptioncomboid) in dataelementoperand table. This constraint was not present in 2.21 version.

When we see dataelementoperand table, there are lot of such duplicates. We would like to understand more around this constraint and why it was introduced.

Is there any note which has the details about the constraints being introduced in a DHIS version?

  1. Also this constraint (dataelement_operand_unique_key) is not present in the instance which was upgraded from 2.21 to 2.24. But it is present in the instance which is directly on 2.24. Any thoughts on this. Did the constraint not get created in upgraded instance because the data was not abiding to the constraint rules? What happens in a dhis upgrade if there is any constraint being introduced and existing data does not confirm to the constraint? Please suggest.

Thanks

Aamer.


With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!

On Sat, Sep 24, 2016 at 5:54 PM, Lars Helge Øverland lars@dhis2.org wrote:

Hi Aamer,

the problem here happens because the hibernate mapping between GroupSet and Group is one-to-many, ie. a Group can only be part of one GroupSet.

There is a reason for this, but it is not very good :wink: When we store favorites with group sets, we need to know which dimension (GroupSet) a dimension item (Group) is part of. If the Group is part of many GroupSets, we need another link object in between to represent which dimension it is part of. This can clearly be done and this is something we plan to fix, but it is a bit of work and it requires an upgrade routine to upgrade existing databases.

So I am pretty sure that is why you see these duplicates: You have groups which are part of multiple group sets. In these cases, Hibernate will create duplicate GroupSets as a result of the database integrity issue.

So short term solution is to change the metadata, long term we will fix this properly and allow for many-to-many between Group and GroupSet.

regards,

Lars

On Thu, Sep 15, 2016 at 7:28 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Aamer

Lars knows what is wrong, he will provide a fix shortly :slight_smile:

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed aamerm@thoughtworks.com wrote:

Thanks Morten. Let us know if any other details are required from our end in this regard.

Thanks

Aamer.

On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen morten@dhis2.org wrote:

That’s interesting.

@Lars any suggestions why this happens? I guess there is something wrong with our mapping. Will have a look tomorrow.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia vgarciabnz@gmail.com wrote:

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, “>= 15” is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.

We don’t know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen morten@dhis2.org wrote:

Hi again

Would it be possible to share your database with me? I haven’t seen this

issues anywhere else.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think

there is nothing suspicious out here.

Regards,

Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen morten@dhis2.org

wrote:

I’m not sure what is causing it, but it shouldn’t be allowed to begin

with. Is there any constraint on the UID column in your database?

[image: Inline image 1]

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

Hope you are doing good. Let us know your thoughts on the above issue

and please let us know if you need any information that you might require.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <

sultanm@thoughtworks.com> wrote:

Hi Morten,

We have pulled latest code from 2.24 and tried import / export again.

We have some interesting observations

DB snapshot:

There are no duplicate items in DB. We double checked.

Entity endpoint snapshot:

​Interestingly we found few duplicate entities when we try to fetch

this data through the API. In Maintenance app, we are not able to see these

duplicate entities. Does it also explain why we get repeating entries (with

same UID’s) in the full export JSON as well? Let us know on how to proceed

and if you need more input.

Thanks for all the help in advance.

Regards,

Sultan Ahamar.

On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should now have been fixed in master and 224. It will now do a

global UID check first, and report back any duplicates, if it finds a

duplicate it will remove them from the import and report back an

ErrorReport (as it would do with other validation issues). So if it finds

a duplicate, you will need to set atomic mode to NONE to have it import and

ignore the duplicates.

(btw, there was several duplicates in your file, not sure how that

happened… maybe something needs to be cleaned in your source instance)

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Hi Vanya

This should have been caught by the importer… but there are several

duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category

option groups, same with oqeVQ71LCgY…

I will look into making the validation process more robust… but at

least you should know that this file have issues

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen morten@dhis2.org

wrote:

Ok, thanks Vanya, I’m looking into it now

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth vanyas@thoughtworks.com

wrote:

Hi Morten

Thanks for the response.

We ran into another issue while trying to do a full export and

import.

As per the previous conversations we took care of all data

migrations (as per the new constraints on the DB- pretty much caught by the

Importer during the validation phase itself).

Having done that we run into this issue:

The ‘categoryoptiongroup’ import is failing with constraint

violation for the UID. We checked the data and there is no repeating UID in

the database.

The error text is attached for your reference, as well the payload

used for the import.

Thanks for all the help in advance.

Regards

Vanya

On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Thanks Aamer, I’m looking at a similar bug right now… seems some

objects don’t get their deps properly attached…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

Please find the payload for below request

Let me know if the complete payload or any other details are

required from my end.

Thanks

Aamer.

On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Could you please share the payload of the object where this

happen?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed <

aamerm@thoughtworks.com> wrote:

Hi Morten,

The Importer was run with atomicMode=ALL. Though the payload is

huge, only few errors were shown in ImportSummary. As per errors in

previous ImportSummary, I have done below.

  1. ensured the ‘short name’ field is unique for each of the

‘data elements’ and ‘data element group’ as well

  1. UID of admin, Super User, tracked entity was changed in

payload to be same to where it is getting imported.

I have manually resolved them and ran the importer again with

atomicMode=ALL.

I got exception as attached in trace file

‘output_besteffort_trace.txt’

Would these kind of issues be known only after the import has

failed? and we need to resolve them one-by-one and as-and-when the importer

throws them? Is there any note which has the details about the constraints

being introduced in a new DHIS version?

Alternatively, If we export a filtered set of metadata using ’

/api/24/metadata?filter=lastUpdated:gt:2016-05’ and run the

importer with atomicMode=NONE

*curl -H “Content-Type: application/json” -X GET -u

username:password

"http://localhost:8888/api/24/metadata?filter=created:gt:2016-07

<http://localhost:8888/api/24/metadata?filter=created:gt:2016-07>" >

metadata.224.newapi.created.json*

Import the payload in a fresh instance for import using

*curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.created.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort.txt*

If the same payload is run with atomicMode=ALL, I am getting

‘Invalid references’ errors which is acceptable. But if it is run with

atomicMode=NONE, it is throwing the error for which the complete stack

trace is attached earlier. filename: ’

output_created_besteffort_trace.txt’>>>>>>>>>>>> *curl -H “Content-Type: application/json” -X POST --data

@metadata.224.newapi.json -u admin:district

http://localhost:8080/api/24/metadata?atomicMode=NONE

<http://localhost:8080/api/24/metadata?atomicMode=NONE>” >

output_created_besteffort_trace.txt*

Getting below exception in trace. Complete trace also attached.


  • INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) of

type ReportTable (DefaultObjectBundleService.java

[qtp289378424-12]) org.hibernate.TransientObjectException:

object references an unsaved transient instance - save the transient

instance before flushing: org.hisp.dhis.indicator.Indicator


Any suggestions please.

Thanks

Aamer.

On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen <

morten@dhis2.org> wrote:

Hi Aamer

Probably what is happening is that the new database has some

constraints that the old one doesn’t. We have had some issues with

hibernate in the past, and for certain databases that means that many of

the constraints was not applied…

If you look at the messages, you will see that e.g shortName

must be unique for data elements… the list of errors is not big though,

probably you can go through

[truncated for moderation]


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

With Regards
ThoughtWorks Technologies

Hyderabad

–Stay Hungry Stay Foolish!!