XML import bug

Hi Devs,

The problem I reported yesterday has actually turned out to be what I think is a new bug in 2.25, which is stopping XML imports for data elements.

No matter what I do, **any attempt at an XML import of data elements seems to throw the error “**DataElementCategoryCombo Missing required property categoryCombo , even if a valid CategoryCombo is supplied. The same XML import works fine in 2.21.

To reproduce on play.dhis2.org:

· Using the import/export module, do an XML export of data elements only.

· Delete all data elements from the XML except DE_1148614 (which uses the default CategoryCombo), and change the user of that data element to .

· To circumvent a couple of other export/import bugs:

  1. delete the three ‘<’ characters (which have been exported as plaintext instead of <)

  2. delete the tags (I have no idea why these tags are failing)

· Import this XML file back into play.dhis2.org.

The import fails, and the import summary has the error message “ DataElementCategoryCombo Missing required property categoryCombo” – despite the fact that the data element has the default categoryCombo id.

I’ve attached the XML I extracted from play.dhis2.org to speed up testing. I’d be very grateful if you could look into this urgently, as I can’t currently do XML imports into my 2.25 instance.

Many thanks,

Sam.

XML import bug (single data element from play.dhis2.org).xml (1.2 KB)

···

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Sam Johnson samuel.johnson@qebo.co.uk

Date: Monday, 17 October 2016 at 04:54

To: DHIS2 Developers dhis2-devs@lists.launchpad.net

Cc: Rodolfo Melia rodolfo.melia@knowtechture.com

Subject: [Dhis2-devs] FW: Null pointer exception calling …/api/me/programs

Hi all,

I’m trying to produce a generic EventCapture XML config that can be imported into any DHIS2 instance, which requires the data elements and program to use the default CategoryCombo of the target instance.

My understanding from the documentation (and the email thread below) is that if the CategoryCombo for a data element is null, then during metadata import the data element will automatically be assigned the local ‘default’ CategoryCombo; the problem is that I can’t figure out how to represent this ‘null’ CategoryCombo in the XML. I’ve tried:

· simply removing all tags from the data elements in the XML;

· using ;

· using

· using .

In 2.25, all of these throw the error “DataElementCategoryCombo Missing required property categoryCombo”… How can I get these data elements to take on the local ‘default’ CategoryCombo as the XML is loaded?

Many thanks for any help anyone is able to give!

Regards, Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Rodolfo Melia rmelia@knowming.com

Date: Monday, 9 May 2016 at 09:37

To: Lars Helge Øverland lars@dhis2.org

Cc: Adebusoye Anifalaje busoye@baosystems.com, DHIS2 Developers dhis2-devs@lists.launchpad.net, JM Alcantara jmalcantara1@gmail.com

Subject: Re: [Dhis2-devs] Null pointer exception calling …/api/me/programs

Noted - we look forward to get a resolution to the setting of CatCombo to ‘default’ in 2.22 soon - we import hundred of new data elements to our servers every week, and many of the templates out there have that column null. This was not a problem in 2.20.

R

On 9 May 2016 at 09:24, Lars Helge Øverland lars@dhis2.org wrote:

On Mon, May 9, 2016 at 10:23 AM, Rodolfo Melia rmelia@knowming.com wrote:

Hi - I just want to confirm that I’m getting my facts right.

The fix for ensuring that a imported DEs with CatCombo=null are set to default is only available in 2.23, latest build.

Any 2.23 build, but yes.

The only workaround available right now for 2.22 , that avoids re-starting tomcat (which makes the necessary correction at start-up) is to make sure that when importing DEs, we set the CatCombo to default in the XML/ CSV.

Correct.

Lars

R

Rodolfo Meliá

*Principal | *rmelia@knowming.com

Skype: rod.melia | +44 777 576 4090 | +1 708 872 7636

www.knowming.com

On 9 May 2016 at 08:19, Morten Olav Hansen morten@dhis2.org wrote:

And yes, we don’t allow null in the categoryCombo field from 2.23 and upwards

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, May 9, 2016 at 2:15 PM, Morten Olav Hansen morten@dhis2.org wrote:

On Mon, May 9, 2016 at 1:47 PM, Lars Helge Øverland lars@dhis2.org wrote:

Morten do you think you could comment?

Hi, these kinds of issues are fixed in our new importer (available in 2.23), for now I would include the default category combo. When you say Import, are you creating this payload yourself, or is it coming from another system?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org

Hi Sam,

Can you try to use the full object name “dataElementCategoryCombo” instead of “categoryCombo” and try again. It works on my local.

Need more investigation to understand why the short name does not work though.

Regards.

···

On Mon, Oct 17, 2016 at 9:55 PM, Sam Johnson samuel.johnson@qebo.co.uk wrote:

Hi Devs,

The problem I reported yesterday has actually turned out to be what I think is a new bug in 2.25, which is stopping XML imports for data elements.

No matter what I do, **any attempt at an XML import of data elements seems to throw the error “**DataElementCategoryCombo Missing required property categoryCombo , even if a valid CategoryCombo is supplied. The same XML import works fine in 2.21.

To reproduce on play.dhis2.org:

· Using the import/export module, do an XML export of data elements only.

· Delete all data elements from the XML except DE_1148614 (which uses the default CategoryCombo), and change the user of that data element to .

· To circumvent a couple of other export/import bugs:

  1. delete the three ‘<’ characters (which have been exported as plaintext instead of <)

  2. delete the tags (I have no idea why these tags are failing)

· Import this XML file back into play.dhis2.org.

The import fails, and the import summary has the error message “ DataElementCategoryCombo Missing required property categoryCombo” – despite the fact that the data element has the default categoryCombo id.

I’ve attached the XML I extracted from play.dhis2.org to speed up testing. I’d be very grateful if you could look into this urgently, as I can’t currently do XML imports into my 2.25 instance.

Many thanks,

Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Sam Johnson samuel.johnson@qebo.co.uk

Date: Monday, 17 October 2016 at 04:54

To: DHIS2 Developers dhis2-devs@lists.launchpad.net

Cc: Rodolfo Melia rodolfo.melia@knowtechture.com

Subject: [Dhis2-devs] FW: Null pointer exception calling …/api/me/programs

Hi all,

I’m trying to produce a generic EventCapture XML config that can be imported into any DHIS2 instance, which requires the data elements and program to use the default CategoryCombo of the target instance.

My understanding from the documentation (and the email thread below) is that if the CategoryCombo for a data element is null, then during metadata import the data element will automatically be assigned the local ‘default’ CategoryCombo; the problem is that I can’t figure out how to represent this ‘null’ CategoryCombo in the XML. I’ve tried:

· simply removing all tags from the data elements in the XML;

· using ;

· using

· using .

In 2.25, all of these throw the error “DataElementCategoryCombo Missing required property categoryCombo”… How can I get these data elements to take on the local ‘default’ CategoryCombo as the XML is loaded?

Many thanks for any help anyone is able to give!

Regards, Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Rodolfo Melia rmelia@knowming.com

Date: Monday, 9 May 2016 at 09:37

To: Lars Helge Øverland lars@dhis2.org

Cc: Adebusoye Anifalaje busoye@baosystems.com, DHIS2 Developers dhis2-devs@lists.launchpad.net, JM Alcantara jmalcantara1@gmail.com

Subject: Re: [Dhis2-devs] Null pointer exception calling …/api/me/programs

Noted - we look forward to get a resolution to the setting of CatCombo to ‘default’ in 2.22 soon - we import hundred of new data elements to our servers every week, and many of the templates out there have that column null. This was not a problem in 2.20.

R

On 9 May 2016 at 09:24, Lars Helge Øverland lars@dhis2.org wrote:

On Mon, May 9, 2016 at 10:23 AM, Rodolfo Melia rmelia@knowming.com wrote:

Hi - I just want to confirm that I’m getting my facts right.

The fix for ensuring that a imported DEs with CatCombo=null are set to default is only available in 2.23, latest build.

Any 2.23 build, but yes.

The only workaround available right now for 2.22 , that avoids re-starting tomcat (which makes the necessary correction at start-up) is to make sure that when importing DEs, we set the CatCombo to default in the XML/ CSV.

Correct.

Lars

R

Rodolfo Meliá

*Principal | *rmelia@knowming.com

Skype: rod.melia | +44 777 576 4090 | +1 708 872 7636

www.knowming.com

On 9 May 2016 at 08:19, Morten Olav Hansen morten@dhis2.org wrote:

And yes, we don’t allow null in the categoryCombo field from 2.23 and upwards

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, May 9, 2016 at 2:15 PM, Morten Olav Hansen morten@dhis2.org wrote:

On Mon, May 9, 2016 at 1:47 PM, Lars Helge Øverland lars@dhis2.org wrote:

Morten do you think you could comment?

Hi, these kinds of issues are fixed in our new importer (available in 2.23), for now I would include the default category combo. When you say Import, are you creating this payload yourself, or is it coming from another system?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Viet Nguyen

Software Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

Hi Sam,

I have applied a fix for this issue on 2.25

Please try again.

Regards,

···

On Tue, Oct 18, 2016 at 12:23 PM, Viet Nguyen viet@dhis2.org wrote:

Hi Sam,

Can you try to use the full object name “dataElementCategoryCombo” instead of “categoryCombo” and try again. It works on my local.

Need more investigation to understand why the short name does not work though.

Regards.

On Mon, Oct 17, 2016 at 9:55 PM, Sam Johnson samuel.johnson@qebo.co.uk wrote:

Hi Devs,

The problem I reported yesterday has actually turned out to be what I think is a new bug in 2.25, which is stopping XML imports for data elements.

No matter what I do, **any attempt at an XML import of data elements seems to throw the error “**DataElementCategoryCombo Missing required property categoryCombo , even if a valid CategoryCombo is supplied. The same XML import works fine in 2.21.

To reproduce on play.dhis2.org:

· Using the import/export module, do an XML export of data elements only.

· Delete all data elements from the XML except DE_1148614 (which uses the default CategoryCombo), and change the user of that data element to .

· To circumvent a couple of other export/import bugs:

  1. delete the three ‘<’ characters (which have been exported as plaintext instead of <)

  2. delete the tags (I have no idea why these tags are failing)

· Import this XML file back into play.dhis2.org.

The import fails, and the import summary has the error message “ DataElementCategoryCombo Missing required property categoryCombo” – despite the fact that the data element has the default categoryCombo id.

I’ve attached the XML I extracted from play.dhis2.org to speed up testing. I’d be very grateful if you could look into this urgently, as I can’t currently do XML imports into my 2.25 instance.

Many thanks,

Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Sam Johnson samuel.johnson@qebo.co.uk

Date: Monday, 17 October 2016 at 04:54

To: DHIS2 Developers dhis2-devs@lists.launchpad.net

Cc: Rodolfo Melia rodolfo.melia@knowtechture.com

Subject: [Dhis2-devs] FW: Null pointer exception calling …/api/me/programs

Hi all,

I’m trying to produce a generic EventCapture XML config that can be imported into any DHIS2 instance, which requires the data elements and program to use the default CategoryCombo of the target instance.

My understanding from the documentation (and the email thread below) is that if the CategoryCombo for a data element is null, then during metadata import the data element will automatically be assigned the local ‘default’ CategoryCombo; the problem is that I can’t figure out how to represent this ‘null’ CategoryCombo in the XML. I’ve tried:

· simply removing all tags from the data elements in the XML;

· using ;

· using

· using .

In 2.25, all of these throw the error “DataElementCategoryCombo Missing required property categoryCombo”… How can I get these data elements to take on the local ‘default’ CategoryCombo as the XML is loaded?

Many thanks for any help anyone is able to give!

Regards, Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Rodolfo Melia rmelia@knowming.com

Date: Monday, 9 May 2016 at 09:37

To: Lars Helge Øverland lars@dhis2.org

Cc: Adebusoye Anifalaje busoye@baosystems.com, DHIS2 Developers dhis2-devs@lists.launchpad.net, JM Alcantara jmalcantara1@gmail.com

Subject: Re: [Dhis2-devs] Null pointer exception calling …/api/me/programs

Noted - we look forward to get a resolution to the setting of CatCombo to ‘default’ in 2.22 soon - we import hundred of new data elements to our servers every week, and many of the templates out there have that column null. This was not a problem in 2.20.

R

On 9 May 2016 at 09:24, Lars Helge Øverland lars@dhis2.org wrote:

On Mon, May 9, 2016 at 10:23 AM, Rodolfo Melia rmelia@knowming.com wrote:

Hi - I just want to confirm that I’m getting my facts right.

The fix for ensuring that a imported DEs with CatCombo=null are set to default is only available in 2.23, latest build.

Any 2.23 build, but yes.

The only workaround available right now for 2.22 , that avoids re-starting tomcat (which makes the necessary correction at start-up) is to make sure that when importing DEs, we set the CatCombo to default in the XML/ CSV.

Correct.

Lars

R

Rodolfo Meliá

*Principal | *rmelia@knowming.com

Skype: rod.melia | +44 777 576 4090 | +1 708 872 7636

www.knowming.com

On 9 May 2016 at 08:19, Morten Olav Hansen morten@dhis2.org wrote:

And yes, we don’t allow null in the categoryCombo field from 2.23 and upwards

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, May 9, 2016 at 2:15 PM, Morten Olav Hansen morten@dhis2.org wrote:

On Mon, May 9, 2016 at 1:47 PM, Lars Helge Øverland lars@dhis2.org wrote:

Morten do you think you could comment?

Hi, these kinds of issues are fixed in our new importer (available in 2.23), for now I would include the default category combo. When you say Import, are you creating this payload yourself, or is it coming from another system?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Viet Nguyen

Software Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

Viet Nguyen

Software Developer, DHIS 2

University of Oslo

http://www.dhis2.org

Hi Viet,

This fix looks like it has worked – the whole program XML is now importing smoothly into my 2.25 instance. Many thanks for sorting this out so quickly! J

Cheers, Sam.

···

From: Viet Nguyen viet@dhis2.org

Date: Wednesday, 19 October 2016 at 05:16

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] XML import bug

Hi Sam,

I have applied a fix for this issue on 2.25

Please try again.

Regards,

On Tue, Oct 18, 2016 at 12:23 PM, Viet Nguyen viet@dhis2.org wrote:

Hi Sam,

Can you try to use the full object name “dataElementCategoryCombo” instead of “categoryCombo” and try again. It works on my local.

Need more investigation to understand why the short name does not work though.

Regards.

On Mon, Oct 17, 2016 at 9:55 PM, Sam Johnson samuel.johnson@qebo.co.uk wrote:

Hi Devs,

The problem I reported yesterday has actually turned out to be what I think is a new bug in 2.25, which is stopping XML imports for data elements.

No matter what I do, any attempt at an XML import of data elements seems to throw the error “DataElementCategoryCombo Missing required property categoryCombo, even if a valid CategoryCombo is supplied. The same XML import works fine in 2.21.

To reproduce on play.dhis2.org:

· Using the import/export module, do an XML export of data elements only.

· Delete all data elements from the XML except DE_1148614 (which uses the default CategoryCombo), and change the user of that data element to .

· To circumvent a couple of other export/import bugs:

  1. delete the three ‘<’ characters (which have been exported as plaintext instead of <)

  2. delete the tags (I have no idea why these tags are failing)

· Import this XML file back into play.dhis2.org.

The import fails, and the import summary has the error message “DataElementCategoryCombo Missing required property categoryCombo” – despite the fact that the data element has the default categoryCombo id.

I’ve attached the XML I extracted from play.dhis2.org to speed up testing. I’d be very grateful if you could look into this urgently, as I can’t currently do XML imports into my 2.25 instance.

Many thanks,

Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Sam Johnson samuel.johnson@qebo.co.uk

Date: Monday, 17 October 2016 at 04:54

To: DHIS2 Developers dhis2-devs@lists.launchpad.net

Cc: Rodolfo Melia rodolfo.melia@knowtechture.com

Subject: [Dhis2-devs] FW: Null pointer exception calling …/api/me/programs

Hi all,

I’m trying to produce a generic EventCapture XML config that can be imported into any DHIS2 instance, which requires the data elements and program to use the default CategoryCombo of the target instance.

My understanding from the documentation (and the email thread below) is that if the CategoryCombo for a data element is null, then during metadata import the data element will automatically be assigned the local ‘default’ CategoryCombo; the problem is that I can’t figure out how to represent this ‘null’ CategoryCombo in the XML. I’ve tried:

· simply removing all tags from the data elements in the XML;

· using ;

· using

· using .

In 2.25, all of these throw the error “DataElementCategoryCombo Missing required property categoryCombo”… How can I get these data elements to take on the local ‘default’ CategoryCombo as the XML is loaded?

Many thanks for any help anyone is able to give!

Regards, Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Rodolfo Melia rmelia@knowming.com

Date: Monday, 9 May 2016 at 09:37

To: Lars Helge Øverland lars@dhis2.org

Cc: Adebusoye Anifalaje busoye@baosystems.com, DHIS2 Developers dhis2-devs@lists.launchpad.net, JM Alcantara jmalcantara1@gmail.com

Subject: Re: [Dhis2-devs] Null pointer exception calling …/api/me/programs

Noted - we look forward to get a resolution to the setting of CatCombo to ‘default’ in 2.22 soon - we import hundred of new data elements to our servers every week, and many of the templates out there have that column null. This was not a problem in 2.20.

R

On 9 May 2016 at 09:24, Lars Helge Øverland lars@dhis2.org wrote:

On Mon, May 9, 2016 at 10:23 AM, Rodolfo Melia rmelia@knowming.com wrote:

Hi - I just want to confirm that I’m getting my facts right.

The fix for ensuring that a imported DEs with CatCombo=null are set to default is only available in 2.23, latest build.

Any 2.23 build, but yes.

The only workaround available right now for 2.22 , that avoids re-starting tomcat (which makes the necessary correction at start-up) is to make sure that when importing DEs, we set the CatCombo to default in the XML/ CSV.

Correct.

Lars

R

Rodolfo Meliá

*Principal | *rmelia@knowming.com

Skype: rod.melia | +44 777 576 4090 | +1 708 872 7636

www.knowming.com

On 9 May 2016 at 08:19, Morten Olav Hansen morten@dhis2.org wrote:

And yes, we don’t allow null in the categoryCombo field from 2.23 and upwards

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Mon, May 9, 2016 at 2:15 PM, Morten Olav Hansen morten@dhis2.org wrote:

On Mon, May 9, 2016 at 1:47 PM, Lars Helge Øverland lars@dhis2.org wrote:

Morten do you think you could comment?

Hi, these kinds of issues are fixed in our new importer (available in 2.23), for now I would include the default category combo. When you say Import, are you creating this payload yourself, or is it coming from another system?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Lars Helge Øverland

Lead developer, DHIS 2

University of Oslo

Skype: larshelgeoverland

lars@dhis2.org

http://www.dhis2.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Viet Nguyen

Software Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

Viet Nguyen

Software Developer, DHIS 2

University of Oslo

http://www.dhis2.org