Getting more info about why data values are ignored during import

Support - Assistance technique
1

Hi,

I am sending bulks of data to DHIS2 using the API specifying value, dataElement, catOptCombo, orgUnit and Period (attrOptionCombo is always empty). An example of the json payload can be found below.

{

“dataValues”: [

{

“dataElement”: “f3650e9e3b6”,

“period”: “2013W01”,

“orgUnit”: “098b208aabf”,

“categoryOptionCombo”: “412b32f973f”,

“attrOptionCombo”: “”,

“value”: “5”

},

{

“dataElement”: “26d0e2dd673”,

“period”: “2013W01”,

“orgUnit”: “098b208aabf”,

“categoryOptionCombo”: “412b32f973f”,

“attrOptionCombo”: “”,

“value”: “4”

},

{

“dataElement”: “e638c648d6b”,

“period”: “2013W01”,

“orgUnit”: “098b208aabf”,

“categoryOptionCombo”: “412b32f973f”,

“attrOptionCombo”: “”,

“value”: “10”

},

{

“dataElement”: “f1ee1df0b4f”,

“period”: “2013W01”,

“orgUnit”: “098b208aabf”,

“categoryOptionCombo”: “412b32f973f”,

“attrOptionCombo”: “”,

“value”: “3”

},

{

“dataElement”: “94448a0b819”,

“period”: “2013W01”,

“orgUnit”: “098b208aabf”,

“categoryOptionCombo”: “412b32f973f”,

“attrOptionCombo”: “”,

“value”: “3”

}

]

}

So far I imported 100000 records in DHIS2 and got 818 ignored (response below). I have queries to check and validate that all references used in my data values exist and can’t find any missing reference. I also made sure there are no zero values… So I can’t figure out the reason why those 818 records are ignored. In there any way to get a response from DHIS2 with more info about ignored records? Thanks

{

“responseType”: “ImportSummary”,

“status”: “SUCCESS”,

“importOptions”: {

“idSchemes”: {},

“dryRun”: false,

“async”: false,

“importStrategy”: “CREATE_AND_UPDATE”,

“mergeMode”: “REPLACE”,

“reportMode”: “FULL”,

“skipExistingCheck”: false,

“sharing”: false,

“skipNotifications”: false,

“datasetAllowsPeriods”: false,

“strictPeriods”: false,

“strictCategoryOptionCombos”: false,

“strictAttributeOptionCombos”: false,

“strictOrganisationUnits”: false,

“requireCategoryOptionCombo”: false,

“requireAttributeOptionCombo”: false

},

“description”: “Import process completed successfully”,

“importCount”: {

“imported”: 99182,

“updated”: 0,

"ignored": 818,

“deleted”: 0

},

“dataSetComplete”: “false”

}

Best regards / Cordialement / Vriendelijke groeten / Atentamente,

1470312630639_PastedImage

2

I remember having similar issues at some point… But often for me it has been related to attributeoptioncomboes – which you are not using…

Which DHIS2 version are you running?

Have you tried multiple imports with the same file – do you get the same results each time?

Another tip for getting more information “back” to try on a database with no data values and then compare the imported file of datavalues against the export post import to see if there is a pattern to which ones doesn’t get imported? Also try the latest build of DHIS if you are using an old build.

Best of luck,

Trøyel

Trøyel Nerbø

Data Reporting and Analysis Adviser

Strategic Planning and Development, Head office

Phone: + 47 99239817 | Skype ID nrc_troyel

Norwegian Refugee Council

www.nrc.no| Facebook | Twitter

cid:image001.png@01D348CB.BE410710

image

···

From: Dhis2-devs [mailto:dhis2-devs-bounces+troyel.nerbo=nrc.no@lists.launchpad.net] On Behalf Of Manuel Silva Gallego

Sent: tirsdag 9. januar 2018 11.09

To: dhis2-devs@lists.launchpad.net

Subject: [Dhis2-devs] Getting more info about why data values are ignored during import

Hi,

I am sending bulks of data to DHIS2 using the API specifying value, dataElement, catOptCombo, orgUnit and Period (attrOptionCombo is always empty). An example of the json payload can be found below.

{

“dataValues”: [

{

  "dataElement": "f3650e9e3b6",

  "period": "2013W01",

  "orgUnit": "098b208aabf",

  "categoryOptionCombo": "412b32f973f",

  "attrOptionCombo": "",

  "value": "5"

},

{

  "dataElement": "26d0e2dd673",

  "period": "2013W01",

  "orgUnit": "098b208aabf",

  "categoryOptionCombo": "412b32f973f",

  "attrOptionCombo": "",

  "value": "4"

},

{

  "dataElement": "e638c648d6b",

  "period": "2013W01",

  "orgUnit": "098b208aabf",

  "categoryOptionCombo": "412b32f973f",

  "attrOptionCombo": "",

  "value": "10"

},

{

  "dataElement": "f1ee1df0b4f",

  "period": "2013W01",

  "orgUnit": "098b208aabf",

  "categoryOptionCombo": "412b32f973f",

  "attrOptionCombo": "",

  "value": "3"

},

{

  "dataElement": "94448a0b819",

  "period": "2013W01",

  "orgUnit": "098b208aabf",

  "categoryOptionCombo": "412b32f973f",

  "attrOptionCombo": "",

  "value": "3"

}

]

}

So far I imported 100000 records in DHIS2 and got 818 ignored (response below). I have queries to check and validate that all references used in my data values exist and can’t find any missing reference. I also made sure there are no zero values… So I can’t figure out the reason why those 818 records are ignored. In there any way to get a response from DHIS2 with more info about ignored records? Thanks

{

“responseType”: “ImportSummary”,

“status”: “SUCCESS”,

“importOptions”: {

"idSchemes": {},

"dryRun": false,

"async": false,

"importStrategy": "CREATE_AND_UPDATE",

"mergeMode": "REPLACE",

"reportMode": "FULL",

"skipExistingCheck": false,

"sharing": false,

"skipNotifications": false,

"datasetAllowsPeriods": false,

"strictPeriods": false,

"strictCategoryOptionCombos": false,

"strictAttributeOptionCombos": false,

"strictOrganisationUnits": false,

"requireCategoryOptionCombo": false,

"requireAttributeOptionCombo": false

},

“description”: “Import process completed successfully”,

“importCount”: {

"imported": 99182,

"updated": 0,

"**ignored": 818**,

"deleted": 0

},

“dataSetComplete”: “false”

}

Best regards / Cordialement / Vriendelijke groeten / Atentamente,

3

Hi,

With the help of Simon Mugisa, we realized that importing records in CSV and through the GUI we get a full import report specifying the reason for each conflict. Uids for the missing references are provided, so it is easy to troubleshoot. It does the trick for now, although I would have preferred to work this out with the API.

image

image

image

···

Best regards / Cordialement / Vriendelijke groeten / Atentamente,

From: Trøyel Nerbø troyel.nerbo@nrc.no

Sent: 10 January 2018 15:23:03

To: Manuel Silva Gallego; dhis2-devs@lists.launchpad.net

Subject: RE: [Dhis2-devs] Getting more info about why data values are ignored during import

I remember having similar issues at some point… But often for me it has been related to attributeoptioncomboes – which you are not using…

Which DHIS2 version are you running?

Have you tried multiple imports with the same file – do you get the same results each time?

Another tip for getting more information “back” to try on a database with no data values and then compare the imported file of datavalues against the export post import to see if there is a pattern to which ones doesn’t get imported? Also try the latest build of DHIS if you are using an old build.

Best of luck,

Trøyel

Trøyel Nerbø

Data Reporting and Analysis Adviser

Strategic Planning and Development, Head office

Phone: + 47 99239817 | Skype ID nrc_troyel

Norwegian Refugee Council

www.nrc.no| Facebook | Twitter

From: Dhis2-devs [mailto:dhis2-devs-bounces+troyel.nerbo=nrc.no@lists.launchpad.net] On Behalf Of Manuel Silva Gallego

Sent: tirsdag 9. januar 2018 11.09

To: dhis2-devs@lists.launchpad.net

Subject: [Dhis2-devs] Getting more info about why data values are ignored during import

Hi,

I am sending bulks of data to DHIS2 using the API specifying value, dataElement, catOptCombo, orgUnit and Period (attrOptionCombo is always empty). An example of the json payload can be found below.

{

“dataValues”: [

{

  "dataElement": "f3650e9e3b6",

  "period": "2013W01",

  "orgUnit": "098b208aabf",

  "categoryOptionCombo": "412b32f973f",

  "attrOptionCombo": "",

  "value": "5"

},

{

  "dataElement": "26d0e2dd673",

  "period": "2013W01",

  "orgUnit": "098b208aabf",

  "categoryOptionCombo": "412b32f973f",

  "attrOptionCombo": "",

  "value": "4"

},

{

  "dataElement": "e638c648d6b",

  "period": "2013W01",

  "orgUnit": "098b208aabf",

  "categoryOptionCombo": "412b32f973f",

  "attrOptionCombo": "",

  "value": "10"

},

{

  "dataElement": "f1ee1df0b4f",

  "period": "2013W01",

  "orgUnit": "098b208aabf",

  "categoryOptionCombo": "412b32f973f",

  "attrOptionCombo": "",

  "value": "3"

},

{

  "dataElement": "94448a0b819",

  "period": "2013W01",

  "orgUnit": "098b208aabf",

  "categoryOptionCombo": "412b32f973f",

  "attrOptionCombo": "",

  "value": "3"

}

]

}

So far I imported 100000 records in DHIS2 and got 818 ignored (response below). I have queries to check and validate that all references used in my data values exist and can’t find any missing reference. I also made sure there are no zero values… So I can’t figure out the reason why those 818 records are ignored. In there any way to get a response from DHIS2 with more info about ignored records? Thanks

{

“responseType”: “ImportSummary”,

“status”: “SUCCESS”,

“importOptions”: {

"idSchemes": {},

"dryRun": false,

"async": false,

"importStrategy": "CREATE_AND_UPDATE",

"mergeMode": "REPLACE",

"reportMode": "FULL",

"skipExistingCheck": false,

"sharing": false,

"skipNotifications": false,

"datasetAllowsPeriods": false,

"strictPeriods": false,

"strictCategoryOptionCombos": false,

"strictAttributeOptionCombos": false,

"strictOrganisationUnits": false,

"requireCategoryOptionCombo": false,

"requireAttributeOptionCombo": false

},

“description”: “Import process completed successfully”,

“importCount”: {

"imported": 99182,

"updated": 0,

"**ignored": 818**,

"deleted": 0

},

“dataSetComplete”: “false”

}

Best regards / Cordialement / Vriendelijke groeten / Atentamente,

4

I have the same problem with my files. some of them present the reasons of error like these one

image

image

But most of them show this kind of result
image

I would like to know how to do.

5

this is not available in 2.35, Any one know how to bring those results in DHIS2 2.35