Poor performance with bulk upload attributeOptionCombo, and dataValueSets endpoint

Support - Assistance technique
1

Hi,

I am using the dataValueSets endpoint to sync data between an external system and some DHIS2 aggregate datasets.

The datasets in question have attribute combinations so that different partners can enter their data.

Up until now I have used the bulk update method of updating these datasets via the dataValueSets api endpoint. This has worked really well, but after adding the attributeOptionCombo field to the object the performance has dropped significantly, to the point of timing out on some of the larger updates.

The data I am sending is in this format:

{
  period: "2021",
  attributeOptionCombo: "partnerCode",
  dataValues: [
    {
      dataSet: "someDataSetCode",
      dataElement: "someDataElementCode",
      orgUnit: "someOrgUnitCode",
      value: "123"
    },
    ...*more values*
}

I am sending it with the following query parameters instructing the api to match on codes rather than IDs:

dataElementIdScheme=code&categoryOptionComboIdScheme=code&orgUnitIdScheme=code

The performance is really, really slow. Anyone run into this issue? Any advice on speeding this process up. Currently I am just syncing one IU and period at a time, which is going to involve a really, really large number of POSTs to the DHIS2 api and is less than elegant.

I could also try using the async query parameter, but I am concerned that the database itself is going to get overloaded if I do that.

Any advice would be greatly appreciated.

2

@HaydnJ

Thank you for your post! I might be able to help you by asking some questions. :smiley:

Please what version of dhis2 are you using? When you reach the point of “timing out” are there any errors that show (in Catalina.out log)?

Does clearing the server cache help with improving the performance (Data Administration app → Maintenance → Clear application cache)?

If possible please share how he size of the data being sent and the time that it takes. Additionally, it might be interesting to test to use the same post request using Postman directly rather than the other external system, is it the same performance?

Thanks!