Data Exchange app for Tracker to Aggregate - how to trigger without scheduler?

Support - Assistance technique
1

Hello,
I’m trying out the Data Exchange app for the first time. I was watching this video which clearly says you can tricker the data exchange ad hoc or with the scheduler. https://www.youtube.com/watch?v=ODEOeVSNj3M

However - I am unable to get it to trigger within the app.

I’ve done these steps:

  1. Created Program Indicators that are mapped to COCs/AOCs and aggregate data elements
  2. Created a new data exchange with those Program Indicators.

Is there a way to trigger the PIs to be evaluated in the actual Data Exchange app?

2

I’m actually having a number of challenges with this app.

  1. I can verify the scheduled job completed successfully a few mins ago via the API, but it still says “Source data was generated 15 hours ago.”
  2. I have it configured to use an attribute for the data element ID, but every time I click Submit data, it gives me an error because its trying to use the id of the program indicator as the DE ID.

Is there any additional documentation on how to configure this with PIs?

3

Just going to continue to add what I find here. Another potential bug is if you create a cron job for data exchange, then you delete one of the exchanges applied to that job, it doesn’t delete from the backend. Even if you open the job to edit it, ensure there is only one data exchange selected, and save, the deleted ID still persists.

Example: TIb9gAsVyZp was deleted, but even when I save it on the UI with only eB7xWM8NsHL, TIb9gAsVyZp persists (verified via API), and causes the job to fail.

“jobParameters”: {
“dataExchangeIds”: [
“eB7xWM8NsHL”,
“TIb9gAsVyZp”
]
},

1 Like
4

Hi @kstankevitz

First, thank you so much for your patience. As the Data Exchange app is still new and there’s still a lot to be learned, documented, experimented with and hopefully improved especially when community members share about their experiences.

I have more questions than answers. Regarding resources, have you gone through the docs: Data Exchange - DHIS2 Documentation Is there anything in the docs that is not clear?

You can create and manage the data exchange using the API as mentioned in the docs, but you can ‘submit data’ from the app. There is a button “submit data” in the app after you have configured the data exchange, are you saying that this button isn’t working when you click it?

I am not sure what you mean by evaluate? The Data Exchange app has a specific purpose: exchange data internally/externally into the aggregate model.

This is possible a cache issue. Would you repeat the same steps and then check using your browser’s Guest mode? If it still not updated maybe it requires Data Administration app → Maintenance → Clear application cache & reload apps.

Would you be able to provide more details of the error:

  1. Error shared by the app
  2. Error that appears in the Network tab in the DevTools (F12 → Network) - check response/request and if possible ‘payload’
  3. Error that appears in the Console (DevTools/F12 → Console). If there’s a backend issue, we might need the Catalina.out log (without authentication/sensitive info)

For further debugging, DHIS2 version of the instance you are using and the app version. if possible provide more details about the metadata configuration if you think it’s relevant. Does this happen to all the metadata or specific ones? If specific ones with a different configuration than the rest of the metadata that are showing this error.

To uncover this issue, we’d need to reproduce it in a demo instance to verify whether it’s a matter of configuration, clearing server cache or if it is a bug.

Please see if you would be able to provide more info that I requested above and we can check to see if how to resolve the issues you are experiencing.

Thank you!

5

Many thanks for the questions @kstankevitz (and @Gassim). I’m one of the developers of the app, so I can hopefully provide a few answers. As Gassim notes this app is quite new and we are actively trying to improve it, so we appreciate the feedback. Apologies for the delay in response as I was out on vacation last week.

Is there a way to trigger the PIs to be evaluated in the actual Data Exchange app?

Since program indicators should be evaluated on the fly, when you open the exchange before submitting, I believe you should see the calculated results in the preview mode before hitting submit. If you are not seeing the data, it might be a good idea to check the mappings. We did an internal training that has an overview of configuring the app and setting up mappings: DX Demo - Hoi An April 2024.mp4 - Google Drive

As Gassim notes, once you see the data, hitting “Submit data” should be sufficient to trigger from within the app. If you’re not able to click on “Submit data”, you might need to update the sharing on the exchange.

I can verify the scheduled job completed successfully a few mins ago via the API, but it still says “Source data was generated 15 hours ago.”

The statement "Source data was generated 15 hours ago." actually refers to when analytics tables were last generated, which would be relevant for aggregated data. We struggled with a way to convey this information within the app. We considered an alternative like “analytics tables were last generated 15 hours ago” but thought that required to much knowledge about the inner workings of DHIS2.

If you have any suggestions for how this language might be made clearer (or explained better in the documentation), please let us know.

I have it configured to use an attribute for the data element ID, but every time I click Submit data, it gives me an error because its trying to use the id of the program indicator as the DE ID.

In version 41 onwards, we have updated the data exchange service to provide a Data item ID scheme which applies to data elements, indicators, and program indicators. This is selectable in the latest version of the app:

image

The data element ID scheme applies specifically to data elements.

Another potential bug is if you create a cron job for data exchange, then you delete one of the exchanges applied to that job, it doesn’t delete from the backend.

I will bring this point up with the backend team. I am not sure if we intend to restrict removal of metadata objects if they are associated with cron jobs.

1 Like
6

Thank you so much @tzemp! :slight_smile:

The video is now on the DHIS2 YouTube channel (https://www.youtube.com/watch?v=zUaj2N11BSw).

1 Like
8

Hi Thomas,
Thank you for the response. Apologies, I hadn’t responded until now.

Can you just confirm a few things for me?

When you have a data exchange configured in the scheduler, what does it do? Does it both calculate the values of the PIs and save them in the aggregate data model, or does it just calculate them and store them in a temporary table, until you click “Submit data”?

The time listed is still confusing me. Here you can see an example. Analytics was run 43m ago. Why does Data Exchange say 5 hours?

image
image1913×458 91.2 KB

9

Hello @tzemp,
I also have an additional follow up question. Can you help me understand why so many things are showing up under ignored? In other parts of DHIS2 (import/export for example), ignored is always accompanied by an error message saying why the values were ignored. Is there a way to determine why it is ignoring so many values?

image

10

@Gassim, do you know if there is someone who could answer some of these questions for me?

Another question - I continually get a status of STOPPED rather than COMPLETED. It seems like its stopping after calculating the values for the PIs but before sending them to the aggregate dataset. How can I troubleshoot this?

image
image1067×1045 78.3 KB

11

Hi @kstankevitz : sorry for the slow response again. I’ve been wrapped up with other tasks.

When you have a data exchange configured in the scheduler, what does it do? Does it both calculate the values of the PIs and save them in the aggregate data model, or does it just calculate them and store them in a temporary table, until you click “Submit data”?

Program indicators should be calculated on the fly, so they will not be saved in the aggregate data model in your system. Only when you hit submit will the values be transmitted to the target system and saved to the data elements you have mapped to. I do not work with this area of our code base, so I do not know the specifics on where the values are temporarily stored on calculation; if you want more details, I can ask around to see if someone can clarify the technical side more.

The time listed is still confusing me. Here you can see an example. Analytics was run 43m ago. Why does Data Exchange say 5 hours?

Thanks for pointing this out. This is a bug in the data exchange app. The relative time calculation was not correcting for the time zone difference (e.g. if you are New York and your server is in UTC, we need to correct for the time zone difference before calculating the time difference). A new version of the app has been released (100.9.7) which fixes this issue and should give the correct relative time since when analytics was last run. The app is available on the App Hub and can be updated from the App Management app.

Can you help me understand why so many things are showing up under ignored? In other parts of DHIS2 (import/export for example), ignored is always accompanied by an error message saying why the values were ignored. Is there a way to determine why it is ignoring so many values?

I don’t know specifically why items are being ignored. This could be, for example, because the data element on the target system has zeroIsSignificant: false and you are trying to import zeros. Or it could be that the configured user does not have the appropriate write access to save new data for the data element. In short, there are many reasons why it might be ignored.
I understand your point that it would make it significantly easier to understand the results if you knew the specific reason for why the data was ignored on import. I will talk with the team about whether this is something we can modify to improve more details.

If you’d like, we can also tag some individuals on this post who have more implementation experience and might be able to suggest common reasons for ignored data with the aggregate data exchange.

I continually get a status of STOPPED rather than COMPLETED. It seems like its stopping after calculating the values for the PIs but before sending them to the aggregate dataset. How can I troubleshoot this

We’re looking into some issues relating to the running of data exchanges as scheduled jobs, and I will flag your problem here as something to look into as part of that troubleshooting. I’m not sure what would cause the job to have stopped, and do think that we might also improve the messaging here to provide some additional detail.

12

Thanks @tzemp !

@kstankevitz would you please share the Caralina.out relevant log errors (without sensitive/authentication info)? The log is very helpful with debugging and could help with understanding when and why an issue happened.

Thanks!

13

It only seems to “ignore” data that has stayed the same. So in my impression, this is a bug. It actually seems to be doing this on the import/export app also.

Okay, I think this is actually working as intended then, even though it is showing STOPPED instead of COMPLETED. Our use case requires the data to be automatically pushed to the aggregate system. We do daily data review and don’t want to have to go in and trigger each exchange. I’m thinking we can just write a script to do this via the aggregateDataExchanges endpoint…

POST /api/aggregateDataExchanges/{id}/exchange

Thanks a lot for your help!

1 Like
14

Thanks @kstankevitz . Just to clarify: it should be possible to run the data exchanges with a scheduled job (such that you do not need to write your own script for scheduling this), but we acknowledge that this is not currently always working as intended. I didn’t mean to imply that you would need to click “Submit” from within the app.

While we are looking into the scheduled job issues, the plan to directly POST to the api sounds sensible.

Please let us know if you notice any other issues or have any other suggestions. We’re trying to improve this service and the ease of transferring data from tracker to aggregate for the next release (v42), so we greatly appreciate any feedback and ideas.

15

Thanks for the clarification.

Sure. One feature request (which I was planning on submitting, after a colleague finishes some further testing of how it currently works) would be the ability for the data exchange to check for values that are no longer relevant and remove them from the aggregate dataset.

For example:
Our indicators count the number of people who were tested for HIV, by age, sex, etc. Say one TEI is entered as a 60 year old woman, and she’s tested for HIV. The aggregate data exchange calculates this and puts it in the right place in the dataset. But later, while cleaning data. We realize her DOB was captured wrong and she’s really 20, not 60. We change her age. When we rerun the data exchange, it will find the new 20 year old and put that value in the dataset. But it wont delete the 60 year old who no longer exists.

In our scripts that we’ve used to do data exchange (pre-app), we first clear out the dataset at the selected org units, then populate it with the newly calculated data. This ensures we don’t have any “ghost” values, as we call them.

I realize not everyone may want this feature though, so it might require a toggle on/off. Its essential for tracker to aggregate, as data are very detailed and minor changes can modify the ultimate aggregate results.

2 Likes
16

Thanks @kstankevitz : Yes. that’s a great point, and it’s a challenge we’re aware of.

We have an open ticket for this (https://dhis2.atlassian.net/issues/DHIS2-16851), and we need to think through the best way how to address this.

When we start working on that and have some more concrete ideas, we’ll reach out in case you’d like to share any thoughts.

Thanks again.

1 Like