Limit number of synchronized events per TEI to improve Android app performance

Support - Assistance technique
,
1

Hi everyone,

We are experiencing a performance bottleneck with the DHIS2 Android Capture app regarding Tracked Entity Instances (TEIs) that contain a massive volume of historical events.

Our Use Case:
We run a Tracker program with repeatable events to log continuous services for a Home-Based Care project. On average, our social helpers only manage about 7 TEIs each on their devices. However, because they visit the same beneficiaries regularly (approximately 8 times per month), some TEIs already have over 200+ events accumulated.

The Problem:
Currently, DHIS2 allows us to limit the total number of TEIs downloaded to the device, but there is no mechanism to limit the number of events synchronized per TEI. As a result, opening these high-event TEIs in the Android app has become incredibly slow. As time goes on and more events are created, this performance degradation will only worsen, making daily data entry highly inefficient for field staff.

Proposed Solution / Question:
We urgently need a way to restrict which events are downloaded per TEI-for example, limiting them by a specific time window (e.g., only sync events from the last 3 months) or by a maximum count (e.g., only sync the 20 most recent events).

  • Are there any plans on the DHIS2 roadmap to implement event-level sync limits per TEI?

  • In the meantime, does the community have any advice, best practices, or workarounds for handling TEIs with long-term, high-frequency event data on mobile devices?

Looking forward to your insights and advice!

2

Hi Mikhailo
We have exactly the same problem for LMIS. I do not know of any immediate plans but we have opened a roadmap issue 310 on 24.10.2024:
“Limit Tracker program events downloaded to a mobile device”.
Which is currently a “Roadmap candidate” which just means that it is under consideration for the future.
If you have access, it would be great if you can add an “Insight” or allow me to copy your exact posting to support the case:

Just in case you do not have access, I am posting my use case for you:

Use case (context)

Storekeepers using the DHIS2-RTS (Real-Time Stock management) application will record anything between 30-300 transactions per day which amounts to up to 100,009 transactions (events) per year. Currently DHIS2 allows limiting the number of events downloaded to a mobile device to be limited for Event programs but not for Tracker programs. Limiting the number TEIs (which is possible) is not an option as storekeepers must be able to effect transactions for all of the TEIs (healthcare products) in their OU (Organisation unit).

Eventually, the number of events which are downloaded from the central server to the local database on the mobile device will increase to a very large number. Without knowing how many events an average mobile device can handle, the concern is that, eventually, the number of events will increase to a level where the memory of the mobile device is exhausted and this will slow down and eventually stall the DHIS2-RTS. The concern is both for the DHIS2-RTS for recording transactions and for event data used for the offline, in-app analytics.

It is critical that for all TEIs the most recent event is available on the mobile device as the DHIS2-RTS uses a program rule to (re)calculate the stock on hand which is based on the stock on hand of the most recent transaction. However, for some items (TEIs) dozens of transactions may be entered every day while for other (slow moving) items, the most recent transaction might have been made months or even more than a year ago. Consequently, any kind of limitation using a time period is not suitable. Instead the number of events which are downloaded must be limited by a defined number for each TEI.

The ideal solution would be that DHIS2 limits the number of downloaded events for each TEI:

  • last 14 days (or another configurable time period) if there were less than 100 events, otherwise
  • most recent 100 events (or another configurable value)

As a DHIS2 user - storekeeper

When - Using the DHIS2-RTS application in Capture Android app on a mobile device and synchronising data from the central server to the local database (on the device)

I want to - limit the number of events which are downloaded (or “refreshed”) on the mobile device

So I can - avoid overloading the memory of the mobile device which eventually will slow down and stall the Capture Android app

Who faces this problem

All DHIS2-RTS users having used the DHIS2-RTS mobile app for a considerable amount of time.

How do they solve these problems today?

Currently the problem is only hypothetical as the DHIS2-RTS is only being implemented. But once a large amount of events are recorded, there will be no remedy. The storekeeper could delete all local data but as at least the most recent Event is indispensable for recalculating the current stock on hand, will have to synchronise all data. And as there is no feature for limiting the number of Tracker program events which can be downloaded, the storekeeper can only download all events. The choice is therefore between not synchronising at all which makes the DHIS2-RTS unusable or downloading all events which will, eventually, stall the mobile app and also make it unusable.

Consequently, eventually and inevitably the DHIS2-RTS becomes unusable.

The only work around would be to delete data from the central server which is not recommended. Moreover, all analytics for the period for which data was deleted would also be deleted.

3

Thanks @George_McGuire and @m.siusko and for raising this to the community’s attention. Definitely contributing with insights and why a certain feature is needed will help the @dhis2-android team prioritize it. :folded_hands:

This is also related to this Jira ticket: Jira and this post: How many Tracker program events can the Capture Android app handle?

Thanks!

4

Thank you both for your requirements and usecases @m.siusko and @George_McGuire!

I wanted to check whether either of you have (or could) check out the new functionality in 3.3 where you can download based on working lists. So you can set up a program stage working list with the parameters you are mentioning (sync events from last 3 months - can’t do x number of events yet with this solution).

More info. Settings configuration - DHIS2 Documentation

Would you be able to test this and see if it works the way you need it? Let us know if you need any support.

Thanks,
Karoline
DHIS2 Product Manager

5

Hi @m.siusko
It would be very useful to know more about where it is slow. Have you performed any analysis or the server using Glowroot or similar performance monitoring tools?

There are numerous, known structural problems with the queries which fetch data for tracker events, many of them solved in v43. For previous versions, some recent backports may help.

Regardless,it would be great if you could share any particularly slow traces you see from Glowroot.

Best regards,
Jason

6

Hi @Karoline ,
As I understand this new functionality clearly,
It can filter TEIs based on certain criteria. In our use case we need ALL TEIs to be downloaded, but limit events per TEI. So this wont help((

7

Hi @m.siusko,

Thanks for bringing this topic here. We are aware of this potential issue in implementations where the number of events per TEI is really high. And we hope to have a solution for next year, it is part of the roadmap.

Just to have a better understanding of the performance bottleneck in your implementation: is the sync process taking a very long time? Or is the app responsiveness the main issue (the app is slow managing so many events)?

8

Hi @vgarciabnz ,
The main issue is responsiveness. Especially when opening a TEI. It can took more then 10 seconds to open. On some comparably old and low-performance android devices (2021 year OPPP A53, 4 gb RAM, SM4250 CPU, Android 12) it even fails to open a TEI and goes to Android App Home screen.

On our newer tablets (Lenovo Tab 11 2nd gen, 6 RAM, Helio G99 CPU, Android 14) it works. But very slowly. Users frequently receive hanging (stop responding) messages like this:

image
image864Ă—988 76.1 KB
(sorry, the screenshot is in Ukrainian)

9

The reason that I have suggested looking at Glowroot on the server is because it can help to pinpoint the actual performance bottleneck, i.e. is it slowness on the server or client?

In certain cases we have also used the reverse proxy to modify the request made by clients in order to limit the amount of data requested by clients (including the android client), which can dramatically speed things up.

10

@jason , TY very much for the idea to test it with Glowroot. We will try it for sure.

11

Thanks a lot Karoline! That sounds promising and I will test it for DHIS2-RTS (LMIS).

12

Hi again @m.siusko

I would particularly pay attention to the parameters that the Android client is sending. The Capture client (by default) does not send any date ranges when loading the working list. I think the feature that @Karoline describes may solve that problem but I am not sure. When loading a program, the client sends a request like this:

https://play.im.dhis2.org/stable-2-42-5/api/42/tracker/events?page=1&pageSize=15&program=lxAQ7Zs9VYR&orgUnit=DiszpKrYNg8&programStage=dBwrot7S420&orgUnitMode=SELECTED&order=occurredAt%3Adesc&fields=...

The problem here is that there query does not contain an occurredBefore or ocurredAfter date. Without this, the Postgres query planner has no entry point to begin to retrieve records, and the result is usually a full sequential scan of the events table. On systems with many events, this can lead to a disastrous database query, which can continue to pile up, and may eventually create server instability.

I am not sure exactly what query the Android client sends, but if you could share some of the Glowroot traces once you have had a chance to look, I might be able to help out more. That is of course if the slowness is actually on the server side. If the queries are fast on the server, but are rendered slowly on the client, that I guess is really a totally different problem,

Best regards,
Jason

13

Hi Mikhailo
I posted your use case and the link to this thread on roadmap 310:

“Limit Tracker program events downloaded to a mobile device”

14

Hi Karoline
I studied the instructions but have the same as issue as Mikhail:

  • the user must have access to all TEIs (in our case always limited to a single OU)
  • the limit is needed on number of events for each of the TEIs

In our use case, every TEI is a drug product and the Tracker Program uses repeatable stages. So if we limit the TEI number, the user can only record stock transactions for some drug products and not others.
In DHIS2-RTS the user cannot even view past transactions. Technically, only the last event for every TEI is needed for the system to have a record of the current stock on hand.
But, for example in Comoros, the system will download all events (transactions) for any specific OU for all TEIs of the last year (from the time of implementation) although they are of no use to the user.
We would need either a filter for the number of Tracker events downloaded for each TEI or time fencing (such as all events for each TEI for the past, say, 3 months).