Sever cache issue resulting in incomplete usergroup list in the user endpoint in 2.38.7

Support - Assistance technique
1

Issue Overview:

We are encountering a sporadic issue where the DHIS2 API returns incomplete usergroup lists for users (in the api/users/x endpoint), while the api/usergroups endpoint shows the correct list of users.
This is critical as it affects user access to resources.

Detailed explanation:
Most usergroups disappear from users, but the api/usergroups endpoint still shows the correct users within the groups.

For example, a user (e.g., exampleuserid) may lose the exampleusergroupid in the user.userGroup list, but when checking /api/userGroups/exampleusergroupid, we still find the user in the group.

We’ve been monitoring the issue, and while logs don’t reveal much, we’ve noticed a common pattern: a ‘Hibernate caches cleared’ message appears before the issue, typically after users import metadata (e.g., creating a dashboard). This may be coincidental and unrelated to the usergroup loss.

In one case, a user with 135 usergroups only had 3 groups displayed in the api/users/x endpoint, and it seems the same groups are consistently missing.

A browser cache issue is discarded.

Suspected Cause:

We suspect the issue is related to Hibernate cache not updating correctly, causing incomplete usergroup lists. Since restarting the instance or clearing the cache resolves the issue without data loss, we believe it’s tied to how the cache handles user-usergroup associations.

Temporary solution:
Currently, we resolve this by restarting DHIS2 or clearing the cache via Data Administration → Maintenance.

We plan to implement a script to reset the cache automatically when the error happears, but this isn’t ideal. We’d like to understand the root cause and whether adding more logs related to Hibernate could help (ideally logging these separately to reduce noise).

Additional Details:

We have over 3,000 users and around 350 usergroups, with some users belonging to as many as 135 groups.
Some of the usergroups that are failing has around 2200 users inside.

System Information:
org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.70
org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/lib/jvm/java-11-openjdk-11.0.21.0.9-2.el9.x86_64
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx23000m
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms4000m
Postgresql version 14
dhis2 version: 2.38.7 (we will migrate in the next months to 2.41, but we need a workaround to fix this problem and know if this could still happens in 2.41).

We have created a jira issue [DHIS2-18071] - Jira
, but since maintenance for version 2.38.7 is no longer available, perhaps this is the right place to discuss it.

1 Like
2

Hi @idelcano

Welcome back to the community, and thank you for the detailed post. May I ask if you can find the full list of users when going to the Users app (dhis2-instance/dhis-web-user/index.html#/users)? And the same question for the other parts of the Users app (User groups)?

Thanks!

3

Hi
I tested the URL you mentioned, and it seems there are fewer users than the total. If I do:

api/users, we have 3724 users.

But this request:
api/38/users?page=36&pageSize=100&userOrgUnits=true&includeChildren=true&query=&fields=id,displayName,access,email,userCredentials[username,disabled,lastLogin,twoFA],teiSearchOrganisationUnits[id,path]&order=firstName:asc,surname:asc&selfRegistered=false
Returns 3692 as the total in the pager object.

I didn’t find any issue with the number of user groups.

This might provide an additional clue: I also found that in these two request, when the error occurs, one works, and the other doesn’t:

This one works:
api/users?fields=username,userGroups&page=1&pageSize=25&query=test_user&filter=userGroups.id:in:[groupuid]&rootJunction=OR&order=firstName:asc
This one returns fewer groups for the user:
api/users?fields=username,userGroups&page=1&pageSize=25&query=test_user&order=firstName:asc

I have tested all these requests while the error is not occurring (since we don’t know how to replicate it).