Capture App - Out of Memory Error in DHIS 2.37.7.1

Support - Assistance technique
1

Hi folks ,

We have upgraded our server from 2.31.5 to 2.37.7.1 and we see an issue while entering data in capture app. The user is being logged out frequently and is getting 502/504 after being logged out couple of times. We increased the RAM size from 4 GB ( which did not give any issue for event capture with version 2.31) to 12 GB but it did not help.

[ERROR] 2022-09-29 16:04:33.258 [taskScheduler-3]
SimpleAsyncUncaughtExceptionHandler - Unexpected exception occurred invoking async
method: public void
org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.
dhis.programrule.engine.StageCompletionEvaluationEvent)
java.lang.OutOfMemoryError: Java heap space

captureAppLogs.docx (23.7 KB)

  1. List item
1 Like
2

Hi
Welcome to the community! @Sneha_Gudi :tada::tada:

I checked the log it says “A soft-locked cache entry was expired by the underlying cache. If this happens regularly you should consider increasing the cache timeouts and/or capacity limits” Did you try that?

Going through the docs, I see these related settings that you might want to check and update:

The Tomcat configuration file is located in tomcat-dhis/conf/server.xml
connectionTimeout

Check the dhis.conf file:

----------------------------------------------------------------------
Database connection for PostgreSQL [Optional]
----------------------------------------------------------------------
Minimum number of Connections a pool will maintain at any given time (default: 5).

connection.pool.min_size=5

Initial size of connection pool (default : 5)
Number of Connections a pool will try to acquire upon startup. Should be between minPoolSize and maxPoolSize

connection.pool.initial_size=5

Determines how many connections at a time will try to acquire when the pool is exhausted.

connection.pool.acquire_incr=5

Seconds a Connection can remain pooled but unused before being discarded. Zero means idle connections never expire. (default: 7200)

connection.pool.max_idle_time=7200

Number of seconds that Connections in excess of minPoolSize should be permitted to remain idle in the pool before being culled (default: 0)

connection.pool.max_idle_time_excess_con=0

Analytics server-side cache expiration in seconds

analytics.cache.expiration = 3600

Please if changes to these settings improves the situation. Thank you!

1 Like
3

Thank You @Gassim , We tried changing connection time out but did not help.

We have realised that this was happening due to program rule actions like Send message and Scheduled message , so are planning to set system.program_rule.server_execution=off and check . We are assuming this will only have impact when we try to import data and nothing to do with UI data entry.

please share your inputs on this .

3 Likes
4

@phil Philip Larsen Donnelly and @Rene Can you help us understand the purpose of the system.program_rule.server_execution and share your inputs.

5

Hi @Sneha_Gudi,
setting system.program_rule.server_execution=off in the configs will only turn off the server side execution of program rules.
The UI will continue to work exactly the same as before.
I hope this was the information you were looking for.
Cheers

3 Likes
6

@enrico Thank You for sharing your inputs !

We would like to know more on server side execution of program rules, does that mean when we import events the execution of program rule actions on those imported events will not be done,
i.e the value to be updated by the ASSIGN VALUE action of a program rule will not be assigned to the given Data element ?

1 Like
7

Hi @Sneha_Gudi,
system.program_rule.server_execution=off will disable server side rules execution.
Said that, when you are using Capture App, program rules are executed in the frontend and any assign that you configured will be executed in the frontend.

Let’s say you have DataElementA that get calculated by some rule, you will see in the Capture App page the value there, then when you save, the value will be send to the server (where rules are not being executed) and correctly saved.

Having rules disabled in the server is a problem just for SENDMESSAGE and SCHEDULEMESSAGE types.
We run also ASSIGN as a form of validation, to double check that a value is correctly calculated but if you are using Capture App, this is already happening in the frontend.

3 Likes
8

Thank You @enrico !

Agreed and got the point but we see that when we import events vide import export app, the Assign value action is not executed even when system.program_rule.server_execution is NOT disabled. Assuming it should execute the server side program rule actions and assign the value and write to DB.

9

Sorry I was giving some wrong information.
The only rules executed in the server in version 2.37.7.1 are SENDMESSAGE and SCHEDULEMESSAGE. What was confusing me is that behind the scenes we are running assign rules but only to have the other actions work fine, assign rules results are not saved to the DB.

Starting from version 2.38, Capture App started to use a new endpoint in the server and there all the relevant rules are actually executed, including the assign ones.
I am not sure what endpoint is used in the Import/Export App in version 2.38, @Markus or @Joakim should be able to answer that

2 Likes
10

Note that this issue has been resolved in the upcoming releases of 2.37, 2.39, and 2.40. It was also fixed in 2.38.3 which was released yesterday.

In affected versions of DHIS2, as Enrico mentioned, you can avoid this issue by setting system.program_rule.server_execution = off in dhis.conf.

Please let us know if you continue to see any performance issues after upgrading to a version in which this issue has been fixed (2.37.10, 2.38.3, 2.39.2, 2.40.0)

3 Likes