Possible Memory Leak with Version 2.37.3

We are seeing very high memory and CPU usage with the version of DHIS2 2.37. I have increased instance size to have 8 Processing cores with 40GB RAM; 32Gs available to Apache Tomcat for DHIS2. (Xmx 32G, Xms 32G)

Following is the screenshot of VisualVM monitoring. You can see the memory usage spiked to near 25GBs before a run of garbage collection brought it down to moderate 10GBs. Once the application tops the maximum value memory available to it it completely crashes and apache needs to be restarted.

I am also attaching the output of Catalina from crash this morning along with DHIS2 logs. If someone can analyze and help us it would be greatly appreciated.

1 Like

CATALINA

05-Mar-2022 10:39:18.626 SEVERE [https-jsse-nio-443-exec-111] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Failed to complete processing of a request
	java.lang.OutOfMemoryError: Java heap space
05-Mar-2022 10:39:18.642 SEVERE [https-jsse-nio-443-exec-99] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Failed to complete processing of a request
	java.lang.OutOfMemoryError: Java heap space
05-Mar-2022 10:39:18.751 SEVERE [https-jsse-nio-443-exec-107] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Failed to complete processing of a request
	java.lang.OutOfMemoryError: Java heap space
05-Mar-2022 11:37:53.338 INFO [Thread-17] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:37:54.400 INFO [Thread-17] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
05-Mar-2022 11:37:55.372 WARNING [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [natm] appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/java.lang.Object.wait(Native Method)
 java.base@11.0.13/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
 java.base@11.0.13/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
 com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)
05-Mar-2022 11:37:55.372 WARNING [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [natm] appears to have started a thread named [cache2k-scheduler] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.13/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
 java.base@11.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.13/java.lang.Thread.run(Thread.java:834)
05-Mar-2022 11:37:55.372 WARNING [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [natm] appears to have started a thread named [cache2k-scheduler] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.13/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
 java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
 java.base@11.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
 java.base@11.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.13/java.lang.Thread.run(Thread.java:834)
05-Mar-2022 11:37:55.385 WARNING [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [natm] appears to have started a thread named [Thread-75 (ActiveMQ-client-global-threads)] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.13/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
 java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
 java.base@11.0.13/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458)
 org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
 org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
05-Mar-2022 11:37:55.385 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@69810994]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5e16581e]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [SecurityContextImpl [Null authentication]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4ab75fd4]) and a value of type [org.hibernate.boot.internal.MetadataImpl] (value [org.hibernate.boot.internal.MetadataImpl@31f1ba65]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@7377071f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1daf6b26]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@4e1affc0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@54d7355f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1fd39f25]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@794321a2]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@603e25b5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@d2e9920]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@e070517]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@6ea97b44]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@7f2bf8a9]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@6b589f5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@30ac1b89]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@157a3583]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@35b47935]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1fe4b457]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@4fc49471]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@b566627]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1d3dcba9]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@27418255]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.432 INFO [Thread-17] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:37:55.432 INFO [Thread-17] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:39:06.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/9.0.39
05-Mar-2022 11:39:06.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Oct 6 2020 14:11:46 UTC
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.39.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Windows Server 2012 R2
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            6.3
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             C:\Program Files\Java\jdk-11.0.13
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           11.0.13+10-LTS-370
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         C:\Program Files\Apache Software Foundation\Tomcat 9.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         C:\Program Files\Apache Software Foundation\Tomcat 9.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 9.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 9.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 9.0\temp
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 9.0\conf\logging.properties
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfile.encoding=UTF8
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dlog4j2.formatMsgNoLookups=true
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote=true
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.port=9090
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.ssl=false
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.authenticate=false
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.rmi.server.hostname=10.10.1.235
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms32144m
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx32144m
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [C:\Program Files\Apache Software Foundation\Tomcat 9.0\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program Files\Java\jdk-11.0.13\bin;C:\Program Files\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Java\jdk1.8.0_271\bin;C:\Program Files\PostgreSQL\12\bin;C:\Users\Administrator\AppData\Local\Programs\Python\Python39;;.]
05-Mar-2022 11:39:06.585 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:39:06.944 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1139] milliseconds
05-Mar-2022 11:39:07.038 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
05-Mar-2022 11:39:07.038 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.39]
05-Mar-2022 11:39:07.069 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\natm.war]
05-Mar-2022 11:39:29.272 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
05-Mar-2022 11:40:10.834 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\natm.war] has finished in [63,765] ms
05-Mar-2022 11:40:10.834 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\manager]
05-Mar-2022 11:40:10.881 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\manager] has finished in [47] ms
05-Mar-2022 11:40:10.881 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\ROOT]
05-Mar-2022 11:40:10.912 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\ROOT] has finished in [31] ms
05-Mar-2022 11:40:10.912 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:40:10.928 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [63989] milliseconds

DHIS2 Logs

* INFO  2022-03-05T09:32:03,269 Validation done for data set: 'GeQcXbMeHzo', period: '202202', org unit: 'f2i45ymtOdM', validation rule count: 17, violations found: 0 (ValidationAction.java [https-jsse-nio-443-exec-7])
* INFO  2022-03-05T09:32:03,363 Starting complete data set registration import, options: ImportOptions{idSchemes=IdSchemes{idScheme=null, dataElementIdScheme=null, categoryOptionComboIdScheme=null, categoryOptionIdScheme=null, categoryIdScheme=null, orgUnitIdScheme=null, programIdScheme=null, programStageIdScheme=null, trackedEntityIdScheme=null, trackedEntityAttributeIdScheme=null, dataSetIdScheme=null, attributeOptionComboIdScheme=null, programStageInstanceIdScheme=null}, dryRun=false, preheatCache=null, async=false, importStrategy=CREATE_AND_UPDATE, mergeMode=REPLACE, skipExistingCheck=false, ignoreEmptyCollection=false, sharing=false, skipNotifications=false, datasetAllowsPeriods=false, strictPeriods=false, strictDataElements=false, strictCategoryOptionCombos=false, strictAttributeOptionCombos=false, strictOrganisationUnits=false, requireCategoryOptionCombo=false, requireAttributeOptionCombo=false, force=false, firstRowIsHeader=true, skipLastUpdated=false, skipCache=false, skipDataValueMandatoryValidationCheck=false}: 00:00:00.000 (Clock.java [https-jsse-nio-443-exec-51])
* INFO  2022-03-05T09:32:03,363 Import options: ImportOptions{idSchemes=IdSchemes{idScheme=null, dataElementIdScheme=null, categoryOptionComboIdScheme=null, categoryOptionIdScheme=null, categoryIdScheme=null, orgUnitIdScheme=null, programIdScheme=null, programStageIdScheme=null, trackedEntityIdScheme=null, trackedEntityAttributeIdScheme=null, dataSetIdScheme=null, attributeOptionComboIdScheme=null, programStageInstanceIdScheme=null}, dryRun=false, preheatCache=null, async=false, importStrategy=CREATE_AND_UPDATE, mergeMode=REPLACE, skipExistingCheck=false, ignoreEmptyCollection=false, sharing=false, skipNotifications=false, datasetAllowsPeriods=false, strictPeriods=false, strictDataElements=false, strictCategoryOptionCombos=false, strictAttributeOptionCombos=false, strictOrganisationUnits=false, requireCategoryOptionCombo=false, requireAttributeOptionCombo=false, force=false, firstRowIsHeader=true, skipLastUpdated=false, skipCache=false, skipDataValueMandatoryValidationCheck=false} (DefaultCompleteDataSetRegistrationExchangeService.java [https-jsse-nio-443-exec-51])
* INFO  2022-03-05T09:32:03,363 Data set scheme: IdScheme{identifiableProperty=UID, attribute=null}, org unit scheme: IdScheme{identifiableProperty=UID, attribute=null}, attribute option combo scheme: IdScheme{identifiableProperty=UID, attribute=null} (ImportConfig.java [https-jsse-nio-443-exec-51])
* INFO  2022-03-05T09:32:14,878 Authentication event: AuthenticationSuccessEvent; username: ml-southwaziristan-de; ip: 39.43.67.102; sessionId: 888d19669421ba6f41987f113a4d9acc9b2f1d0e0d358ef23ec3654c49aa0cc6;  (AuthenticationLoggerListener.java [https-jsse-nio-443-exec-43])
* INFO  2022-03-05T09:32:25,378 No template found (DefaultDataSetNotificationService.java [https-jsse-nio-443-exec-51])
* WARN  2022-03-05T09:32:56,674 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@16cb0958 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! (Slf4jMLog.java [C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-AdminTaskTimer])
* WARN  2022-03-05T10:39:18,626 Java heap space (OrderedExecutor.java [Thread-618 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
java.lang.OutOfMemoryError: Java heap space
* WARN  2022-03-05T10:39:18,783 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@2068d50d<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@1ca32dcb [wrapping: null]> (LoadContexts.java [taskScheduler-11])
* WARN  2022-03-05T10:39:18,783 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-11])
* WARN  2022-03-05T10:39:18,829 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@1a7880a0<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@604c0338 [wrapping: null]> (LoadContexts.java [taskScheduler-1])
* WARN  2022-03-05T10:39:18,829 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-1])
* WARN  2022-03-05T10:39:18,845 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@508f7e0c<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@5a0471e3 [wrapping: null]> (LoadContexts.java [taskScheduler-13])
* WARN  2022-03-05T10:39:18,845 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-13])
* WARN  2022-03-05T10:39:18,861 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@5ca0bc37<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@31778609 [wrapping: null]> (LoadContexts.java [taskScheduler-25])
* WARN  2022-03-05T10:39:18,861 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-25])
* ERROR 2022-03-05T10:39:18,861 AMQ214013: Failed to decode packet (RemotingConnectionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:18,861 AMQ224016: Caught exception (ServerSessionPacketHandler.java [Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@1b8d9495)])
java.lang.OutOfMemoryError: Java heap space
* WARN  2022-03-05T10:39:18,892 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@5841c2d3<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@63a8b06d [wrapping: null]> (LoadContexts.java [taskScheduler-10])
* WARN  2022-03-05T10:39:18,892 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-10])
* WARN  2022-03-05T10:39:18,908 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@1a850f80<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@27a91d20 [wrapping: null]> (LoadContexts.java [taskScheduler-21])
* WARN  2022-03-05T10:39:18,908 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-21])
* WARN  2022-03-05T10:39:18,923 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@756f116<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@3d6e51a3 [wrapping: null]> (LoadContexts.java [taskScheduler-23])
* WARN  2022-03-05T10:39:18,923 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-23])
* WARN  2022-03-05T10:39:18,923 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@7b8010e3<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@7baaee4 [wrapping: null]> (LoadContexts.java [taskScheduler-24])
* WARN  2022-03-05T10:39:18,923 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-24])
* WARN  2022-03-05T10:39:25,707 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@59a9f946<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@712d3dfb [wrapping: null]> (LoadContexts.java [taskScheduler-2])
* WARN  2022-03-05T10:39:25,707 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@4bfab067<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@51f4daf4 [wrapping: null]> (LoadContexts.java [taskScheduler-20])
* WARN  2022-03-05T10:39:25,707 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-2])
* WARN  2022-03-05T10:39:25,754 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@39faa0ad<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@225a0ff6 [wrapping: null]> (LoadContexts.java [taskScheduler-14])
* WARN  2022-03-05T10:39:25,754 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-14])
* WARN  2022-03-05T10:39:25,707 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-20])
* WARN  2022-03-05T10:39:26,364 AMQ222061: Client connection failed, clearing up resources for session 34b26f81-9c3d-11ec-b29c-00155d315008 (ServerSessionImpl.java [https-jsse-nio-443-exec-28])
* WARN  2022-03-05T10:39:26,364 AMQ222107: Cleared up resources for session 34b26f81-9c3d-11ec-b29c-00155d315008 (ServerSessionImpl.java [https-jsse-nio-443-exec-28])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session 2bba7208-9c3d-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-69 (ActiveMQ-client-global-threads)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session 2bba7208-9c3d-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-69 (ActiveMQ-client-global-threads)])
* WARN  2022-03-05T10:39:26,379 AMQ222218: Server disconnecting: Error decoding buffer (RemotingServiceImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
java.lang.IllegalStateException: java.lang.OutOfMemoryError: Java heap space
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:390) ~[artemis-core-client-2.17.0.jar:2.17.0]
	at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:683) ~[artemis-server-2.17.0.jar:2.17.0]
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:209) ~[artemis-server-2.17.0.jar:2.17.0]
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) ~[artemis-commons-2.17.0.jar:2.17.0]
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) ~[artemis-commons-2.17.0.jar:2.17.0]
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) ~[artemis-commons-2.17.0.jar:2.17.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.17.0.jar:2.17.0]
Caused by: java.lang.OutOfMemoryError: Java heap space
* WARN  2022-03-05T10:39:26,379 AMQ212037: Connection failure to invm:0 has been detected: java.lang.OutOfMemoryError: Java heap space [code=GENERIC_EXCEPTION] (RemotingConnectionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b529c909-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b529c909-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b529c90a-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b529c90a-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b53820f7-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b53820f7-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b548c2c9-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b548c2c9-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b548c2cc-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b548c2cc-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b54b33ce-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b54b33ce-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,395 Setup of JMS message listener invoker failed for destination 'dhis2.jobs.tracker.notifications' - trying to recover. Cause: Session is closed (DefaultMessageListenerContainer.java [org.springframework.jms.JmsListenerEndpointContainer#4-1])
* WARN  2022-03-05T10:39:26,801 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@16cb0958 -- APPARENT DEADLOCK!!! Complete Status: 
	Managed Threads: 3
	Active Threads: 3
	Active Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@2789a43d
			on thread: C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#1
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@75afb5ff
			on thread: C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#2
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1398af81
			on thread: C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#0
	Pending Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5b0c4707
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@314eb817
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@39c6f38c
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@6ff4a792
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5679e114
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@12b375cc
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1c67ae85
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@566552ac
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@4f0d7835
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@4619690d
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@672be4cd
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1b273ab6
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@3f4f2537
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@321362d2
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@aa6f5d1
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@28768cb9
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@6ed6721d
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@47fa98e7
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@7acff5b4
Pool thread stack traces:
	Thread[C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#1,5,main]
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:720)
	Thread[C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#2,5,main]
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:720)
	Thread[C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#0,5,main]
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:720)
 (Slf4jMLog.java [C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-AdminTaskTimer])
* ERROR 2022-03-05T10:39:26,817 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-20])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,817 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-14])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,833 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-23])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,833 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-24])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,833 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-2])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,833 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-21])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-25])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-10])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-13])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-1])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-11])
java.lang.OutOfMemoryError: Java heap space
* INFO  2022-03-05T10:39:27,333 Complete data set registration import done, total: 1, imported: 1, updated: 0, deleted: 0: 01:07:24.207 (Clock.java [https-jsse-nio-443-exec-51])
* ERROR 2022-03-05T10:39:27,333 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-5])
java.lang.OutOfMemoryError: Java heap space
	at org.hibernate.property.access.spi.GetterMethodImpl.get(GetterMethodImpl.java:42) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.tuple.entity.AbstractEntityTuplizer.getPropertyValues(AbstractEntityTuplizer.java:574) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.tuple.entity.PojoEntityTuplizer.getPropertyValues(PojoEntityTuplizer.java:154) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.persister.entity.AbstractEntityPersister.getPropertyValues(AbstractEntityPersister.java:5174) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.event.internal.DefaultFlushEntityEventListener.getValues(DefaultFlushEntityEventListener.java:207) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.event.internal.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:164) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.event.internal.AbstractFlushingEventListener$$Lambda$1147/0x00000017da9d7440.accept(Unknown Source) ~[?:?]
	at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:93) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.event.internal.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:230) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:93) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.internal.SessionImpl$$Lambda$1143/0x00000017da9d1040.accept(Unknown Source) ~[?:?]
	at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:93) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1362) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:453) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3212) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2380) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:447) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:183) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$300(JdbcResourceLocalTransactionCoordinatorImpl.java:40) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:281) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:101) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.springframework.orm.hibernate5.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:643) ~[spring-orm-5.3.6.jar:5.3.6]
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743) ~[spring-tx-5.3.6.jar:5.3.6]
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711) ~[spring-tx-5.3.6.jar:5.3.6]
	at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654) ~[spring-tx-5.3.6.jar:5.3.6]
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407) ~[spring-tx-5.3.6.jar:5.3.6]
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.6.jar:5.3.6]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at com.sun.proxy.$Proxy314.evaluateEventAndRunEffects(Unknown Source) ~[?:?]
	at org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(ProgramRuleEngineListener.java:61) ~[dhis-service-program-rule-2.37.3.jar:?]

Hello @Junaid_Mufti Can you specify -Xmx20g -Xms10g
and make sure you kill first the processid running java

2 Likes

Thank you. What effect will this have other than setting MAX size to 20GB and startup size to 10GB ? Just trying to understand.

1 Like

Hello @Junaid_Mufti here the point is that the allocation of Memory to JAVA should not exceed the total available memory, whatever allocation of memory should not exceed total available server memory

1 Like