Possible Memory Leak with Version 2.37.3

Junaid_Mufti · 5 March 2022 12:58

We are seeing very high memory and CPU usage with the version of DHIS2 2.37. I have increased instance size to have 8 Processing cores with 40GB RAM; 32Gs available to Apache Tomcat for DHIS2. (Xmx 32G, Xms 32G)

Following is the screenshot of VisualVM monitoring. You can see the memory usage spiked to near 25GBs before a run of garbage collection brought it down to moderate 10GBs. Once the application tops the maximum value memory available to it it completely crashes and apache needs to be restarted.

I am also attaching the output of Catalina from crash this morning along with DHIS2 logs. If someone can analyze and help us it would be greatly appreciated.

Junaid_Mufti · 5 March 2022 12:58

CATALINA

05-Mar-2022 10:39:18.626 SEVERE [https-jsse-nio-443-exec-111] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Failed to complete processing of a request
	java.lang.OutOfMemoryError: Java heap space
05-Mar-2022 10:39:18.642 SEVERE [https-jsse-nio-443-exec-99] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Failed to complete processing of a request
	java.lang.OutOfMemoryError: Java heap space
05-Mar-2022 10:39:18.751 SEVERE [https-jsse-nio-443-exec-107] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Failed to complete processing of a request
	java.lang.OutOfMemoryError: Java heap space
05-Mar-2022 11:37:53.338 INFO [Thread-17] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:37:54.400 INFO [Thread-17] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
05-Mar-2022 11:37:55.372 WARNING [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [natm] appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/java.lang.Object.wait(Native Method)
 java.base@11.0.13/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
 java.base@11.0.13/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
 com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)
05-Mar-2022 11:37:55.372 WARNING [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [natm] appears to have started a thread named [cache2k-scheduler] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.13/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
 java.base@11.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.13/java.lang.Thread.run(Thread.java:834)
05-Mar-2022 11:37:55.372 WARNING [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [natm] appears to have started a thread named [cache2k-scheduler] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.13/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
 java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
 java.base@11.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
 java.base@11.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.13/java.lang.Thread.run(Thread.java:834)
05-Mar-2022 11:37:55.385 WARNING [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [natm] appears to have started a thread named [Thread-75 (ActiveMQ-client-global-threads)] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.13/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
 java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
 java.base@11.0.13/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458)
 org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
 org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
05-Mar-2022 11:37:55.385 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@69810994]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5e16581e]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [SecurityContextImpl [Null authentication]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4ab75fd4]) and a value of type [org.hibernate.boot.internal.MetadataImpl] (value [org.hibernate.boot.internal.MetadataImpl@31f1ba65]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@7377071f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1daf6b26]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@4e1affc0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@54d7355f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1fd39f25]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@794321a2]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@603e25b5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@d2e9920]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@e070517]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@6ea97b44]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@7f2bf8a9]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@6b589f5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@30ac1b89]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@157a3583]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@35b47935]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1fe4b457]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@4fc49471]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@b566627]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@1d3dcba9]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.400 SEVERE [Thread-17] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [natm] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40b46d2c]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@27418255]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
05-Mar-2022 11:37:55.432 INFO [Thread-17] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:37:55.432 INFO [Thread-17] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:39:06.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/9.0.39
05-Mar-2022 11:39:06.210 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Oct 6 2020 14:11:46 UTC
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.39.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Windows Server 2012 R2
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            6.3
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             C:\Program Files\Java\jdk-11.0.13
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           11.0.13+10-LTS-370
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         C:\Program Files\Apache Software Foundation\Tomcat 9.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         C:\Program Files\Apache Software Foundation\Tomcat 9.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 9.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 9.0
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 9.0\temp
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 9.0\conf\logging.properties
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dfile.encoding=UTF8
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dlog4j2.formatMsgNoLookups=true
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote=true
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.port=9090
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.ssl=false
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.authenticate=false
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.rmi.server.hostname=10.10.1.235
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms32144m
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx32144m
05-Mar-2022 11:39:06.225 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [C:\Program Files\Apache Software Foundation\Tomcat 9.0\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program Files\Java\jdk-11.0.13\bin;C:\Program Files\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Java\jdk1.8.0_271\bin;C:\Program Files\PostgreSQL\12\bin;C:\Users\Administrator\AppData\Local\Programs\Python\Python39;;.]
05-Mar-2022 11:39:06.585 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:39:06.944 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1139] milliseconds
05-Mar-2022 11:39:07.038 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
05-Mar-2022 11:39:07.038 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.39]
05-Mar-2022 11:39:07.069 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\natm.war]
05-Mar-2022 11:39:29.272 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
05-Mar-2022 11:40:10.834 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\natm.war] has finished in [63,765] ms
05-Mar-2022 11:40:10.834 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\manager]
05-Mar-2022 11:40:10.881 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\manager] has finished in [47] ms
05-Mar-2022 11:40:10.881 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\ROOT]
05-Mar-2022 11:40:10.912 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [C:\Program Files\Apache Software Foundation\Tomcat 9.0\webapps\ROOT] has finished in [31] ms
05-Mar-2022 11:40:10.912 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio-443"]
05-Mar-2022 11:40:10.928 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [63989] milliseconds

Junaid_Mufti · 5 March 2022 13:06

DHIS2 Logs

* INFO  2022-03-05T09:32:03,269 Validation done for data set: 'GeQcXbMeHzo', period: '202202', org unit: 'f2i45ymtOdM', validation rule count: 17, violations found: 0 (ValidationAction.java [https-jsse-nio-443-exec-7])
* INFO  2022-03-05T09:32:03,363 Starting complete data set registration import, options: ImportOptions{idSchemes=IdSchemes{idScheme=null, dataElementIdScheme=null, categoryOptionComboIdScheme=null, categoryOptionIdScheme=null, categoryIdScheme=null, orgUnitIdScheme=null, programIdScheme=null, programStageIdScheme=null, trackedEntityIdScheme=null, trackedEntityAttributeIdScheme=null, dataSetIdScheme=null, attributeOptionComboIdScheme=null, programStageInstanceIdScheme=null}, dryRun=false, preheatCache=null, async=false, importStrategy=CREATE_AND_UPDATE, mergeMode=REPLACE, skipExistingCheck=false, ignoreEmptyCollection=false, sharing=false, skipNotifications=false, datasetAllowsPeriods=false, strictPeriods=false, strictDataElements=false, strictCategoryOptionCombos=false, strictAttributeOptionCombos=false, strictOrganisationUnits=false, requireCategoryOptionCombo=false, requireAttributeOptionCombo=false, force=false, firstRowIsHeader=true, skipLastUpdated=false, skipCache=false, skipDataValueMandatoryValidationCheck=false}: 00:00:00.000 (Clock.java [https-jsse-nio-443-exec-51])
* INFO  2022-03-05T09:32:03,363 Import options: ImportOptions{idSchemes=IdSchemes{idScheme=null, dataElementIdScheme=null, categoryOptionComboIdScheme=null, categoryOptionIdScheme=null, categoryIdScheme=null, orgUnitIdScheme=null, programIdScheme=null, programStageIdScheme=null, trackedEntityIdScheme=null, trackedEntityAttributeIdScheme=null, dataSetIdScheme=null, attributeOptionComboIdScheme=null, programStageInstanceIdScheme=null}, dryRun=false, preheatCache=null, async=false, importStrategy=CREATE_AND_UPDATE, mergeMode=REPLACE, skipExistingCheck=false, ignoreEmptyCollection=false, sharing=false, skipNotifications=false, datasetAllowsPeriods=false, strictPeriods=false, strictDataElements=false, strictCategoryOptionCombos=false, strictAttributeOptionCombos=false, strictOrganisationUnits=false, requireCategoryOptionCombo=false, requireAttributeOptionCombo=false, force=false, firstRowIsHeader=true, skipLastUpdated=false, skipCache=false, skipDataValueMandatoryValidationCheck=false} (DefaultCompleteDataSetRegistrationExchangeService.java [https-jsse-nio-443-exec-51])
* INFO  2022-03-05T09:32:03,363 Data set scheme: IdScheme{identifiableProperty=UID, attribute=null}, org unit scheme: IdScheme{identifiableProperty=UID, attribute=null}, attribute option combo scheme: IdScheme{identifiableProperty=UID, attribute=null} (ImportConfig.java [https-jsse-nio-443-exec-51])
* INFO  2022-03-05T09:32:14,878 Authentication event: AuthenticationSuccessEvent; username: ml-southwaziristan-de; ip: 39.43.67.102; sessionId: 888d19669421ba6f41987f113a4d9acc9b2f1d0e0d358ef23ec3654c49aa0cc6;  (AuthenticationLoggerListener.java [https-jsse-nio-443-exec-43])
* INFO  2022-03-05T09:32:25,378 No template found (DefaultDataSetNotificationService.java [https-jsse-nio-443-exec-51])
* WARN  2022-03-05T09:32:56,674 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@16cb0958 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! (Slf4jMLog.java [C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-AdminTaskTimer])
* WARN  2022-03-05T10:39:18,626 Java heap space (OrderedExecutor.java [Thread-618 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
java.lang.OutOfMemoryError: Java heap space
* WARN  2022-03-05T10:39:18,783 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@2068d50d<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@1ca32dcb [wrapping: null]> (LoadContexts.java [taskScheduler-11])
* WARN  2022-03-05T10:39:18,783 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-11])
* WARN  2022-03-05T10:39:18,829 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@1a7880a0<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@604c0338 [wrapping: null]> (LoadContexts.java [taskScheduler-1])
* WARN  2022-03-05T10:39:18,829 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-1])
* WARN  2022-03-05T10:39:18,845 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@508f7e0c<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@5a0471e3 [wrapping: null]> (LoadContexts.java [taskScheduler-13])
* WARN  2022-03-05T10:39:18,845 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-13])
* WARN  2022-03-05T10:39:18,861 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@5ca0bc37<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@31778609 [wrapping: null]> (LoadContexts.java [taskScheduler-25])
* WARN  2022-03-05T10:39:18,861 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-25])
* ERROR 2022-03-05T10:39:18,861 AMQ214013: Failed to decode packet (RemotingConnectionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:18,861 AMQ224016: Caught exception (ServerSessionPacketHandler.java [Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@1b8d9495)])
java.lang.OutOfMemoryError: Java heap space
* WARN  2022-03-05T10:39:18,892 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@5841c2d3<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@63a8b06d [wrapping: null]> (LoadContexts.java [taskScheduler-10])
* WARN  2022-03-05T10:39:18,892 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-10])
* WARN  2022-03-05T10:39:18,908 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@1a850f80<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@27a91d20 [wrapping: null]> (LoadContexts.java [taskScheduler-21])
* WARN  2022-03-05T10:39:18,908 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-21])
* WARN  2022-03-05T10:39:18,923 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@756f116<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@3d6e51a3 [wrapping: null]> (LoadContexts.java [taskScheduler-23])
* WARN  2022-03-05T10:39:18,923 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-23])
* WARN  2022-03-05T10:39:18,923 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@7b8010e3<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@7baaee4 [wrapping: null]> (LoadContexts.java [taskScheduler-24])
* WARN  2022-03-05T10:39:18,923 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-24])
* WARN  2022-03-05T10:39:25,707 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@59a9f946<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@712d3dfb [wrapping: null]> (LoadContexts.java [taskScheduler-2])
* WARN  2022-03-05T10:39:25,707 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@4bfab067<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@51f4daf4 [wrapping: null]> (LoadContexts.java [taskScheduler-20])
* WARN  2022-03-05T10:39:25,707 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-2])
* WARN  2022-03-05T10:39:25,754 HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@39faa0ad<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@225a0ff6 [wrapping: null]> (LoadContexts.java [taskScheduler-14])
* WARN  2022-03-05T10:39:25,754 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-14])
* WARN  2022-03-05T10:39:25,707 HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries (CollectionLoadContext.java [taskScheduler-20])
* WARN  2022-03-05T10:39:26,364 AMQ222061: Client connection failed, clearing up resources for session 34b26f81-9c3d-11ec-b29c-00155d315008 (ServerSessionImpl.java [https-jsse-nio-443-exec-28])
* WARN  2022-03-05T10:39:26,364 AMQ222107: Cleared up resources for session 34b26f81-9c3d-11ec-b29c-00155d315008 (ServerSessionImpl.java [https-jsse-nio-443-exec-28])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session 2bba7208-9c3d-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-69 (ActiveMQ-client-global-threads)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session 2bba7208-9c3d-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-69 (ActiveMQ-client-global-threads)])
* WARN  2022-03-05T10:39:26,379 AMQ222218: Server disconnecting: Error decoding buffer (RemotingServiceImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
java.lang.IllegalStateException: java.lang.OutOfMemoryError: Java heap space
	at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:390) ~[artemis-core-client-2.17.0.jar:2.17.0]
	at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:683) ~[artemis-server-2.17.0.jar:2.17.0]
	at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:209) ~[artemis-server-2.17.0.jar:2.17.0]
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) ~[artemis-commons-2.17.0.jar:2.17.0]
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) ~[artemis-commons-2.17.0.jar:2.17.0]
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) ~[artemis-commons-2.17.0.jar:2.17.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.17.0.jar:2.17.0]
Caused by: java.lang.OutOfMemoryError: Java heap space
* WARN  2022-03-05T10:39:26,379 AMQ212037: Connection failure to invm:0 has been detected: java.lang.OutOfMemoryError: Java heap space [code=GENERIC_EXCEPTION] (RemotingConnectionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b529c909-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b529c909-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b529c90a-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b529c90a-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b53820f7-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b53820f7-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b548c2c9-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b548c2c9-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b548c2cc-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b548c2cc-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222061: Client connection failed, clearing up resources for session b54b33ce-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,379 AMQ222107: Cleared up resources for session b54b33ce-9bab-11ec-b29c-00155d315008 (ServerSessionImpl.java [Thread-621 (ActiveMQ-remoting-threads-ActiveMQServerImpl::serverUUID=b2dcf2c4-9bab-11ec-b29c-00155d315008-415906701)])
* WARN  2022-03-05T10:39:26,395 Setup of JMS message listener invoker failed for destination 'dhis2.jobs.tracker.notifications' - trying to recover. Cause: Session is closed (DefaultMessageListenerContainer.java [org.springframework.jms.JmsListenerEndpointContainer#4-1])
* WARN  2022-03-05T10:39:26,801 com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@16cb0958 -- APPARENT DEADLOCK!!! Complete Status: 
	Managed Threads: 3
	Active Threads: 3
	Active Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@2789a43d
			on thread: C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#1
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@75afb5ff
			on thread: C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#2
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1398af81
			on thread: C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#0
	Pending Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5b0c4707
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@314eb817
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@39c6f38c
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@6ff4a792
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@5679e114
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@12b375cc
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1c67ae85
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@566552ac
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@4f0d7835
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@4619690d
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@672be4cd
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1b273ab6
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@3f4f2537
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@321362d2
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@aa6f5d1
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@28768cb9
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@6ed6721d
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@47fa98e7
		com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@7acff5b4
Pool thread stack traces:
	Thread[C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#1,5,main]
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:720)
	Thread[C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#2,5,main]
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:720)
	Thread[C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-HelperThread-#0,5,main]
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:720)
 (Slf4jMLog.java [C3P0PooledConnectionPoolManager[identityToken->1bqomxiannh9tsighm229|62d716e]-AdminTaskTimer])
* ERROR 2022-03-05T10:39:26,817 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-20])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,817 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-14])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,833 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-23])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,833 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-24])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,833 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-2])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,833 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-21])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-25])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-10])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-13])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-1])
java.lang.OutOfMemoryError: Java heap space
* ERROR 2022-03-05T10:39:26,880 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-11])
java.lang.OutOfMemoryError: Java heap space
* INFO  2022-03-05T10:39:27,333 Complete data set registration import done, total: 1, imported: 1, updated: 0, deleted: 0: 01:07:24.207 (Clock.java [https-jsse-nio-443-exec-51])
* ERROR 2022-03-05T10:39:27,333 Unexpected exception occurred invoking async method: public void org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(org.hisp.dhis.programrule.engine.StageCompletionEvaluationEvent) (SimpleAsyncUncaughtExceptionHandler.java [taskScheduler-5])
java.lang.OutOfMemoryError: Java heap space
	at org.hibernate.property.access.spi.GetterMethodImpl.get(GetterMethodImpl.java:42) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.tuple.entity.AbstractEntityTuplizer.getPropertyValues(AbstractEntityTuplizer.java:574) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.tuple.entity.PojoEntityTuplizer.getPropertyValues(PojoEntityTuplizer.java:154) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.persister.entity.AbstractEntityPersister.getPropertyValues(AbstractEntityPersister.java:5174) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.event.internal.DefaultFlushEntityEventListener.getValues(DefaultFlushEntityEventListener.java:207) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.event.internal.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:164) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.event.internal.AbstractFlushingEventListener$$Lambda$1147/0x00000017da9d7440.accept(Unknown Source) ~[?:?]
	at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:93) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.event.internal.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:230) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:93) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.internal.SessionImpl$$Lambda$1143/0x00000017da9d1040.accept(Unknown Source) ~[?:?]
	at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:93) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1362) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:453) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3212) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2380) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:447) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:183) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$300(JdbcResourceLocalTransactionCoordinatorImpl.java:40) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:281) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:101) ~[hibernate-core-5.4.28.Final.jar:5.4.28.Final]
	at org.springframework.orm.hibernate5.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:643) ~[spring-orm-5.3.6.jar:5.3.6]
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743) ~[spring-tx-5.3.6.jar:5.3.6]
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711) ~[spring-tx-5.3.6.jar:5.3.6]
	at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654) ~[spring-tx-5.3.6.jar:5.3.6]
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407) ~[spring-tx-5.3.6.jar:5.3.6]
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.6.jar:5.3.6]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.15.RELEASE.jar:5.2.15.RELEASE]
	at com.sun.proxy.$Proxy314.evaluateEventAndRunEffects(Unknown Source) ~[?:?]
	at org.hisp.dhis.programrule.engine.ProgramRuleEngineListener.onEventCompletion(ProgramRuleEngineListener.java:61) ~[dhis-service-program-rule-2.37.3.jar:?]

tuzoengelbert · 7 March 2022 09:36

Hello @Junaid_Mufti Can you specify -Xmx20g -Xms10g
and make sure you kill first the processid running java

Junaid_Mufti · 7 March 2022 11:51

Thank you. What effect will this have other than setting MAX size to 20GB and startup size to 10GB ? Just trying to understand.

tuzoengelbert · 11 March 2022 04:32

Hello @Junaid_Mufti here the point is that the allocation of Memory to JAVA should not exceed the total available memory, whatever allocation of memory should not exceed total available server memory

WaluQ · 23 August 2022 06:54

@janagha I tried both but still no data shows up in event reports, I tried to export the programs so that I can just reinstall but am getting the attached error. Not sure what could be causing this.

Gassim · 23 August 2022 14:23

@WaluQ
Please try again in a different/new browser/profile (preferably Guest mode) Do you still see the issues? If not then it’s a browser cache issue, but if you still see this issue, and still see it even after clearing the server cache using the Administration app then would you please share the Catalina.out log (without the sensitive info)?

Are you using a superuser account? If not does this also appear to all the main admin accounts? Please mention the dhis2 version number if you share the log.

Thanks!

WaluQ · 23 August 2022 14:43

I am still getting the same response and behavior. Attached in the catalina log. I can’t seem to find the better way to attached the log here. DHIS2 Version 2.38.1.1
catalina log.docx (18.7 KB)

Gassim · 23 August 2022 14:55

Thanks! What about the Java, tomcat, and postgresql versions?

WaluQ · 23 August 2022 15:40

Tomcat 8.5, Java 11 and Postgresql 14.

Gassim · 24 August 2022 09:27

Thank you @WaluQ! Based on the Catalina.out log, I found a similar issue reported in the CoP and there was a suggestion by community member and developer @tuzoengelbert , please have a look:

I am moving the posts to the post linked above because this new issue you posted is relevant to the memory leak issue.

Thank you!

WaluQ · 25 August 2022 16:55

I have read through the comments above but can’t seem to find a related solution. The server currently is running on 28BG Ram while the Java options are set at 512min and 2048 max. Would this contibute to the event not pulling results or not being able to export metadata?

Gassim · 29 August 2022 19:31

@WaluQ,
I’m asking for support and here are some suggestions I received from @hendrik:

“To establish a definite cause of the app crashing, it’d be good to check the response to the GET request to /api/37/dataSets?paging=false”

Please let us know what response are you getting. This could also happen by checking the Network tab in your DevTools.

Thanks!