Analytics tables update process failed at Event tables

paleu256 · 11 December 2019 02:46

Dear All,
After update from 2.30 to 2.33.1, analytics tables update fails at stage of updating event tables.
analyticsfailed
If i skip generation of even and enrollment data, it succeeds.
I am also not able to view TEI dashboards (all i get is a blank page) with the following in browser console.
Failed to load resource: the server responded with a status of 500 ()

Below is my error log at the point at which the analytics table update fails:

*** INFO  2019-12-11 05:39:17,535 Starting update of type: EVENT, table name: 'analytics_event', processes: 3: 00:00:00.000 (Clock.java [taskScheduler-5])**
*** INFO  2019-12-11 05:39:17,536 Get tables using earliest: null, spatial support: true (JdbcEventAnalyticsTableManager.java [taskScheduler-5])**
*** ERROR 2019-12-11 05:39:17,553 Analytics table process failed: java.lang.NullPointerException**
**	at org.hisp.dhis.program.Program.lambda$getTrackedEntityAttributes$6(Program.java:285)**
**	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)**
**	at java.util.Iterator.forEachRemaining(Iterator.java:116)**
**	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)**
**	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)**
**	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)**
**	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)**
**	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)**
**	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)**
**	at org.hisp.dhis.program.Program.getTrackedEntityAttributes(Program.java:286)**
**	at org.hisp.dhis.program.Program.getNonConfidentialTrackedEntityAttributes(Program.java:295)**
**	at org.hisp.dhis.analytics.table.JdbcEventAnalyticsTableManager.getDimensionColumns(JdbcEventAnalyticsTableManager.java:353)**
**	at org.hisp.dhis.analytics.table.JdbcEventAnalyticsTableManager.getRegularAnalyticsTables(JdbcEventAnalyticsTableManager.java:163)**
**	at org.hisp.dhis.analytics.table.JdbcEventAnalyticsTableManager.getAnalyticsTables(JdbcEventAnalyticsTableManager.java:139)**
**	at sun.reflect.GeneratedMethodAccessor1371.invoke(Unknown Source)**
**	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)**
**	at java.lang.reflect.Method.invoke(Method.java:498)**
**	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)**
**	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)**
**	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)**
**	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)**
**	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)**
**	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)**
**	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)**
**	at com.sun.proxy.$Proxy202.getAnalyticsTables(Unknown Source)**
**	at org.hisp.dhis.analytics.table.DefaultAnalyticsTableService.update(DefaultAnalyticsTableService.java:140)**
**	at org.hisp.dhis.analytics.table.DefaultAnalyticsTableGenerator.generateTables(DefaultAnalyticsTableGenerator.java:138)**
**	at org.hisp.dhis.analytics.table.scheduling.AnalyticsTableJob.execute(AnalyticsTableJob.java:84)**
**	at org.hisp.dhis.scheduling.DefaultJobInstance.executeJob(DefaultJobInstance.java:157)**
**	at org.hisp.dhis.scheduling.DefaultJobInstance.execute(DefaultJobInstance.java:71)**
**	at org.hisp.dhis.scheduling.DefaultSchedulingManager.lambda$internalExecuteJobConfiguration$3(DefaultSchedulingManager.java:293)**
**	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)**
**	at java.util.concurrent.FutureTask.run(FutureTask.java:266)**
**	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)**
**	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)**
**	at java.util.concurrent.FutureTask.run(FutureTask.java:266)**
**	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)**
**	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)**
**	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)**
**	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)**
**	at java.lang.Thread.run(Thread.java:748)**
** (DefaultAnalyticsTableGenerator.java [taskScheduler-5])**
**java.lang.NullPointerException**
**	at org.hisp.dhis.program.Program.lambda$getTrackedEntityAttributes$6(Program.java:285)**
**	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)**
**	at java.util.Iterator.forEachRemaining(Iterator.java:116)**
**	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)**
**	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)**
**	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)**
**	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)**
**	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)**
**	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)**
**	at org.hisp.dhis.program.Program.getTrackedEntityAttributes(Program.java:286)**
**	at org.hisp.dhis.program.Program.getNonConfidentialTrackedEntityAttributes(Program.java:295)**
**	at org.hisp.dhis.analytics.table.JdbcEventAnalyticsTableManager.getDimensionColumns(JdbcEventAnalyticsTableManager.java:353)**
**	at org.hisp.dhis.analytics.table.JdbcEventAnalyticsTableManager.getRegularAnalyticsTables(JdbcEventAnalyticsTableManager.java:163)**
**	at org.hisp.dhis.analytics.table.JdbcEventAnalyticsTableManager.getAnalyticsTables(JdbcEventAnalyticsTableManager.java:139)**
**	at sun.reflect.GeneratedMethodAccessor1371.invoke(Unknown Source)**
**	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)**
**	at java.lang.reflect.Method.invoke(Method.java:498)**
**	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)**
**	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)**


**I am also unable to register new TEIs and fails with the following in log;**

**     INFO  2019-12-11 05:48:22,485 'paleu' create org.hisp.dhis.trackedentity.TrackedEntityInstance, uid: I9ep9Mj9Evy (AuditLogUtil.java [ajp-bio-8081-exec-162])**
*** INFO  2019-12-11 05:48:22,496 Import done: 00:00:00.000 (Clock.java [ajp-bio-8081-exec-162])**
*** INFO  2019-12-11 05:48:22,497 Import done: 00:00:00.000 (Clock.java [ajp-bio-8081-exec-162])**
*** INFO  2019-12-11 05:48:22,498 Import done: 00:00:00.000 (Clock.java [ajp-bio-8081-exec-162])**
*** INFO  2019-12-11 05:48:22,498 Import done: 00:00:00.000 (Clock.java [ajp-bio-8081-exec-162])**
**java.lang.NullPointerException**
**	at org.hisp.dhis.program.Program.lambda$getTrackedEntityAttributes$6(Program.java:285)**
**	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)**
**	at java.util.Iterator.forEachRemaining(Iterator.java:116)**
**	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)**
**	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)**
**	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)**
**	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)**
**	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)**
**	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)**
**	at org.hisp.dhis.program.Program.getTrackedEntityAttributes(Program.java:286)**
**	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)**
**	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1380)**
**	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)**
**	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)**
**	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)**
**	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)**
**	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)**
**	at org.hisp.dhis.trackedentity.DefaultTrackedEntityAttributeService.getAllUserReadableTrackedEntityAttributes(DefaultTrackedEntityAttributeService.java:288)**
**	at org.hisp.dhis.trackedentity.DefaultTrackedEntityAttributeService.getAllUserReadableTrackedEntityAttributes(DefaultTrackedEntityAttributeService.java:278)**
**	at org.hisp.dhis.trackedentity.DefaultTrackedEntityAttributeService.getAllUserReadableTrackedEntityAttributes(DefaultTrackedEntityAttributeService.java:268)**
**	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)**
**	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)**

Server specifications:
Version:2.33.1
Build revision:7ab6211
Build date:2019-11-27 12
Database version:PostgreSQL 9.6.15
Java version:1.8.0_151
OS:Ubuntu 16.04

How do i go about this error? Thanks

jomutsani · 11 December 2019 06:11

Hi @paleu256, tagging the @dhis2-analytics team to advise on this.

paleu256 · 11 December 2019 06:59

@Calle_Hedberg, @Abyot_Gizaw, @bridget @Scott any head-ups on this, where I can start looking at? Thanks

paleu256 · 19 December 2019 10:13

Hello Devs, is there anyone who has faced something similar?

Calle_Hedberg · 11 January 2020 11:07

@paleu256

Mmm - I’m pretty sure I’ve seen something like that error before. It’s a nullpointer exception, which most likely means that you have at least one record with a NULL in a compulsory field. OR you have a foreign key value where the constraint at some stage was removed and the lookup value subsequently deleted.

It can be really hard to find such things, but you could try to verify that the following tables

trackedentityinstance
trackedentityattributevalue
programinstance
programstageinstance
do not have any NULL “values” in key mandatory fields (I presume you have full access to the database).

For instance, make sure that fields like trackedentityattributeid, enrollmentdate, and executiondate have legal values for ALL records.

A few simple scripts with LEFT/RIGHT JOIN will tell you if any foreign key values are missing (and if yes, it means the related constraint is missing too).

Best of luck
Calle