Analytics Cron Job Failing

Support - Assistance technique
1

Hello Community,

We have an ongoing issue as shown in this thread here [FIXED] [Line Listing app - v100.0.0 / DHIS2v2.38] Error displaying data

We noticed in addition that the analytics CRON job has been failing. Every morning we now have to manually generate the analytics table

Below is the error message from 2 days ago

Click to view error message

Message: StatementCallback; bad SQL grammar [delete from “analytics_event_dergnr0f2d2” ax where ax.psi in (select psi.uid from programstageinstance psi inner join programinstance pi on psi.programinstanceid=pi.programinstanceid where pi.programid = 1223036 and psi.lastupdated >= ‘2022-07-16T14:08:28’ and psi.lastupdated < ‘2022-07-17T00:00:00’)]; nested exception is org.postgresql.util.PSQLException: ERROR: relation “analytics_event_dergnr0f2d2” does not exist Position: 13 Cause: org.postgresql.util.PSQLException: ERROR: relation “analytics_event_dergnr0f2d2” does not exist Position: 13 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:329) at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:315) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:291) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:286) at com.mchange.v2.c3p0.impl.NewProxyStatement.execute(NewProxyStatement.java:75) at org.springframework.jdbc.core.JdbcTemplate$1ExecuteStatementCallback.doInStatement(JdbcTemplate.java:422) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:381) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:431) at org.hisp.dhis.analytics.table.AbstractJdbcTableManager.invokeTimeAndLog(AbstractJdbcTableManager.java:527) at org.hisp.dhis.analytics.table.JdbcEventAnalyticsTableManager.removeUpdatedData(JdbcEventAnalyticsTableManager.java:314) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) at com.sun.proxy.$Proxy314.removeUpdatedData(Unknown Source) at org.hisp.dhis.analytics.table.DefaultAnalyticsTableService.update(DefaultAnalyticsTableService.java:163) at org.hisp.dhis.analytics.table.DefaultAnalyticsTableGenerator.generateTables(DefaultAnalyticsTableGenerator.java:112) at org.hisp.dhis.analytics.table.scheduling.AnalyticsTableJob.execute(AnalyticsTableJob.java:82) at org.hisp.dhis.scheduling.AbstractSchedulingManager.execute(AbstractSchedulingManager.java:277) at org.hisp.dhis.scheduling.AbstractSchedulingManager.execute(AbstractSchedulingManager.java:229) at org.hisp.dhis.scheduling.DefaultSchedulingManager.lambda$scheduleTask$3(DefaultSchedulingManager.java:129) at org.hisp.dhis.scheduling.DefaultSchedulingManager.lambda$runIfPossible$5(DefaultSchedulingManager.java:212) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

Please could you help us look into this?

We are running version 2.38.1.

Thank you.

Ifeanyi

1 Like
2

Hi @ifeanyiokoye

Tagging @netroms to check this please! We might need a Jira bug issue for this one.

From your other post, you are using 2.38.1. (adding this for reference)

Thank yoU!

1 Like
3

Thank you @Gassim

1 Like
4

Hello @ifeanyiokoye and @Gassim.

This seems to be a bug. It’s also reproducible in Play.
For more details, please see the Jira related to this issue: [DHIS2-13514] - Jira

We will investigate it during the next few days.

Thanks,
Maikel

2 Likes
5

Great. Thank you.

6

On this issue, I have a schedule job running for analytics at 1am daily. Data entry hasn’t happened on the system for a couple months so I was just doing some maintenance when I realized the job had failed for almost two months and I didn’t notice.

When I opened the job, it just said error and didn’t say what the error was. When I manually ran analytics it run successfully and the job now runs successfully daily since.

Is there a way to set up notifications in case of errors, because I might have noted sooner if was getting notifications for this.

1 Like
7

Note that the error stack trace above comes from an analytics query (e.g. sent from the dashboard or visualizer app) where the required analytics tables are not present.

If the analytics table generation process is failing, it would be useful to see the stack trace which is produced during the generation.

best,

Lars

1 Like
8

@Lars How do we generate the stack trace?

1 Like
9

@ifeanyiokoye, the stack trace appears in the logs (Catalina.out) after running the export.