Troubles running datamart

Support - Assistance technique
1

Some organisationunities does not have data on reportting. I am using DHIS 2.10 under CentOS 6.3 64bit.

The last log has this:

  • INFO 2013-05-14 07:54:38,130 Created indicator index: 0:23:21.391 (Clock.java [taskScheduler-1])

  • INFO 2013-05-14 07:54:38,223 Aggregated data export done: 0:23:21.485 (Clock.java [taskScheduler-1])

  • ERROR 2013-05-14 07:54:45,224 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.NullPointerException
at org.hibernate.event.internal.AbstractFlushingEventListener.prepareEntityFlushes(AbstractFlushingEventListener.java:148)
at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:88)

at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:55)
at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1186)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1615)

at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:374)
at org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImpl.java:396)
at org.hisp.dhis.hibernate.HibernateGenericStore.getObject(HibernateGenericStore.java:178)

at org.hisp.dhis.hibernate.HibernateGenericStore.getByName(HibernateGenericStore.java:238)
at org.hisp.dhis.setting.DefaultSystemSettingManager.getSystemSetting(DefaultSystemSettingManager.java:99)
at sun.reflect.GeneratedMethodAccessor164.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy78.getSystemSetting(Unknown Source)
at org.hisp.dhis.datamart.engine.DefaultDataMartEngine.export(DefaultDataMartEngine.java:449)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)

at org.amplecode.quick.StatementInterceptor.intercept(StatementInterceptor.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy104.export(Unknown Source)
at org.hisp.dhis.system.scheduling.DataMartTask.run(DataMartTask.java:163)

at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

Can someone help?

Orvalho

2

Dear Orvalho,

DHIS2 Datamart sometim sometimes fails to run due to Invalid Data submitted during Reporting e.g. if it is expected a Number and either Text or just space or NULL (java.lang.NullPointerException) is submitted, It fails to run. This has been sorted in the latest version. Please try Upgrading your DHIS2 to the latest version and Try running again. The latest version will skip invalid data values and you can track it from the Log file for all the Ignored data values …

Hopefully this helps

Stephen

···

On Tue, May 14, 2013 at 8:46 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Some organisationunities does not have data on reportting. I am using DHIS 2.10 under CentOS 6.3 64bit.

The last log has this:

  • INFO 2013-05-14 07:54:38,130 Created indicator index: 0:23:21.391 (Clock.java [taskScheduler-1])

  • INFO 2013-05-14 07:54:38,223 Aggregated data export done: 0:23:21.485 (Clock.java [taskScheduler-1])

  • ERROR 2013-05-14 07:54:45,224 Unexpected error occurred in scheduled task. (TaskUtils.java [taskScheduler-1])

java.lang.NullPointerException
at org.hibernate.event.internal.AbstractFlushingEventListener.prepareEntityFlushes(AbstractFlushingEventListener.java:148)
at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:88)

at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:55)
at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1186)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1615)


at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:374)
at org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImpl.java:396)
at org.hisp.dhis.hibernate.HibernateGenericStore.getObject(HibernateGenericStore.java:178)


at org.hisp.dhis.hibernate.HibernateGenericStore.getByName(HibernateGenericStore.java:238)
at org.hisp.dhis.setting.DefaultSystemSettingManager.getSystemSetting(DefaultSystemSettingManager.java:99)
at sun.reflect.GeneratedMethodAccessor164.invoke(Unknown Source)


at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)


at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)


at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)


at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy78.getSystemSetting(Unknown Source)
at org.hisp.dhis.datamart.engine.DefaultDataMartEngine.export(DefaultDataMartEngine.java:449)


at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)


at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)


at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)


at org.amplecode.quick.StatementInterceptor.intercept(StatementInterceptor.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)


at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)


at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)


at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)


at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy104.export(Unknown Source)
at org.hisp.dhis.system.scheduling.DataMartTask.run(DataMartTask.java:163)


at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)


at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)


at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

Can someone help?

Orvalho

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

–

You can do anything, but not everything.

3

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don't think you would these issues.

The values are probably set to be text or so.

···

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----
From: Ocaya Stephen <stephocay@gmail.com>
Sender: "Dhis2-users"
  <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>
Date: Tue, 14 May 2013 10:22:45
To: Orvalho Augusto<orvaquim@gmail.com>
Cc: dhis1-users<dhis2-users@lists.launchpad.net>; DHIS 2 developers<dhis2-devs@lists.launchpad.net>
Subject: Re: [Dhis2-users] Troubles running datamart

_______________________________________________
Mailing list: DHIS 2 Users in Launchpad
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : DHIS 2 Users in Launchpad
More help : ListHelp - Launchpad Help

4

Dears!

Thank you for the answers.

I have no text on value field according to this

dhis2=# select * from datavalue where value is NULL;

dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup

···

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

5

What are the properties of the Fields for the Data Elements used for reporting?

Could you try manually generating the datamart and send us the complete LOG from where the error start upto where you can see Caused by: …

That kind of gives a clear picture of what is causing the problem
Waiting …

···

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

6

Thank you!

The dataset report does not bring any data that I see on dataentry. For other reports the fields are numeric.

For the datamart. Pardon my ignorance. How do I run it mannualy?

Orvalho

···

On Tue, May 14, 2013 at 9:36 AM, Ocaya Stephen stephocay@gmail.com wrote:

What are the properties of the Fields for the Data Elements used for reporting?

Could you try manually generating the datamart and send us the complete LOG from where the error start upto where you can see Caused by: …

That kind of gives a clear picture of what is causing the problem
Waiting …

On Tue, May 14, 2013 at 10:33 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Dears!

Thank you for the answers.

I have no text on value field according to this

dhis2=# select * from datavalue where value is NULL;

dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) = ‘’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) ~ ‘[A-Za-z]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) !~ ‘[0-9]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

7

To run DataMart Manually (U need to have the Right i guess),

Analytics and Data Mart

  • Analytics and Data Mart (In version 2.11 Hopefully 2.10 also)
    But idealy it should be under Reports then on the left pane, select Analytics and Data Mart

There, You can select the periods, and other parameters …

The hit Start Export

···

On Tue, May 14, 2013 at 9:36 AM, Ocaya Stephen stephocay@gmail.com wrote:

What are the properties of the Fields for the Data Elements used for reporting?

Could you try manually generating the datamart and send us the complete LOG from where the error start upto where you can see Caused by: …

That kind of gives a clear picture of what is causing the problem
Waiting …

On Tue, May 14, 2013 at 10:33 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Dears!

Thank you for the answers.

I have no text on value field according to this

dhis2=# select * from datavalue where value is NULL;

dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) = ‘’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) ~ ‘[A-Za-z]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) !~ ‘[0-9]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

8

To run the datamart manually, you go to Services, Reports and to the left you see Datamart for version 2.10 or Analytic Datamart for version 2.11.

Click on it, and make the fields as you desire then period and run

Dayo

···

On Tue, May 14, 2013 at 9:36 AM, Ocaya Stephen stephocay@gmail.com wrote:

What are the properties of the Fields for the Data Elements used for reporting?

Could you try manually generating the datamart and send us the complete LOG from where the error start upto where you can see Caused by: …

That kind of gives a clear picture of what is causing the problem
Waiting …

On Tue, May 14, 2013 at 10:33 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Dears!

Thank you for the answers.

I have no text on value field according to this

dhis2=# select * from datavalue where value is NULL;

dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) = ‘’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) ~ ‘[A-Za-z]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) !~ ‘[0-9]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

9

OK!

The log I gave is after I start datamart the same way you describe. I was not aware that it is manual way :slight_smile:

The catalinaout.txt is attached… since the begining of the datamart. How I wish I have zipped that file!

I saw some invalid indicators there but those have been there before these symptoms.

Orvalho

partialcatalinaout.txt (1.37 MB)

···

On Tue, May 14, 2013 at 9:54 AM, Ocaya Stephen stephocay@gmail.com wrote:

To run DataMart Manually (U need to have the Right i guess),

Analytics and Data Mart

  • Analytics and Data Mart (In version 2.11 Hopefully 2.10 also)
    But idealy it should be under Reports then on the left pane, select Analytics and Data Mart

There, You can select the periods, and other parameters …

The hit Start Export

On Tue, May 14, 2013 at 10:44 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

The dataset report does not bring any data that I see on dataentry. For other reports the fields are numeric.

For the datamart. Pardon my ignorance. How do I run it mannualy?

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:36 AM, Ocaya Stephen stephocay@gmail.com wrote:

What are the properties of the Fields for the Data Elements used for reporting?

Could you try manually generating the datamart and send us the complete LOG from where the error start upto where you can see Caused by: …

That kind of gives a clear picture of what is causing the problem
Waiting …

On Tue, May 14, 2013 at 10:33 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Dears!

Thank you for the answers.

I have no text on value field according to this

dhis2=# select * from datavalue where value is NULL;

dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) = ‘’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) ~ ‘[A-Za-z]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) !~ ‘[0-9]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

10

Did the manual generation produce any data/reports ??

Looking at the Log file, I looks rather crowded and Setting Tomcat LOG LEVELS to INFO, DEBUG (I am not sure how now) only would be great so that it skips out all the logs like Hibernate:…

Could also try to rectify the indicators(I am not sure whether it also affects DATAMART generation). I did not see the cause of the error in the log.

To View Log on the Fly, you could

tail -f /var/logs/tomcat7/catalina.out (if the log file is in /var/logs/tomcat7 path … and the log file is catalina.out

Thanks

···

On Tue, May 14, 2013 at 9:54 AM, Ocaya Stephen stephocay@gmail.com wrote:

To run DataMart Manually (U need to have the Right i guess),

Analytics and Data Mart

  • Analytics and Data Mart (In version 2.11 Hopefully 2.10 also)
    But idealy it should be under Reports then on the left pane, select Analytics and Data Mart

There, You can select the periods, and other parameters …

The hit Start Export

On Tue, May 14, 2013 at 10:44 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

The dataset report does not bring any data that I see on dataentry. For other reports the fields are numeric.

For the datamart. Pardon my ignorance. How do I run it mannualy?

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:36 AM, Ocaya Stephen stephocay@gmail.com wrote:

What are the properties of the Fields for the Data Elements used for reporting?

Could you try manually generating the datamart and send us the complete LOG from where the error start upto where you can see Caused by: …

That kind of gives a clear picture of what is causing the problem
Waiting …

On Tue, May 14, 2013 at 10:33 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Dears!

Thank you for the answers.

I have no text on value field according to this

dhis2=# select * from datavalue where value is NULL;

dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) = ‘’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) ~ ‘[A-Za-z]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) !~ ‘[0-9]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

11

Thank you!

Yes those logs seemed to be clean.

I can not run datamart now because it is a modest machine and there is a lot of users.

I will try to set logs to DEBUG level.

By the way tail, is a nice tool but when one expects huge logs like that is better to have a new log file… and use less, grep etc; And my connection to the server keeps failing a lot of times.

Orvalho

···

On Tue, May 14, 2013 at 11:09 AM, Ocaya Stephen stephocay@gmail.com wrote:

Did the manual generation produce any data/reports ??

Looking at the Log file, I looks rather crowded and Setting Tomcat LOG LEVELS to INFO, DEBUG (I am not sure how now) only would be great so that it skips out all the logs like Hibernate:…

Could also try to rectify the indicators(I am not sure whether it also affects DATAMART generation). I did not see the cause of the error in the log.

To View Log on the Fly, you could

tail -f /var/logs/tomcat7/catalina.out (if the log file is in /var/logs/tomcat7 path … and the log file is catalina.out

Thanks

On Tue, May 14, 2013 at 11:18 AM, Orvalho Augusto orvaquim@gmail.com wrote:

OK!

The log I gave is after I start datamart the same way you describe. I was not aware that it is manual way :slight_smile:

The catalinaout.txt is attached… since the begining of the datamart. How I wish I have zipped that file!

I saw some invalid indicators there but those have been there before these symptoms.

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:54 AM, Ocaya Stephen stephocay@gmail.com wrote:

To run DataMart Manually (U need to have the Right i guess),

Analytics and Data Mart

  • Analytics and Data Mart (In version 2.11 Hopefully 2.10 also)
    But idealy it should be under Reports then on the left pane, select Analytics and Data Mart

There, You can select the periods, and other parameters …

The hit Start Export

On Tue, May 14, 2013 at 10:44 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

The dataset report does not bring any data that I see on dataentry. For other reports the fields are numeric.

For the datamart. Pardon my ignorance. How do I run it mannualy?

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:36 AM, Ocaya Stephen stephocay@gmail.com wrote:

What are the properties of the Fields for the Data Elements used for reporting?

Could you try manually generating the datamart and send us the complete LOG from where the error start upto where you can see Caused by: …

That kind of gives a clear picture of what is causing the problem
Waiting …

On Tue, May 14, 2013 at 10:33 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Dears!

Thank you for the answers.

I have no text on value field according to this

dhis2=# select * from datavalue where value is NULL;

dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) = ‘’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) ~ ‘[A-Za-z]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) !~ ‘[0-9]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

12

One thing:

Couldn’t negative numbers trouble datamart?

I do not know how people here count negatively!

Orvalho

···

On Tue, May 14, 2013 at 11:34 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

Yes those logs seemed to be clean.

I can not run datamart now because it is a modest machine and there is a lot of users.

I will try to set logs to DEBUG level.

By the way tail, is a nice tool but when one expects huge logs like that is better to have a new log file… and use less, grep etc; And my connection to the server keeps failing a lot of times.

Orvalho

On Tue, May 14, 2013 at 11:09 AM, Ocaya Stephen stephocay@gmail.com wrote:

Did the manual generation produce any data/reports ??

Looking at the Log file, I looks rather crowded and Setting Tomcat LOG LEVELS to INFO, DEBUG (I am not sure how now) only would be great so that it skips out all the logs like Hibernate:…

Could also try to rectify the indicators(I am not sure whether it also affects DATAMART generation). I did not see the cause of the error in the log.

To View Log on the Fly, you could

tail -f /var/logs/tomcat7/catalina.out (if the log file is in /var/logs/tomcat7 path … and the log file is catalina.out

Thanks

On Tue, May 14, 2013 at 11:18 AM, Orvalho Augusto orvaquim@gmail.com wrote:

OK!

The log I gave is after I start datamart the same way you describe. I was not aware that it is manual way :slight_smile:

The catalinaout.txt is attached… since the begining of the datamart. How I wish I have zipped that file!

I saw some invalid indicators there but those have been there before these symptoms.

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:54 AM, Ocaya Stephen stephocay@gmail.com wrote:

To run DataMart Manually (U need to have the Right i guess),

Analytics and Data Mart

  • Analytics and Data Mart (In version 2.11 Hopefully 2.10 also)
    But idealy it should be under Reports then on the left pane, select Analytics and Data Mart

There, You can select the periods, and other parameters …

The hit Start Export

On Tue, May 14, 2013 at 10:44 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

The dataset report does not bring any data that I see on dataentry. For other reports the fields are numeric.

For the datamart. Pardon my ignorance. How do I run it mannualy?

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:36 AM, Ocaya Stephen stephocay@gmail.com wrote:

What are the properties of the Fields for the Data Elements used for reporting?

Could you try manually generating the datamart and send us the complete LOG from where the error start upto where you can see Caused by: …

That kind of gives a clear picture of what is causing the problem
Waiting …

On Tue, May 14, 2013 at 10:33 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Dears!

Thank you for the answers.

I have no text on value field according to this

dhis2=# select * from datavalue where value is NULL;

dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) = ‘’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) ~ ‘[A-Za-z]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) !~ ‘[0-9]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

13

There is not suppose to be negative counting though. That is why it is very necessary to have validation rules in place.

···

On Tue, May 14, 2013 at 11:34 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

Yes those logs seemed to be clean.

I can not run datamart now because it is a modest machine and there is a lot of users.

I will try to set logs to DEBUG level.

By the way tail, is a nice tool but when one expects huge logs like that is better to have a new log file… and use less, grep etc; And my connection to the server keeps failing a lot of times.

Orvalho

On Tue, May 14, 2013 at 11:09 AM, Ocaya Stephen stephocay@gmail.com wrote:

Did the manual generation produce any data/reports ??

Looking at the Log file, I looks rather crowded and Setting Tomcat LOG LEVELS to INFO, DEBUG (I am not sure how now) only would be great so that it skips out all the logs like Hibernate:…

Could also try to rectify the indicators(I am not sure whether it also affects DATAMART generation). I did not see the cause of the error in the log.

To View Log on the Fly, you could

tail -f /var/logs/tomcat7/catalina.out (if the log file is in /var/logs/tomcat7 path … and the log file is catalina.out

Thanks

On Tue, May 14, 2013 at 11:18 AM, Orvalho Augusto orvaquim@gmail.com wrote:

OK!

The log I gave is after I start datamart the same way you describe. I was not aware that it is manual way :slight_smile:

The catalinaout.txt is attached… since the begining of the datamart. How I wish I have zipped that file!

I saw some invalid indicators there but those have been there before these symptoms.

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:54 AM, Ocaya Stephen stephocay@gmail.com wrote:

To run DataMart Manually (U need to have the Right i guess),

Analytics and Data Mart

  • Analytics and Data Mart (In version 2.11 Hopefully 2.10 also)
    But idealy it should be under Reports then on the left pane, select Analytics and Data Mart

There, You can select the periods, and other parameters …

The hit Start Export

On Tue, May 14, 2013 at 10:44 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

The dataset report does not bring any data that I see on dataentry. For other reports the fields are numeric.

For the datamart. Pardon my ignorance. How do I run it mannualy?

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:36 AM, Ocaya Stephen stephocay@gmail.com wrote:

What are the properties of the Fields for the Data Elements used for reporting?

Could you try manually generating the datamart and send us the complete LOG from where the error start upto where you can see Caused by: …

That kind of gives a clear picture of what is causing the problem
Waiting …

On Tue, May 14, 2013 at 10:33 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Dears!

Thank you for the answers.

I have no text on value field according to this

dhis2=# select * from datavalue where value is NULL;

dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) = ‘’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) ~ ‘[A-Za-z]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) !~ ‘[0-9]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

14

Lord forgive me for my sins.

These criminal negatives come from a legacy system. They have been migrated… I am in the process of migrating them actually:

308527 records < 1

2275573 records are zeros!

1096806 records > 0

I will report this to the owners of the data.

Thank you

Orvalho

···

On Tue, May 14, 2013 at 2:35 PM, deemoyes@yahoo.com wrote:

There is not suppose to be negative counting though. That is why it is very necessary to have validation rules in place.
Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

From: Orvalho Augusto orvaquim@gmail.com

Date: Tue, 14 May 2013 12:49:39 +0200

To: Ocaya Stephenstephocay@gmail.com

Cc: Dayo Adeyomoyedeemoyes@yahoo.com; Dhis2-usersdhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net; dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

One thing:

Couldn’t negative numbers trouble datamart?

I do not know how people here count negatively!

Orvalho

On Tue, May 14, 2013 at 11:34 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

Yes those logs seemed to be clean.

I can not run datamart now because it is a modest machine and there is a lot of users.

I will try to set logs to DEBUG level.

By the way tail, is a nice tool but when one expects huge logs like that is better to have a new log file… and use less, grep etc; And my connection to the server keeps failing a lot of times.

Orvalho

On Tue, May 14, 2013 at 11:09 AM, Ocaya Stephen stephocay@gmail.com wrote:

Did the manual generation produce any data/reports ??

Looking at the Log file, I looks rather crowded and Setting Tomcat LOG LEVELS to INFO, DEBUG (I am not sure how now) only would be great so that it skips out all the logs like Hibernate:…

Could also try to rectify the indicators(I am not sure whether it also affects DATAMART generation). I did not see the cause of the error in the log.

To View Log on the Fly, you could

tail -f /var/logs/tomcat7/catalina.out (if the log file is in /var/logs/tomcat7 path … and the log file is catalina.out

Thanks

On Tue, May 14, 2013 at 11:18 AM, Orvalho Augusto orvaquim@gmail.com wrote:

OK!

The log I gave is after I start datamart the same way you describe. I was not aware that it is manual way :slight_smile:

The catalinaout.txt is attached… since the begining of the datamart. How I wish I have zipped that file!

I saw some invalid indicators there but those have been there before these symptoms.

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:54 AM, Ocaya Stephen stephocay@gmail.com wrote:

To run DataMart Manually (U need to have the Right i guess),

Analytics and Data Mart

  • Analytics and Data Mart (In version 2.11 Hopefully 2.10 also)
    But idealy it should be under Reports then on the left pane, select Analytics and Data Mart

There, You can select the periods, and other parameters …

The hit Start Export

On Tue, May 14, 2013 at 10:44 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

The dataset report does not bring any data that I see on dataentry. For other reports the fields are numeric.

For the datamart. Pardon my ignorance. How do I run it mannualy?

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:36 AM, Ocaya Stephen stephocay@gmail.com wrote:

What are the properties of the Fields for the Data Elements used for reporting?

Could you try manually generating the datamart and send us the complete LOG from where the error start upto where you can see Caused by: …

That kind of gives a clear picture of what is causing the problem
Waiting …

On Tue, May 14, 2013 at 10:33 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Dears!

Thank you for the answers.

I have no text on value field according to this

dhis2=# select * from datavalue where value is NULL;

dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) = ‘’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) ~ ‘[A-Za-z]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) !~ ‘[0-9]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

15

Hi Orvalho,

I am working on a fix, in the meantime you can provide the following medicine as SQL:

delete from systemsetting where name=‘orgUnitGroupSetAggregationLevel’;

Then go to data admin → cache and clear cache.

regards,

Lars

···

On Tue, May 14, 2013 at 2:49 PM, Orvalho Augusto orvaquim@gmail.com wrote:

Lord forgive me for my sins.

These criminal negatives come from a legacy system. They have been migrated… I am in the process of migrating them actually:

308527 records < 1

2275573 records are zeros!

1096806 records > 0

I will report this to the owners of the data.

Thank you

Orvalho

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Tue, May 14, 2013 at 2:35 PM, deemoyes@yahoo.com wrote:

There is not suppose to be negative counting though. That is why it is very necessary to have validation rules in place.
Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

From: Orvalho Augusto orvaquim@gmail.com

Date: Tue, 14 May 2013 12:49:39 +0200

To: Ocaya Stephenstephocay@gmail.com

Cc: Dayo Adeyomoyedeemoyes@yahoo.com; Dhis2-usersdhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net; dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

One thing:

Couldn’t negative numbers trouble datamart?

I do not know how people here count negatively!

Orvalho

On Tue, May 14, 2013 at 11:34 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

Yes those logs seemed to be clean.

I can not run datamart now because it is a modest machine and there is a lot of users.

I will try to set logs to DEBUG level.

By the way tail, is a nice tool but when one expects huge logs like that is better to have a new log file… and use less, grep etc; And my connection to the server keeps failing a lot of times.

Orvalho

On Tue, May 14, 2013 at 11:09 AM, Ocaya Stephen stephocay@gmail.com wrote:

Did the manual generation produce any data/reports ??

Looking at the Log file, I looks rather crowded and Setting Tomcat LOG LEVELS to INFO, DEBUG (I am not sure how now) only would be great so that it skips out all the logs like Hibernate:…

Could also try to rectify the indicators(I am not sure whether it also affects DATAMART generation). I did not see the cause of the error in the log.

To View Log on the Fly, you could

tail -f /var/logs/tomcat7/catalina.out (if the log file is in /var/logs/tomcat7 path … and the log file is catalina.out

Thanks

On Tue, May 14, 2013 at 11:18 AM, Orvalho Augusto orvaquim@gmail.com wrote:

OK!

The log I gave is after I start datamart the same way you describe. I was not aware that it is manual way :slight_smile:

The catalinaout.txt is attached… since the begining of the datamart. How I wish I have zipped that file!

I saw some invalid indicators there but those have been there before these symptoms.

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:54 AM, Ocaya Stephen stephocay@gmail.com wrote:

To run DataMart Manually (U need to have the Right i guess),

Analytics and Data Mart

  • Analytics and Data Mart (In version 2.11 Hopefully 2.10 also)
    But idealy it should be under Reports then on the left pane, select Analytics and Data Mart

There, You can select the periods, and other parameters …

The hit Start Export

On Tue, May 14, 2013 at 10:44 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

The dataset report does not bring any data that I see on dataentry. For other reports the fields are numeric.

For the datamart. Pardon my ignorance. How do I run it mannualy?

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:36 AM, Ocaya Stephen stephocay@gmail.com wrote:

What are the properties of the Fields for the Data Elements used for reporting?

Could you try manually generating the datamart and send us the complete LOG from where the error start upto where you can see Caused by: …

That kind of gives a clear picture of what is causing the problem
Waiting …

On Tue, May 14, 2013 at 10:33 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Dears!

Thank you for the answers.

I have no text on value field according to this

dhis2=# select * from datavalue where value is NULL;

dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) = ‘’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) ~ ‘[A-Za-z]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) !~ ‘[0-9]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

16

Thank you every one!

I was granted permission to remove zeros and negative “energy” from the dataset. And applied the Lars’s medicine and Mr Datamart got back to our lives again:

  • INFO 2013-05-14 20:46:08,118 Completeness export task done (AbstractDataSetCompletenessService.java [SimpleAsyncTaskExecutor-14])

  • INFO 2013-05-14 20:46:08,708 Completeness export task done (AbstractDataSetCompletenessService.java [SimpleAsyncTaskExecutor-15])

  • INFO 2013-05-14 20:46:08,952 Completeness export task done (AbstractDataSetCompletenessService.java [SimpleAsyncTaskExecutor-13])

  • INFO 2013-05-14 20:46:11,194 Completeness export task done (AbstractDataSetCompletenessService.java [SimpleAsyncTaskExecutor-16])

  • INFO 2013-05-14 20:46:11,391 Created index: 0:00:11.515 (Clock.java [taskScheduler-1])

  • INFO 2013-05-14 20:46:11,392 Completeness export process completed: 0:00:11.516 (Clock.java [taskScheduler-1])

  • INFO 2013-05-14 20:46:11,392 [Level: INFO, category: DATAMART, time: Tue May 14 20:46:11 UTC 2013, message: Completeness process completed] (InMemoryNotifier.java [taskScheduler-1])

Thank you!

Orvalho

···

On Tue, May 14, 2013 at 4:34 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Orvalho,

I am working on a fix, in the meantime you can provide the following medicine as SQL:

delete from systemsetting where name=‘orgUnitGroupSetAggregationLevel’;

Then go to data admin → cache and clear cache.

regards,

Lars

On Tue, May 14, 2013 at 2:49 PM, Orvalho Augusto orvaquim@gmail.com wrote:

Lord forgive me for my sins.

These criminal negatives come from a legacy system. They have been migrated… I am in the process of migrating them actually:

308527 records < 1

2275573 records are zeros!

1096806 records > 0

I will report this to the owners of the data.

Thank you

Orvalho

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Tue, May 14, 2013 at 2:35 PM, deemoyes@yahoo.com wrote:

There is not suppose to be negative counting though. That is why it is very necessary to have validation rules in place.
Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

From: Orvalho Augusto orvaquim@gmail.com

Date: Tue, 14 May 2013 12:49:39 +0200

To: Ocaya Stephenstephocay@gmail.com

Cc: Dayo Adeyomoyedeemoyes@yahoo.com; Dhis2-usersdhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net; dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

One thing:

Couldn’t negative numbers trouble datamart?

I do not know how people here count negatively!

Orvalho

On Tue, May 14, 2013 at 11:34 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

Yes those logs seemed to be clean.

I can not run datamart now because it is a modest machine and there is a lot of users.

I will try to set logs to DEBUG level.

By the way tail, is a nice tool but when one expects huge logs like that is better to have a new log file… and use less, grep etc; And my connection to the server keeps failing a lot of times.

Orvalho

On Tue, May 14, 2013 at 11:09 AM, Ocaya Stephen stephocay@gmail.com wrote:

Did the manual generation produce any data/reports ??

Looking at the Log file, I looks rather crowded and Setting Tomcat LOG LEVELS to INFO, DEBUG (I am not sure how now) only would be great so that it skips out all the logs like Hibernate:…

Could also try to rectify the indicators(I am not sure whether it also affects DATAMART generation). I did not see the cause of the error in the log.

To View Log on the Fly, you could

tail -f /var/logs/tomcat7/catalina.out (if the log file is in /var/logs/tomcat7 path … and the log file is catalina.out

Thanks

On Tue, May 14, 2013 at 11:18 AM, Orvalho Augusto orvaquim@gmail.com wrote:

OK!

The log I gave is after I start datamart the same way you describe. I was not aware that it is manual way :slight_smile:

The catalinaout.txt is attached… since the begining of the datamart. How I wish I have zipped that file!

I saw some invalid indicators there but those have been there before these symptoms.

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:54 AM, Ocaya Stephen stephocay@gmail.com wrote:

To run DataMart Manually (U need to have the Right i guess),

Analytics and Data Mart

  • Analytics and Data Mart (In version 2.11 Hopefully 2.10 also)
    But idealy it should be under Reports then on the left pane, select Analytics and Data Mart

There, You can select the periods, and other parameters …

The hit Start Export

On Tue, May 14, 2013 at 10:44 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Thank you!

The dataset report does not bring any data that I see on dataentry. For other reports the fields are numeric.

For the datamart. Pardon my ignorance. How do I run it mannualy?

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:36 AM, Ocaya Stephen stephocay@gmail.com wrote:

What are the properties of the Fields for the Data Elements used for reporting?

Could you try manually generating the datamart and send us the complete LOG from where the error start upto where you can see Caused by: …

That kind of gives a clear picture of what is causing the problem
Waiting …

On Tue, May 14, 2013 at 10:33 AM, Orvalho Augusto orvaquim@gmail.com wrote:

Dears!

Thank you for the answers.

I have no text on value field according to this

dhis2=# select * from datavalue where value is NULL;

dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) = ‘’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) ~ ‘[A-Za-z]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

dhis2=# select * from datavalue where trim(value) !~ ‘[0-9]’;
dataelementid | periodid | sourceid | categoryoptioncomboid | value | storedby | lastupdated | comment | followup
---------------±---------±---------±----------------------±------±---------±------------±--------±---------

(0 rows)

Orvalho

–

You can do anything, but not everything.

On Tue, May 14, 2013 at 9:30 AM, deemoyes@yahoo.com wrote:

I think the value type for the data elements are not properly set because if you set the value to be α number and an integer and also specify to or not to allow zeros plus your validation rules then I don’t think you would these issues.

The values are probably set to be text or so.

Sent from my BlackBerry® smartphone from Etisalat. Enjoy high speed mobile broadband on our easyblaze and plans for BlackBerry. Visit www.etisalat.com.ng for details.

-----Original Message-----

From: Ocaya Stephen stephocay@gmail.com

Sender: “Dhis2-users”

    <dhis2-users-bounces+deemoyes=yahoo.com@lists.launchpad.net>

Date: Tue, 14 May 2013 10:22:45

To: Orvalho Augustoorvaquim@gmail.com

Cc: dhis1-usersdhis2-users@lists.launchpad.net; DHIS 2 developersdhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-users] Troubles running datamart

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp