Urgent: Analytics Tables Not Generating or Populating

Dear all,

HISP-SA requires urgent assistance please.

One of our provinces (Free State) is using DHIS2 for data capturing, but currently we cannot report on any of the data that has been captured because the analytics tables won’t generate and populate.

Lars determined that the reason for this is because some of our UIDs start with numbers (we imported them from the DHIS1.4 data file).

Excerpt from Gmail Chat:

Lars Helge Øverland

"the underlying problem here is that we use uids as column names, and that the uid starts with a number

in dhis2 all uids always start with an alpha character so that they become valid column names and xml element names

there is a small flaw in the dhis2 code - this should have been escaped

Crizelle Nel

We have changed the 1.4 data file conversion process to pull in our existing DHIS1.4 UIDs, since 1.4 is still currently our master system.

Is it an easy fix?

Lars Helge Øverland

ok

yes

but it could cause problems further down the line

Lars made a code change in DHIS2 which partially addressed the problem (the resource tables are now generating), but was not able to provide further support due to prior committments [Refer to http://apps.dhis2.org/ci/job/dhis-2.13/, #4]. The Tomcat logs and errors are attached and in the email below.

We are hoping someone is available and able to provide immediate assistance in finalizing this code fix as we urgently need to report on this data.

Any help will be much appreciated.

localhost_access_log.2013-11-26.txt (177 KB)

catalina.2013-11-26.log (8.81 KB)

localhost.2013-11-26.log (1.76 KB)

commons-daemon.2013-11-26.log (916 Bytes)

···


Regards,

Crizelle Nel

HISP_Logo_sm

Software Development and Information Systems

**Health Information Systems Program (HISP) - SA

**Email: crizellenel@gmail.com

Cell: +27 84 580 3342
Fax: 0866 509 502

Skype: crizelle_nel
Website: http://hispsa.org

---------- Forwarded message ----------
From: Crizelle Nel crizellenel@gmail.com

Date: Tue, Nov 26, 2013 at 7:21 AM
Subject: Fwd: Issue with Analytics tables
To: Lars Helge Øverland larshelge@gmail.com

Hi Lars,

Tomcat logs attached and and screen shot of my Tomcat window and DHIS2 screen below (I attached all the log files since my Tomcat window and my log files are not the same…don’t know why not).

Inline image 1

Inline image 2

Inline image 3

Inline image 4

As mentioned, the analytics tables are not generating. Seems like the same UID issue that prohibited the resource tables to generate.

Hi Crizelle,
The one log (catalina.out) which you did not append (screenshots only) is the one which might help to get to the bottom of this. It is not very clear from the screenshots. Could you post this someplace (pastebin.com?), or at least the part when the analytics process starts? This might help to diagnose exactly what is going on. Also, please let us know the version of DHIS2 and Postgres which you are using.

Best regards,
Jason

image

image

image

image

image

···

On Wed, Nov 27, 2013 at 2:52 PM, Crizelle Nel crizellenel@gmail.com wrote:

Dear all,

HISP-SA requires urgent assistance please.

One of our provinces (Free State) is using DHIS2 for data capturing, but currently we cannot report on any of the data that has been captured because the analytics tables won’t generate and populate.

Lars determined that the reason for this is because some of our UIDs start with numbers (we imported them from the DHIS1.4 data file).

Excerpt from Gmail Chat:

Lars Helge Øverland

"the underlying problem here is that we use uids as column names, and that the uid starts with a number

in dhis2 all uids always start with an alpha character so that they become valid column names and xml element names

there is a small flaw in the dhis2 code - this should have been escaped

Crizelle Nel

We have changed the 1.4 data file conversion process to pull in our existing DHIS1.4 UIDs, since 1.4 is still currently our master system.

Is it an easy fix?

Lars Helge Øverland

ok

yes

but it could cause problems further down the line

Lars made a code change in DHIS2 which partially addressed the problem (the resource tables are now generating), but was not able to provide further support due to prior committments [Refer to http://apps.dhis2.org/ci/job/dhis-2.13/, #4]. The Tomcat logs and errors are attached and in the email below.

We are hoping someone is available and able to provide immediate assistance in finalizing this code fix as we urgently need to report on this data.

Any help will be much appreciated.


Regards,

Crizelle Nel

Software Development and Information Systems

**Health Information Systems Program (HISP) - SA

**Email: crizellenel@gmail.com

Cell: +27 84 580 3342
Fax: 0866 509 502

Skype: crizelle_nel
Website: http://hispsa.org

---------- Forwarded message ----------
From: Crizelle Nel crizellenel@gmail.com

Date: Tue, Nov 26, 2013 at 7:21 AM
Subject: Fwd: Issue with Analytics tables
To: Lars Helge Øverland larshelge@gmail.com

Hi Lars,

Tomcat logs attached and and screen shot of my Tomcat window and DHIS2 screen below (I attached all the log files since my Tomcat window and my log files are not the same…don’t know why not).

As mentioned, the analytics tables are not generating. Seems like the same UID issue that prohibited the resource tables to generate.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi I can confirm that a bug slipped into r 13021 in analytics table generation. Fixing now.

Lars

image

image

image

image

image

···

On Wed, Nov 27, 2013 at 3:51 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Crizelle,
The one log (catalina.out) which you did not append (screenshots only) is the one which might help to get to the bottom of this. It is not very clear from the screenshots. Could you post this someplace (pastebin.com?), or at least the part when the analytics process starts? This might help to diagnose exactly what is going on. Also, please let us know the version of DHIS2 and Postgres which you are using.

Best regards,
Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Wed, Nov 27, 2013 at 2:52 PM, Crizelle Nel crizellenel@gmail.com wrote:

Dear all,

HISP-SA requires urgent assistance please.

One of our provinces (Free State) is using DHIS2 for data capturing, but currently we cannot report on any of the data that has been captured because the analytics tables won’t generate and populate.

Lars determined that the reason for this is because some of our UIDs start with numbers (we imported them from the DHIS1.4 data file).

Excerpt from Gmail Chat:

Lars Helge Øverland

"the underlying problem here is that we use uids as column names, and that the uid starts with a number

in dhis2 all uids always start with an alpha character so that they become valid column names and xml element names

there is a small flaw in the dhis2 code - this should have been escaped

Crizelle Nel

We have changed the 1.4 data file conversion process to pull in our existing DHIS1.4 UIDs, since 1.4 is still currently our master system.

Is it an easy fix?

Lars Helge Øverland

ok

yes

but it could cause problems further down the line

Lars made a code change in DHIS2 which partially addressed the problem (the resource tables are now generating), but was not able to provide further support due to prior committments [Refer to http://apps.dhis2.org/ci/job/dhis-2.13/, #4]. The Tomcat logs and errors are attached and in the email below.

We are hoping someone is available and able to provide immediate assistance in finalizing this code fix as we urgently need to report on this data.

Any help will be much appreciated.


Regards,

Crizelle Nel

Software Development and Information Systems

**Health Information Systems Program (HISP) - SA

**Email: crizellenel@gmail.com

Cell: +27 84 580 3342
Fax: 0866 509 502

Skype: crizelle_nel
Website: http://hispsa.org

---------- Forwarded message ----------
From: Crizelle Nel crizellenel@gmail.com

Date: Tue, Nov 26, 2013 at 7:21 AM
Subject: Fwd: Issue with Analytics tables
To: Lars Helge Øverland larshelge@gmail.com

Hi Lars,

Tomcat logs attached and and screen shot of my Tomcat window and DHIS2 screen below (I attached all the log files since my Tomcat window and my log files are not the same…don’t know why not).

As mentioned, the analytics tables are not generating. Seems like the same UID issue that prohibited the resource tables to generate.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

This bug has been fixed and back-ported to version 2.13.

It was a weakness which only surfaced when having UIDs in the system
starting with a numeric character, meaning they must have been created
outside DHIS and imported.

A consequence, related to upgrading to the latest 2.13 build, is that the
analytics tables must be re-generated for the system to work properly, so a
tip is up upgrade shortly before the scheduled analytics task is being run.

Lars