Data not displaying in Pivot tables and Data Set reports

DHIS2, version 2.18

We have recently been experiencing a problem in getting data to appear in Pivot Tables and Data Set Reports.

I can verify that data exists on the data entry screen. Furthermore, all users, user roles, data elements, categories, indicators, data sets and Org Units exist. However, after successfully running Analytics (no errors in the front end, at least) I cannot produce a Pivot Table with any results for a simple query I am certain should return results. Furthermore, I cannot pull up a Data Set report for an Org Unit/Reporting Period for which there is data on the corresponding Data Entry screen. (The form displays with no data in the cells.)

We have a test version of this DHIS instance with data that is about 6 weeks old. I am able to run the simple Pivot Table and Data Set Reports without issue – they produce the expected data.

What might be wrong with our live production instance? What tables can I query on the backend to determine what’s in the Analytics tables? Could it be due to User Access changes (even though the same user sees the data in Data Entry)? We had to reboot the production server 4 days ago, but I believe the issue with data not appearing in the pivot tables was happening before that.

Any help would be appreciated. Thank you.

···

Laura E. Lincks
Database Manager/Developer
ICAP - Columbia University
Mailman School of Public Health
60 Haven Ave, Floor B1
New York, NY 10032
Tel: 212 304 7132

As a follow up, I notice that all the resource tables (preceded by an underscore) and all analytics_ and analytics_completeness tables contain 0 columns. I recreated resource tables and reran Analytics, both without error (from the front end). I’m very confused.

···

On Tue, Aug 18, 2015 at 5:06 PM, Laura E. Lincks laura.lincks@icap.columbia.edu wrote:

DHIS2, version 2.18

We have recently been experiencing a problem in getting data to appear in Pivot Tables and Data Set Reports.

I can verify that data exists on the data entry screen. Furthermore, all users, user roles, data elements, categories, indicators, data sets and Org Units exist. However, after successfully running Analytics (no errors in the front end, at least) I cannot produce a Pivot Table with any results for a simple query I am certain should return results. Furthermore, I cannot pull up a Data Set report for an Org Unit/Reporting Period for which there is data on the corresponding Data Entry screen. (The form displays with no data in the cells.)

We have a test version of this DHIS instance with data that is about 6 weeks old. I am able to run the simple Pivot Table and Data Set Reports without issue – they produce the expected data.

What might be wrong with our live production instance? What tables can I query on the backend to determine what’s in the Analytics tables? Could it be due to User Access changes (even though the same user sees the data in Data Entry)? We had to reboot the production server 4 days ago, but I believe the issue with data not appearing in the pivot tables was happening before that.

Any help would be appreciated. Thank you.

Laura E. Lincks
Database Manager/Developer
ICAP - Columbia University
Mailman School of Public Health
60 Haven Ave, Floor B1
New York, NY 10032
Tel: 212 304 7132

Laura E. Lincks
Database Manager/Developer
ICAP - Columbia University
Mailman School of Public Health
60 Haven Ave, Floor B1
New York, NY 10032
Tel: 212 304 7132

Hi Laura,

Did you try running Data Administrationà Maintenance?Then reboot your server (optional). May this could help.

Regards,

Joao

···

DHIS2, version 2.18

We have recently been experiencing a problem in getting data to appear in Pivot Tables and Data Set Reports.

I can verify that data exists on the data entry screen. Furthermore, all users, user roles, data elements, categories, indicators, data sets and Org Units exist. However, after successfully running Analytics (no errors in the front end, at least) I cannot produce a Pivot Table with any results for a simple query I am certain should return results. Furthermore, I cannot pull up a Data Set report for an Org Unit/Reporting Period for which there is data on the corresponding Data Entry screen. (The form displays with no data in the cells.)

We have a test version of this DHIS instance with data that is about 6 weeks old. I am able to run the simple Pivot Table and Data Set Reports without issue – they produce the expected data.

What might be wrong with our live production instance? What tables can I query on the backend to determine what’s in the Analytics tables? Could it be due to User Access changes (even though the same user sees the data in Data Entry)? We had to reboot the production server 4 days ago, but I believe the issue with data not appearing in the pivot tables was happening before that.

Any help would be appreciated. Thank you.

Laura E. Lincks

Database Manager/Developer

ICAP - Columbia University

Mailman School of Public Health

60 Haven Ave, Floor B1

New York, NY 10032

Tel: 212 304 7132

Thank you Joao,

We followed your advice to no avail. The problem still exists, although I AM able to see data in the Analytics tables but only in the back end. From what I’ve spot-checked, it looks to be correct.

We have another instance on the same physical server which is also causing problems with regard to data not appearing in Analytics as expected (it shows some data, just not the most recent data. It appears to be data from a previous run.) Might there be something wrong with the server? We cannot see anything. We have resolved, I think, all memory leaks and have rebooted the instance and yet the problem persists.

···

On Wed, Aug 19, 2015 at 1:46 AM, Joao Mazuze Joao_Mazuze@chasssmt.com wrote:

Hi Laura,

Did you try running Data Administrationà Maintenance?Then reboot your server (optional). May this could help.

Regards,

Joao

From: Dhis2-users [mailto:dhis2-users-bounces+joao_mazuze=chasssmt.com@lists.launchpad.net] On Behalf Of Laura E. Lincks

Sent: Tuesday, August 18, 2015 11:07 PM

To: DHIS Users

Subject: [Dhis2-users] Data not displaying in Pivot tables and Data Set reports

DHIS2, version 2.18

We have recently been experiencing a problem in getting data to appear in Pivot Tables and Data Set Reports.

I can verify that data exists on the data entry screen. Furthermore, all users, user roles, data elements, categories, indicators, data sets and Org Units exist. However, after successfully running Analytics (no errors in the front end, at least) I cannot produce a Pivot Table with any results for a simple query I am certain should return results. Furthermore, I cannot pull up a Data Set report for an Org Unit/Reporting Period for which there is data on the corresponding Data Entry screen. (The form displays with no data in the cells.)

We have a test version of this DHIS instance with data that is about 6 weeks old. I am able to run the simple Pivot Table and Data Set Reports without issue – they produce the expected data.

What might be wrong with our live production instance? What tables can I query on the backend to determine what’s in the Analytics tables? Could it be due to User Access changes (even though the same user sees the data in Data Entry)? We had to reboot the production server 4 days ago, but I believe the issue with data not appearing in the pivot tables was happening before that.

Any help would be appreciated. Thank you.

Laura E. Lincks

Database Manager/Developer

ICAP - Columbia University

Mailman School of Public Health

60 Haven Ave, Floor B1

New York, NY 10032

Tel: 212 304 7132


This message may contain privileged and confidential information intended solely for the addressee. Please do not read, disseminate or copy it unless you are the intended recipient. If this message has been received in error, we kindly ask that you notify the sender immediately by return email and delete all copies of the message from your system.

Laura E. Lincks
Database Manager/Developer
ICAP - Columbia University
Mailman School of Public Health
60 Haven Ave, Floor B1
New York, NY 10032
Tel: 212 304 7132

Hi Laura,
Seems a bit strange. Here are a few ideas.

  1. I assume that the analytics finished without any error? It might be good to post a log of the analytics process someplace for us to review.

  2. Check the Cache Strategy , documented here. What is this set to? Try setting it to “No cache”, completely clear your browser cache (or use incognito mode) and try again.

  3. Are you using a caching proxy, such as nginx (as documented here) ? If so, disable it.

Regards,

Jason

···

On Thu, Aug 20, 2015 at 12:27 AM, Laura E. Lincks laura.lincks@icap.columbia.edu wrote:

Thank you Joao,

We followed your advice to no avail. The problem still exists, although I AM able to see data in the Analytics tables but only in the back end. From what I’ve spot-checked, it looks to be correct.

We have another instance on the same physical server which is also causing problems with regard to data not appearing in Analytics as expected (it shows some data, just not the most recent data. It appears to be data from a previous run.) Might there be something wrong with the server? We cannot see anything. We have resolved, I think, all memory leaks and have rebooted the instance and yet the problem persists.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Laura E. Lincks
Database Manager/Developer
ICAP - Columbia University
Mailman School of Public Health
60 Haven Ave, Floor B1
New York, NY 10032
Tel: 212 304 7132

On Wed, Aug 19, 2015 at 1:46 AM, Joao Mazuze Joao_Mazuze@chasssmt.com wrote:

Hi Laura,

Did you try running Data Administrationà Maintenance?Then reboot your server (optional). May this could help.

Regards,

Joao

From: Dhis2-users [mailto:dhis2-users-bounces+joao_mazuze=chasssmt.com@lists.launchpad.net] On Behalf Of Laura E. Lincks

Sent: Tuesday, August 18, 2015 11:07 PM

To: DHIS Users

Subject: [Dhis2-users] Data not displaying in Pivot tables and Data Set reports

DHIS2, version 2.18

We have recently been experiencing a problem in getting data to appear in Pivot Tables and Data Set Reports.

I can verify that data exists on the data entry screen. Furthermore, all users, user roles, data elements, categories, indicators, data sets and Org Units exist. However, after successfully running Analytics (no errors in the front end, at least) I cannot produce a Pivot Table with any results for a simple query I am certain should return results. Furthermore, I cannot pull up a Data Set report for an Org Unit/Reporting Period for which there is data on the corresponding Data Entry screen. (The form displays with no data in the cells.)

We have a test version of this DHIS instance with data that is about 6 weeks old. I am able to run the simple Pivot Table and Data Set Reports without issue – they produce the expected data.

What might be wrong with our live production instance? What tables can I query on the backend to determine what’s in the Analytics tables? Could it be due to User Access changes (even though the same user sees the data in Data Entry)? We had to reboot the production server 4 days ago, but I believe the issue with data not appearing in the pivot tables was happening before that.

Any help would be appreciated. Thank you.

Laura E. Lincks

Database Manager/Developer

ICAP - Columbia University

Mailman School of Public Health

60 Haven Ave, Floor B1

New York, NY 10032

Tel: 212 304 7132


This message may contain privileged and confidential information intended solely for the addressee. Please do not read, disseminate or copy it unless you are the intended recipient. If this message has been received in error, we kindly ask that you notify the sender immediately by return email and delete all copies of the message from your system.

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Thanks for your response, Jason…

We discovered in the log there was a Dimension Restriction on my account… It didn’t cause Analytics creation to fail, but anyone with that restriction could not create pivot tables. I have no idea how it got there, but it seems to work like a virus… since it’s a restriction on my account, anytime I updated or added a user account the restriction was applied there as well (which makes sense, but it wasn’t obvious.) So, other users had the restriction as well, unbeknownst to all of us. We removed the restriction from all accounts and while pivot table creation is not perfect (I’m able to create pivot tables, but not consistently even when I’m consistent in my query selection… and our Pivot table and Data Visualizations no longer appear on the dashboard) I can remedy that and work around it.

But in investigating this, I found some quirks/bugs in DHIS… maybe they’re things that have been corrected in the more recent versions. I’ll test them out in 2.20 and send a bug report if need be.

Thanks all for your help… Problem almost fully resolved.

···

On Thu, Aug 20, 2015 at 12:04 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Laura,
Seems a bit strange. Here are a few ideas.

  1. I assume that the analytics finished without any error? It might be good to post a log of the analytics process someplace for us to review.

  2. Check the Cache Strategy , documented here. What is this set to? Try setting it to “No cache”, completely clear your browser cache (or use incognito mode) and try again.

  1. Are you using a caching proxy, such as nginx (as documented here) ? If so, disable it.

Regards,

Jason

Laura E. Lincks
Database Manager/Developer
ICAP - Columbia University
Mailman School of Public Health
60 Haven Ave, Floor B1
New York, NY 10032
Tel: 212 304 7132

On Thu, Aug 20, 2015 at 12:27 AM, Laura E. Lincks laura.lincks@icap.columbia.edu wrote:

Thank you Joao,

We followed your advice to no avail. The problem still exists, although I AM able to see data in the Analytics tables but only in the back end. From what I’ve spot-checked, it looks to be correct.

We have another instance on the same physical server which is also causing problems with regard to data not appearing in Analytics as expected (it shows some data, just not the most recent data. It appears to be data from a previous run.) Might there be something wrong with the server? We cannot see anything. We have resolved, I think, all memory leaks and have rebooted the instance and yet the problem persists.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Laura E. Lincks
Database Manager/Developer
ICAP - Columbia University
Mailman School of Public Health
60 Haven Ave, Floor B1
New York, NY 10032
Tel: 212 304 7132

On Wed, Aug 19, 2015 at 1:46 AM, Joao Mazuze Joao_Mazuze@chasssmt.com wrote:

Hi Laura,

Did you try running Data Administrationà Maintenance?Then reboot your server (optional). May this could help.

Regards,

Joao

From: Dhis2-users [mailto:dhis2-users-bounces+joao_mazuze=chasssmt.com@lists.launchpad.net] On Behalf Of Laura E. Lincks

Sent: Tuesday, August 18, 2015 11:07 PM

To: DHIS Users

Subject: [Dhis2-users] Data not displaying in Pivot tables and Data Set reports

DHIS2, version 2.18

We have recently been experiencing a problem in getting data to appear in Pivot Tables and Data Set Reports.

I can verify that data exists on the data entry screen. Furthermore, all users, user roles, data elements, categories, indicators, data sets and Org Units exist. However, after successfully running Analytics (no errors in the front end, at least) I cannot produce a Pivot Table with any results for a simple query I am certain should return results. Furthermore, I cannot pull up a Data Set report for an Org Unit/Reporting Period for which there is data on the corresponding Data Entry screen. (The form displays with no data in the cells.)

We have a test version of this DHIS instance with data that is about 6 weeks old. I am able to run the simple Pivot Table and Data Set Reports without issue – they produce the expected data.

What might be wrong with our live production instance? What tables can I query on the backend to determine what’s in the Analytics tables? Could it be due to User Access changes (even though the same user sees the data in Data Entry)? We had to reboot the production server 4 days ago, but I believe the issue with data not appearing in the pivot tables was happening before that.

Any help would be appreciated. Thank you.

Laura E. Lincks

Database Manager/Developer

ICAP - Columbia University

Mailman School of Public Health

60 Haven Ave, Floor B1

New York, NY 10032

Tel: 212 304 7132


This message may contain privileged and confidential information intended solely for the addressee. Please do not read, disseminate or copy it unless you are the intended recipient. If this message has been received in error, we kindly ask that you notify the sender immediately by return email and delete all copies of the message from your system.

Hi Laura,

Could you please check data aggregation level of each data element in Data Element module. If the aggregation level of the data element is higher than the organization level at which the data are entered, the data are not shown in pivot table.

I’m not sure if it can solve your problem. But we had the same problem previously. After selecting the lowest organization unit level in aggregation level, the data appeared in pivot table and data set report.

···

On Fri, Aug 21, 2015 at 1:10 AM, Laura E. Lincks laura.lincks@icap.columbia.edu wrote:

Thanks for your response, Jason…

We discovered in the log there was a Dimension Restriction on my account… It didn’t cause Analytics creation to fail, but anyone with that restriction could not create pivot tables. I have no idea how it got there, but it seems to work like a virus… since it’s a restriction on my account, anytime I updated or added a user account the restriction was applied there as well (which makes sense, but it wasn’t obvious.) So, other users had the restriction as well, unbeknownst to all of us. We removed the restriction from all accounts and while pivot table creation is not perfect (I’m able to create pivot tables, but not consistently even when I’m consistent in my query selection… and our Pivot table and Data Visualizations no longer appear on the dashboard) I can remedy that and work around it.

But in investigating this, I found some quirks/bugs in DHIS… maybe they’re things that have been corrected in the more recent versions. I’ll test them out in 2.20 and send a bug report if need be.

Thanks all for your help… Problem almost fully resolved.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Laura E. Lincks
Database Manager/Developer
ICAP - Columbia University
Mailman School of Public Health
60 Haven Ave, Floor B1
New York, NY 10032
Tel: 212 304 7132

On Thu, Aug 20, 2015 at 12:04 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Laura,
Seems a bit strange. Here are a few ideas.

  1. I assume that the analytics finished without any error? It might be good to post a log of the analytics process someplace for us to review.

  2. Check the Cache Strategy , documented here. What is this set to? Try setting it to “No cache”, completely clear your browser cache (or use incognito mode) and try again.

  1. Are you using a caching proxy, such as nginx (as documented here) ? If so, disable it.

Regards,

Jason

On Thu, Aug 20, 2015 at 12:27 AM, Laura E. Lincks laura.lincks@icap.columbia.edu wrote:

Thank you Joao,

We followed your advice to no avail. The problem still exists, although I AM able to see data in the Analytics tables but only in the back end. From what I’ve spot-checked, it looks to be correct.

We have another instance on the same physical server which is also causing problems with regard to data not appearing in Analytics as expected (it shows some data, just not the most recent data. It appears to be data from a previous run.) Might there be something wrong with the server? We cannot see anything. We have resolved, I think, all memory leaks and have rebooted the instance and yet the problem persists.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Laura E. Lincks
Database Manager/Developer
ICAP - Columbia University
Mailman School of Public Health
60 Haven Ave, Floor B1
New York, NY 10032
Tel: 212 304 7132

On Wed, Aug 19, 2015 at 1:46 AM, Joao Mazuze Joao_Mazuze@chasssmt.com wrote:

Hi Laura,

Did you try running Data Administrationà Maintenance?Then reboot your server (optional). May this could help.

Regards,

Joao

From: Dhis2-users [mailto:dhis2-users-bounces+joao_mazuze=chasssmt.com@lists.launchpad.net] On Behalf Of Laura E. Lincks

Sent: Tuesday, August 18, 2015 11:07 PM

To: DHIS Users

Subject: [Dhis2-users] Data not displaying in Pivot tables and Data Set reports

DHIS2, version 2.18

We have recently been experiencing a problem in getting data to appear in Pivot Tables and Data Set Reports.

I can verify that data exists on the data entry screen. Furthermore, all users, user roles, data elements, categories, indicators, data sets and Org Units exist. However, after successfully running Analytics (no errors in the front end, at least) I cannot produce a Pivot Table with any results for a simple query I am certain should return results. Furthermore, I cannot pull up a Data Set report for an Org Unit/Reporting Period for which there is data on the corresponding Data Entry screen. (The form displays with no data in the cells.)

We have a test version of this DHIS instance with data that is about 6 weeks old. I am able to run the simple Pivot Table and Data Set Reports without issue – they produce the expected data.

What might be wrong with our live production instance? What tables can I query on the backend to determine what’s in the Analytics tables? Could it be due to User Access changes (even though the same user sees the data in Data Entry)? We had to reboot the production server 4 days ago, but I believe the issue with data not appearing in the pivot tables was happening before that.

Any help would be appreciated. Thank you.

Laura E. Lincks

Database Manager/Developer

ICAP - Columbia University

Mailman School of Public Health

60 Haven Ave, Floor B1

New York, NY 10032

Tel: 212 304 7132


This message may contain privileged and confidential information intended solely for the addressee. Please do not read, disseminate or copy it unless you are the intended recipient. If this message has been received in error, we kindly ask that you notify the sender immediately by return email and delete all copies of the message from your system.

Dr. Lwin Lwin Aung

Health Information Division

Department of Public Health

Office No.(47), Nay Pyi Taw