Problem rendering a pivot table

We are experiencing a problem running pivot tables on version 2.26 of DHIS 2. When the data is requested via the API, the data is returned very quickly. However when we launch a pivot table, specifically with the all the data elements, the pivot table hangs.

From what we can tell via the Chrome developer tools (and other tools), there is a lot of Javascript garbage collection. The same sort of behavior happens in IE and Firefox, but the upshot is that the pivot table hangs as it is trying to render, and eventually the browser crashes.

Has anyone seen this type of thing before? Any suggestions on what to look for?

Thanks in advance,

CvH

Chris van Hasselt

IT Project Manager

359 Blackwell Street, Suite 200 | Durham, NC 27701

O: 919.544.7040 x11271 | cvanhasselt@fhi360.org |

Description: Description: fhi 360 email.jpg

Visit us at
www.fhi360.org

Hi Chris,

This normally is a result of loading a pivot with “too much” data. What too much is is not entirely clear and can depend on a number of factors. Do you get an “Aww snap!” message in the browser? Are you trying to load a “big” pivot with lots of data?

If possible, I would suggest to load smaller subsets of data or use the “Download CSV” function if you are experiencing this issue.

image

Also, any browser console and/or server messages when executing the pivot would be helpful to diagnose the problem.

Regards,
Jason

image

···

On Tue, Jun 20, 2017 at 4:46 PM, Chris van Hasselt CVanHasselt@fhi360.org wrote:

We are experiencing a problem running pivot tables on version 2.26 of DHIS 2. When the data is requested via the API, the data is returned very quickly. However when we launch a pivot table, specifically with the all the data elements, the pivot table hangs.

From what we can tell via the Chrome developer tools (and other tools), there is a lot of Javascript garbage collection. The same sort of behavior happens in IE and Firefox, but the upshot is that the pivot table hangs as it is trying to render, and eventually the browser crashes.

Has anyone seen this type of thing before? Any suggestions on what to look for?

Thanks in advance,

CvH

Chris van Hasselt
| IT Project Manager

359 Blackwell Street, Suite 200 | Durham, NC 27701

O: 919.544.7040 x11271 | cvanhasselt@fhi360.org |


Visit us at
www.fhi360.org


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Hi Jason,

Upgraded to the WAR 2.26 see below details.

Version:

2.26

Build revision:

b853b77

Build date:

2017-06-21 06:41

I also seem to have a similar problem with opening favourites or trying to create a new pivot table. It just hanging.

I have an earlier dated of the 2.26 WAR and don’t seem to have the same issue

image

image

···

From: Dhis2-users [mailto:dhis2-users-bounces+brent=hisp.org@lists.launchpad.net] On Behalf Of Jason Pickering
Sent: 20 June 2017 08:50 PM
To: Chris van Hasselt
Cc: dhis2-users@lists.launchpad.net
Subject: Re: [Dhis2-users] Problem rendering a pivot table

Hi Chris,

This normally is a result of loading a pivot with “too much” data. What too much is is not entirely clear and can depend on a number of factors. Do you get an “Aww snap!” message in the browser? Are you trying to load a “big” pivot with lots of data?

If possible, I would suggest to load smaller subsets of data or use the “Download CSV” function if you are experiencing this issue.

Also, any browser console and/or server messages when executing the pivot would be helpful to diagnose the problem.

Regards,
Jason

On Tue, Jun 20, 2017 at 4:46 PM, Chris van Hasselt CVanHasselt@fhi360.org wrote:

We are experiencing a problem running pivot tables on version 2.26 of DHIS 2. When the data is requested via the API, the data is returned very quickly. However when we launch a pivot table, specifically with the all the data elements, the pivot table hangs.

From what we can tell via the Chrome developer tools (and other tools), there is a lot of Javascript garbage collection. The same sort of behavior happens in IE and Firefox, but the upshot is that the pivot table hangs as it is trying to render, and eventually the browser crashes.

Has anyone seen this type of thing before? Any suggestions on what to look for?

Thanks in advance,

CvH

Chris van Hasselt | IT Project Manager

359 Blackwell Street, Suite 200 | Durham, NC 27701

O: 919.544.7040 x11271 | cvanhasselt@fhi360.org |


Description: Description: fhi 360 email.jpg

Visit us at www.fhi360.org


Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

This message and any attachments are subject to a disclaimer published at http://www.hisp.org/policies.html#comms_disclaimer. Please read the disclaimer before opening any attachment or taking any other action in terms of this electronic transmission. If you cannot access the disclaimer, kindly send an email to disclaimer@hisp.org and a copy will be provided to you. By replying to this e-mail or opening any attachment you agree to be bound by the provisions of the disclaimer.

Thanks Jason – Yes, it is the “Aw, snap!” message in Chrome.

There are no browser console messages when this happens.

As far as the size of the pivot table, the “all elements” option for the data has many, many data elements. There are six data element groups, and when I break it down group by group, there is only one group that fails, with 73 data elements. When this happens, I’m selecting data at the “District” level, and there are many districts.

Yesterday, I started breaking down the 73 elements into smaller groups, selecting 10 at a time. When I did this, the table rendered, but with no data, in all cases. But no “Aw Snap” message.

The downloaded CSV file, however, has data. The file is 212 MB, and takes an exceedingly long time to process, even though I don’t think of his as a huge file.

image

image

···

Hi Chris,

This normally is a result of loading a pivot with “too much” data. What too much is is not entirely clear and can depend on a number of factors. Do you get an “Aww snap!” message in the browser? Are you trying to load a “big” pivot with lots of data?

If possible, I would suggest to load smaller subsets of data or use the “Download CSV” function if you are experiencing this issue.

Also, any browser console and/or server messages when executing the pivot would be helpful to diagnose the problem.

Regards,

Jason

On Tue, Jun 20, 2017 at 4:46 PM, Chris van Hasselt CVanHasselt@fhi360.org wrote:

We are experiencing a problem running pivot tables on version 2.26 of DHIS 2. When the data is requested via the API, the data is returned very quickly. However when we launch a pivot table, specifically with the all the data elements, the pivot table hangs.

From what we can tell via the Chrome developer tools (and other tools), there is a lot of Javascript garbage collection. The same sort of behavior happens in IE and Firefox, but the upshot is that the pivot table hangs as it is trying to render, and eventually the browser crashes.

Has anyone seen this type of thing before? Any suggestions on what to look for?

Thanks in advance,

CvH

Chris van Hasselt
| IT Project Manager

359 Blackwell Street, Suite 200 | Durham, NC 27701

O: 919.544.7040 x11271 | cvanhasselt@fhi360.org |


Visit us at www.fhi360.org


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049

Hi Chris,

Yes, this sounds pretty much as we have seen in other cases when trying to render large pivots in the app. If the downloaded CSV file is 212 MB, that would certainly qualify as a “huge pivot” I think.

The dev team is working on a new implementation of the pivots, which may deal with this issue, but there are always going to be limitations in rendering very large data tables inside a browser window. I think if you are dealing with a CSV file that large, then you may need tools with higher horsepower (R, SAS, SPSS, etc), as opposed to the web browser.

I think the only immediate way forward would be for you to use other tools, or better yet, break the table down into component pieces if at all possible. Smaller more focused tables assembled on a dashboard might allow you to see all of the data in chunks as opposed to one very large table.

Regards,

Jason

image

image

···

On Thu, Jun 22, 2017 at 4:41 PM, Chris van Hasselt CVanHasselt@fhi360.org wrote:

Thanks Jason – Yes, it is the “Aw, snap!” message in Chrome.

There are no browser console messages when this happens.

As far as the size of the pivot table, the “all elements” option for the data has many, many data elements. There are six data element groups, and when I break it down group by group, there is only one group that fails, with 73 data elements. When this happens, I’m selecting data at the “District” level, and there are many districts.

Yesterday, I started breaking down the 73 elements into smaller groups, selecting 10 at a time. When I did this, the table rendered, but with no data, in all cases. But no “Aw Snap” message.

The downloaded CSV file, however, has data. The file is 212 MB, and takes an exceedingly long time to process, even though I don’t think of his as a huge file.

From: Jason Pickering [mailto:jason.p.pickering@gmail.com]

Sent: Tuesday, June 20, 2017 2:50 PM

To: Chris van Hasselt CVanHasselt@fhi360.org

Cc: dhis2-users@lists.launchpad.net

Subject: Re: [Dhis2-users] Problem rendering a pivot table

Hi Chris,

This normally is a result of loading a pivot with “too much” data. What too much is is not entirely clear and can depend on a number of factors. Do you get an “Aww snap!” message in the browser? Are you trying to load a “big” pivot with lots of data?

If possible, I would suggest to load smaller subsets of data or use the “Download CSV” function if you are experiencing this issue.

Also, any browser console and/or server messages when executing the pivot would be helpful to diagnose the problem.

Regards,

Jason

On Tue, Jun 20, 2017 at 4:46 PM, Chris van Hasselt CVanHasselt@fhi360.org wrote:

We are experiencing a problem running pivot tables on version 2.26 of DHIS 2. When the data is requested via the API, the data is returned very quickly. However when we launch a pivot table, specifically with the all the data elements, the pivot table hangs.

From what we can tell via the Chrome developer tools (and other tools), there is a lot of Javascript garbage collection. The same sort of behavior happens in IE and Firefox, but the upshot is that the pivot table hangs as it is trying to render, and eventually the browser crashes.

Has anyone seen this type of thing before? Any suggestions on what to look for?

Thanks in advance,

CvH

Chris van Hasselt
| IT Project Manager

359 Blackwell Street, Suite 200 | Durham, NC 27701

O: 919.544.7040 x11271 | cvanhasselt@fhi360.org |


Visit us at www.fhi360.org


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

This is more or less the conclusion we came to – recommend that if you want that much data, do an API query from Power BI (which we are having a good deal of success with).

Thanks

Chris

image

image

···

Hi Chris,

Yes, this sounds pretty much as we have seen in other cases when trying to render large pivots in the app. If the downloaded CSV file is 212 MB, that would certainly qualify as a “huge pivot” I think.

The dev team is working on a new implementation of the pivots, which may deal with this issue, but there are always going to be limitations in rendering very large data tables inside a browser window. I think if you are dealing with a CSV file that large, then you may need tools with higher horsepower (R, SAS, SPSS, etc), as opposed to the web browser.

I think the only immediate way forward would be for you to use other tools, or better yet, break the table down into component pieces if at all possible. Smaller more focused tables assembled on a dashboard might allow you to see all of the data in chunks as opposed to one very large table.

Regards,

Jason

On Thu, Jun 22, 2017 at 4:41 PM, Chris van Hasselt CVanHasselt@fhi360.org wrote:

Thanks Jason – Yes, it is the “Aw, snap!” message in Chrome.

There are no browser console messages when this happens.

As far as the size of the pivot table, the “all elements” option for the data has many, many data elements. There are six data element groups, and when I break it down group by group, there is only one group that fails, with 73 data elements. When this happens, I’m selecting data at the “District” level, and there are many districts.

Yesterday, I started breaking down the 73 elements into smaller groups, selecting 10 at a time. When I did this, the table rendered, but with no data, in all cases. But no “Aw Snap” message.

The downloaded CSV file, however, has data. The file is 212 MB, and takes an exceedingly long time to process, even though I don’t think of his as a huge file.

From: Jason Pickering [mailto:jason.p.pickering@gmail.com]

Sent: Tuesday, June 20, 2017 2:50 PM

To: Chris van Hasselt CVanHasselt@fhi360.org

Cc: dhis2-users@lists.launchpad.net

Subject: Re: [Dhis2-users] Problem rendering a pivot table

Hi Chris,

This normally is a result of loading a pivot with “too much” data. What too much is is not entirely clear and can depend on a number of factors. Do you get an “Aww snap!” message in the browser? Are you trying to load a “big” pivot with lots of data?

If possible, I would suggest to load smaller subsets of data or use the “Download CSV” function if you are experiencing this issue.

Also, any browser console and/or server messages when executing the pivot would be helpful to diagnose the problem.

Regards,

Jason

On Tue, Jun 20, 2017 at 4:46 PM, Chris van Hasselt CVanHasselt@fhi360.org wrote:

We are experiencing a problem running pivot tables on version 2.26 of DHIS 2. When the data is requested via the API, the data is returned very quickly. However when we launch a pivot table, specifically with the all the data elements, the pivot table hangs.

From what we can tell via the Chrome developer tools (and other tools), there is a lot of Javascript garbage collection. The same sort of behavior happens in IE and Firefox, but the upshot is that the pivot table hangs as it is trying to render, and eventually the browser crashes.

Has anyone seen this type of thing before? Any suggestions on what to look for?

Thanks in advance,

CvH

Chris van Hasselt
| IT Project Manager

359 Blackwell Street, Suite 200 | Durham, NC 27701

O: 919.544.7040 x11271 | cvanhasselt@fhi360.org |


Visit us at www.fhi360.org


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049