We’re using a lot of new iReports in the Standard reports and they seem to be working well, but there seems to be a serious caching problem. I have a report listing health facilities that I ran this morning, now I’ve added 10 more facilities and they don’t appear in the list…. Though they do appear in the iReport preview screen. I’ve cleared my local cache and even the cache statistics in Data Administration, but I can’t get the new facilities to display.
Any suggestions about how end users can force a cache refresh before running the report? It is critical that this report shows the most up-to date information as we use it to determine the next health facility ID code to assign when creating new organization units.
this might be caused by a web server cache, ie. nginx/apache caching content on the server. In that case it does not help to refresh the browser cache.
These changes to your report will become available tomorrow morning. Caching for anlaytics/reports is controlled under system settings → general → cache strategy (assuming this is set to “6 am next morning”).
When developing reports this can be a bit inconvenient. This is a weakness with ireport and developing reports directly on the production server in general.
Some “fixes”:
Disable server side caching.
Disable caching in dhis.
Refresh cache on the server on demand.
All of these are bad in the sense that it will make end-users wait longer for reports and the server to get increased load.
cheers
Lars
···
On Fri, Sep 13, 2013 at 2:47 PM, Wilson,Randy rwilson@msh.org wrote:
We’re using a lot of new iReports in the Standard reports and they seem to be working well, but there seems to be a serious caching problem. I have a report listing health facilities that I ran this morning, now I’ve added 10 more facilities and they don’t appear in the list…. Though they do appear in the iReport preview screen. I’ve cleared my local cache and even the cache statistics in Data Administration, but I can’t get the new facilities to display.
Any suggestions about how end users can force a cache refresh before running the report? It is critical that this report shows the most up-to date information as we use it to determine the next health facility ID code to assign when creating new organization units.
I am not sure that completely removing the cache would be required. As Lars points out, it is very helpful to have it, so would try and avoid that if all possible. Of course, if you have not enabled caching, this mail is a bit pointless!
I think one solution would be to cache everything, except the report which you do not want to cache. This would require a bit more complex nginx setup, but it should certainly be possible. You could simply define a “location” with a regular expression corresponding to the report URL, and tell nginx not to cache it. Other reports, which could be cached, might have another location. You can see something like that in the implementation docs, which specify two different strategies for caching different types of resources.
Not sure if it will work. Have not tried, but it might be worth a shot.
Regards,
Jason
···
On Fri, Sep 13, 2013 at 7:40 PM, Lars Helge Øverland larshelge@gmail.com wrote:
Hi Randy,
this might be caused by a web server cache, ie. nginx/apache caching content on the server. In that case it does not help to refresh the browser cache.
These changes to your report will become available tomorrow morning. Caching for anlaytics/reports is controlled under system settings → general → cache strategy (assuming this is set to “6 am next morning”).
When developing reports this can be a bit inconvenient. This is a weakness with ireport and developing reports directly on the production server in general.
Some “fixes”:
Disable server side caching.
Disable caching in dhis.
Refresh cache on the server on demand.
All of these are bad in the sense that it will make end-users wait longer for reports and the server to get increased load.
On Fri, Sep 13, 2013 at 2:47 PM, Wilson,Randy rwilson@msh.org wrote:
We’re using a lot of new iReports in the Standard reports and they seem to be working well, but there seems to be a serious caching problem. I have a report listing health facilities that I ran this morning, now I’ve added 10 more facilities and they don’t appear in the list…. Though they do appear in the iReport preview screen. I’ve cleared my local cache and even the cache statistics in Data Administration, but I can’t get the new facilities to display.
Any suggestions about how end users can force a cache refresh before running the report? It is critical that this report shows the most up-to date information as we use it to determine the next health facility ID code to assign when creating new organization units.
Another approach might be to clear specific elements from the nginx cache, rather than to clear the entire cache, though this would require some communication between front end users and back end administrators which would probably be too cumbersome in practice. I wonder if it could be possible to automate this … ie iterate through all reports in the cache and remove them if that report has been updated in the database. And run that every 5 minutes …
I am not sure that completely removing the cache would be required. As Lars points out, it is very helpful to have it, so would try and avoid that if all possible. Of course, if you have not enabled caching, this mail is a bit pointless!
I think one solution would be to cache everything, except the report which you do not want to cache. This would require a bit more complex nginx setup, but it should certainly be possible. You could simply define a “location” with a regular expression corresponding to the report URL, and tell nginx not to cache it. Other reports, which could be cached, might have another location. You can see something like that in the implementation docs, which specify two different strategies for caching different types of resources.
Not sure if it will work. Have not tried, but it might be worth a shot.
On Fri, Sep 13, 2013 at 7:40 PM, Lars Helge Øverland larshelge@gmail.com wrote:
Hi Randy,
this might be caused by a web server cache, ie. nginx/apache caching content on the server. In that case it does not help to refresh the browser cache.
These changes to your report will become available tomorrow morning. Caching for anlaytics/reports is controlled under system settings → general → cache strategy (assuming this is set to “6 am next morning”).
When developing reports this can be a bit inconvenient. This is a weakness with ireport and developing reports directly on the production server in general.
Some “fixes”:
Disable server side caching.
Disable caching in dhis.
Refresh cache on the server on demand.
All of these are bad in the sense that it will make end-users wait longer for reports and the server to get increased load.
On Fri, Sep 13, 2013 at 2:47 PM, Wilson,Randy rwilson@msh.org wrote:
We’re using a lot of new iReports in the Standard reports and they seem to be working well, but there seems to be a serious caching problem. I have a report listing health facilities that I ran this morning, now I’ve added 10 more facilities and they don’t appear in the list…. Though they do appear in the iReport preview screen. I’ve cleared my local cache and even the cache statistics in Data Administration, but I can’t get the new facilities to display.
Any suggestions about how end users can force a cache refresh before running the report? It is critical that this report shows the most up-to date information as we use it to determine the next health facility ID code to assign when creating new organization units.
Perhaps, but for this use case five minutes may not be enough. The point is that this report should not be cached at all. How would you propose to write a script to clean out these specific resources from the cache, and leave everything else? Seems not so straightforward to me.
Considering it is rather trivial to implement this in nginx by creating a different location which would not be cached, I am not convinced some script to clean up the mess is such a good idea. It would seem some script which would be triggered every X minutes would be quite un-eco-friendly, when these resources should have never been cahced to begin with?
Perhaps the real fix would be to allow the report creators to specify whether the report should be cached or not cached?
Another approach might be to clear specific elements from the nginx cache, rather than to clear the entire cache, though this would require some communication between front end users and back end administrators which would probably be too cumbersome in practice. I wonder if it could be possible to automate this … ie iterate through all reports in the cache and remove them if that report has been updated in the database. And run that every 5 minutes …
I am not sure that completely removing the cache would be required. As Lars points out, it is very helpful to have it, so would try and avoid that if all possible. Of course, if you have not enabled caching, this mail is a bit pointless!
I think one solution would be to cache everything, except the report which you do not want to cache. This would require a bit more complex nginx setup, but it should certainly be possible. You could simply define a “location” with a regular expression corresponding to the report URL, and tell nginx not to cache it. Other reports, which could be cached, might have another location. You can see something like that in the implementation docs, which specify two different strategies for caching different types of resources.
Not sure if it will work. Have not tried, but it might be worth a shot.
On Fri, Sep 13, 2013 at 7:40 PM, Lars Helge Øverland larshelge@gmail.com wrote:
Hi Randy,
this might be caused by a web server cache, ie. nginx/apache caching content on the server. In that case it does not help to refresh the browser cache.
These changes to your report will become available tomorrow morning. Caching for anlaytics/reports is controlled under system settings → general → cache strategy (assuming this is set to “6 am next morning”).
When developing reports this can be a bit inconvenient. This is a weakness with ireport and developing reports directly on the production server in general.
Some “fixes”:
Disable server side caching.
Disable caching in dhis.
Refresh cache on the server on demand.
All of these are bad in the sense that it will make end-users wait longer for reports and the server to get increased load.
On Fri, Sep 13, 2013 at 2:47 PM, Wilson,Randy rwilson@msh.org wrote:
We’re using a lot of new iReports in the Standard reports and they seem to be working well, but there seems to be a serious caching problem. I have a report listing health facilities that I ran this morning, now I’ve added 10 more facilities and they don’t appear in the list…. Though they do appear in the iReport preview screen. I’ve cleared my local cache and even the cache statistics in Data Administration, but I can’t get the new facilities to display.
Any suggestions about how end users can force a cache refresh before running the report? It is critical that this report shows the most up-to date information as we use it to determine the next health facility ID code to assign when creating new organization units.
I think the simplest would be if we created a query param &disableCache=false|true , which could be appended to the report URL manually during development phase. If true, DHIS would set a no-cache header on the response, disallowing any cache to store it.
Of course that doesn’t fix the problem with the constantly updated report (unless you create a Resource and points it to that Report including that query param). If this is a special case for just one report then I guess Jason’s suggestion on a special nginx location would be feasible. We could implement a system feature but that will take longer.
is the following mentioned in nginx.conf
sendfile=on;
This will make the file sending cached by nginx.
If you turn this off… nginx will send the generated file to user, instead of the one that’s cached
Of course that doesn’t fix the problem with the constantly updated report (unless you create a Resource and points it to that Report including that query param). If this is a special case for just one report then I guess Jason’s suggestion on a special nginx location would be feasible. We could implement a system feature but that will take longer.
I think the sendfile feature has to do with whether nginx does asynchronous i/o using the sendfile() function in linux kernel, so I doubt turning it on or off is going to help us here.
Agree with Jason that pulling stuff back out of the cache is a hack but trying hard to think of a workaround.
Lars suggestion is interesting. Probably the proper eventual solution is to have a checkbox on the report design (“Don’t Cache”). Mind you once its cached once you are stuck till its expired again.
Randy I can disable caching on nginx temporarily and we can monitor how the performance is. You will still have browser cache and your postgres is quite well provisioned so it might be ok.
Bob
···
On 15 September 2013 19:00, Saptarshi Purkayastha sunbiz@gmail.com wrote:
is the following mentioned in nginx.conf
sendfile=on;
This will make the file sending cached by nginx.
If you turn this off… nginx will send the generated file to user, instead of the one that’s cached
Of course that doesn’t fix the problem with the constantly updated report (unless you create a Resource and points it to that Report including that query param). If this is a special case for just one report then I guess Jason’s suggestion on a special nginx location would be feasible. We could implement a system feature but that will take longer.
I think the sendfile feature has to do with whether nginx does asynchronous i/o using the sendfile() function in linux kernel, so I doubt turning it on or off is going to help us here.
Agree with Jason that pulling stuff back out of the cache is a hack but trying hard to think of a workaround.
Lars suggestion is interesting. Probably the proper eventual solution is to have a checkbox on the report design (“Don’t Cache”). Mind you once its cached once you are stuck till its expired again.
Randy I can disable caching on nginx temporarily and we can monitor how the performance is. You will still have browser cache and your postgres is quite well provisioned so it might be ok.
On 15 September 2013 19:00, Saptarshi Purkayastha sunbiz@gmail.com wrote:
is the following mentioned in nginx.conf
sendfile=on;
This will make the file sending cached by nginx.
If you turn this off… nginx will send the generated file to user, instead of the one that’s cached
Of course that doesn’t fix the problem with the constantly updated report (unless you create a Resource and points it to that Report including that query param). If this is a special case for just one report then I guess Jason’s suggestion on a special nginx location would be feasible. We could implement a system feature but that will take longer.
I think the sendfile feature has to do with whether nginx does asynchronous i/o using the sendfile() function in linux kernel, so I doubt turning it on or off is going to help us here.
Agree with Jason that pulling stuff back out of the cache is a hack but trying hard to think of a workaround.
Lars suggestion is interesting. Probably the proper eventual solution is to have a checkbox on the report design (“Don’t Cache”). Mind you once its cached once you are stuck till its expired again.
Randy I can disable caching on nginx temporarily and we can monitor how the performance is. You will still have browser cache and your postgres is quite well provisioned so it might be ok.
On 15 September 2013 19:00, Saptarshi Purkayastha sunbiz@gmail.com wrote:
is the following mentioned in nginx.conf
sendfile=on;
This will make the file sending cached by nginx.
If you turn this off… nginx will send the generated file to user, instead of the one that’s cached
Of course that doesn’t fix the problem with the constantly updated report (unless you create a Resource and points it to that Report including that query param). If this is a special case for just one report then I guess Jason’s suggestion on a special nginx location would be feasible. We could implement a system feature but that will take longer.
I think the sendfile feature has to do with whether nginx does asynchronous i/o using the sendfile() function in linux kernel, so I doubt turning it on or off is going to help us here.
Agree with Jason that pulling stuff back out of the cache is a hack but trying hard to think of a workaround.
Lars suggestion is interesting. Probably the proper eventual solution is to have a checkbox on the report design (“Don’t Cache”). Mind you once its cached once you are stuck till its expired again.
Randy I can disable caching on nginx temporarily and we can monitor how the performance is. You will still have browser cache and your postgres is quite well provisioned so it might be ok.
On 15 September 2013 19:00, Saptarshi Purkayastha sunbiz@gmail.com wrote:
is the following mentioned in nginx.conf
sendfile=on;
This will make the file sending cached by nginx.
If you turn this off… nginx will send the generated file to user, instead of the one that’s cached
Of course that doesn’t fix the problem with the constantly updated report (unless you create a Resource and points it to that Report including that query param). If this is a special case for just one report then I guess Jason’s suggestion on a special nginx location would be feasible. We could implement a system feature but that will take longer.
I think the sendfile feature has to do with whether nginx does asynchronous i/o using the sendfile() function in linux kernel, so I doubt turning it on or off is going to help us here.
Agree with Jason that pulling stuff back out of the cache is a hack but trying hard to think of a workaround.
Lars suggestion is interesting. Probably the proper eventual solution is to have a checkbox on the report design (“Don’t Cache”). Mind you once its cached once you are stuck till its expired again.
Randy I can disable caching on nginx temporarily and we can monitor how the performance is. You will still have browser cache and your postgres is quite well provisioned so it might be ok.
On 15 September 2013 19:00, Saptarshi Purkayastha sunbiz@gmail.com wrote:
is the following mentioned in nginx.conf
sendfile=on;
This will make the file sending cached by nginx.
If you turn this off… nginx will send the generated file to user, instead of the one that’s cached
Of course that doesn’t fix the problem with the constantly updated report (unless you create a Resource and points it to that Report including that query param). If this is a special case for just one report then I guess Jason’s suggestion on a special nginx location would be feasible. We could implement a system feature but that will take longer.
I’am trying to use person attribute and of cause i also adding option,but when i go to tabular report====>attribute i can not send that one that has options in person attribute to the selected attribute means when i add option just to have drop down,i can not send it to the selected attribute in tabular report.i’am using individual records(tracker).
I think the sendfile feature has to do with whether nginx does asynchronous i/o using the sendfile() function in linux kernel, so I doubt turning it on or off is going to help us here.
Agree with Jason that pulling stuff back out of the cache is a hack but trying hard to think of a workaround.
Lars suggestion is interesting. Probably the proper eventual solution is to have a checkbox on the report design (“Don’t Cache”). Mind you once its cached once you are stuck till its expired again.
Randy I can disable caching on nginx temporarily and we can monitor how the performance is. You will still have browser cache and your postgres is quite well provisioned so it might be ok.
Bob
On 15 September 2013 19:00, Saptarshi Purkayastha sunbiz@gmail.com wrote:
is the following mentioned in nginx.conf
sendfile=on;
This will make the file sending cached by nginx.
If you turn this off… nginx will send the generated file to user, instead of the one that’s cached
Of course that doesn’t fix the problem with the constantly updated report (unless you create a Resource and points it to that Report including that query param). If this is a special case for just one report then I guess Jason’s suggestion on a special nginx location would be feasible. We could implement a system feature but that will take longer.
I’am trying to use person attribute and of cause i also adding option,but when i go to tabular report====>attribute i can not send that one that has options in person attribute to the selected attribute means when i add option just to have drop down,i can not send it to the selected attribute in tabular report.i’am using individual records(tracker).
I’am trying to use person attribute and of cause i also adding option,but when i go to tabular report====>attribute i can not send that one that has options in person attribute to the selected attribute means when i add option just to have drop down,i can not send it to the selected attribute in tabular report.i’am using individual records(tracker).
I think the sendfile feature has to do with whether nginx does asynchronous i/o using the sendfile() function in linux kernel, so I doubt turning it on or off is going to help us here.
Agree with Jason that pulling stuff back out of the cache is a hack but trying hard to think of a workaround.
Lars suggestion is interesting. Probably the proper eventual solution is to have a checkbox on the report design (“Don’t Cache”). Mind you once its cached once you are stuck till its expired again.
Randy I can disable caching on nginx temporarily and we can monitor how the performance is. You will still have browser cache and your postgres is quite well provisioned so it might be ok.
Bob
On 15 September 2013 19:00, Saptarshi Purkayastha sunbiz@gmail.com wrote:
is the following mentioned in nginx.conf
sendfile=on;
This will make the file sending cached by nginx.
If you turn this off… nginx will send the generated file to user, instead of the one that’s cached
Of course that doesn’t fix the problem with the constantly updated report (unless you create a Resource and points it to that Report including that query param). If this is a special case for just one report then I guess Jason’s suggestion on a special nginx location would be feasible. We could implement a system feature but that will take longer.