Realtime Analytics

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.

···

Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

Hi Bayoh,
Have a look here

http://dhis2.github.io/dhis2-docs/2.22/en/developer/html/ch01s33.html

You can acheive near-real time analytics by ensuring you have enough horsepower in your servers and only aggregating smaller pieces of the data (i.e. last year only).

Regards,

Jason

···

On Fri, Apr 22, 2016 at 12:19 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

Bayoh.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Hi Jason

While this is happening, are reports still available or not?

···

On Fri, Apr 22, 2016 at 5:31 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Bayoh,
Have a look here

http://dhis2.github.io/dhis2-docs/2.22/en/developer/html/ch01s33.html

You can acheive near-real time analytics by ensuring you have enough horsepower in your servers and only aggregating smaller pieces of the data (i.e. last year only).

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 12:19 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

Bayoh.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Thanks Jason for your prompt response, It seems to be the api for using the manual export tables and the scheduling Analytics table externally; please correct me if am wrong.

···

On Fri, Apr 22, 2016 at 10:40 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Jason

While this is happening, are reports still available or not?

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 5:31 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Bayoh,
Have a look here

http://dhis2.github.io/dhis2-docs/2.22/en/developer/html/ch01s33.html

You can acheive near-real time analytics by ensuring you have enough horsepower in your servers and only aggregating smaller pieces of the data (i.e. last year only).

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

On Fri, Apr 22, 2016 at 12:19 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

Bayoh.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.

···


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Hi,
There may be a brief window of time when certain resources are not available. This happens when the analytics tables are switched. Lars can explain more about this. But it should be very brief.

This API endpoint simply triggers the analytics, and and allows more control over what is actually done. So, you might aggregate data once an hour throughout the day (near real time), then run a full analytics run at night per usual.

I think what you want, as many people have asked for, is to enter your data and immediately see it instantaneously in the Pivots It does not work like that, as the data has to be aggregated, and indexed to make it highly available. This is the reason that the pivots are usually very fast, because the data has been pre-processed from the raw data, and heavily indexed to speed up the query from the analytics tables. Its a tradeoff between a highly available server for everyone, or choking the server with lots of expensive real-time aggregation requests.

Usually, this is not the “normal” workflow. People enter monthly data for instance once a month, and whether it gets aggregated one second or one hour after, should not make a difference. So whether they see the data the instant it was entered is not important. What is important is being able to serve lots of data to many people, so that is why the analytics solution exists. Data is pre-processed to make it quickly readable. The downside of this, there is a lag between when the data is entered and when it is actually available through the analytics resources.

Having said all of that, the developers are looking into ways to speed this process up, but for now, for instance by only aggregating what is actually needed (dirty data) along with offloading the analytics onto a separate server. But that is in the pipeline. For now, the best thing which you can do is to be sure you get a very fast database server with lots of RAM, and schedule the analytics run (simply by making a curl call to that API endpoint) once every X minutes. X minutes will depend on a number of factors, like how much data you have, how powerful your server is, etc. So, you will need to experiment a bit.

Regards,

Jason

···

On Fri, Apr 22, 2016 at 12:51 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

Thanks Jason for your prompt response, It seems to be the api for using the manual export tables and the scheduling Analytics table externally; please correct me if am wrong.

On Fri, Apr 22, 2016 at 10:40 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Jason

While this is happening, are reports still available or not?


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 5:31 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Bayoh,
Have a look here

http://dhis2.github.io/dhis2-docs/2.22/en/developer/html/ch01s33.html

You can acheive near-real time analytics by ensuring you have enough horsepower in your servers and only aggregating smaller pieces of the data (i.e. last year only).

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

On Fri, Apr 22, 2016 at 12:19 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

Bayoh.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

But I thought all analytic tables was cleared out during re-generation? is this not true?

···

On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring knutst@gmail.com wrote:

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

Jason

Basically what most people are asking for is:

  1. Enter data

  2. give me report

They don’t say… give me report for full country… they don’t care about that, it’s all about their leaf node…

···

On Fri, Apr 22, 2016 at 6:03 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi,
There may be a brief window of time when certain resources are not available. This happens when the analytics tables are switched. Lars can explain more about this. But it should be very brief.

This API endpoint simply triggers the analytics, and and allows more control over what is actually done. So, you might aggregate data once an hour throughout the day (near real time), then run a full analytics run at night per usual.

I think what you want, as many people have asked for, is to enter your data and immediately see it instantaneously in the Pivots It does not work like that, as the data has to be aggregated, and indexed to make it highly available. This is the reason that the pivots are usually very fast, because the data has been pre-processed from the raw data, and heavily indexed to speed up the query from the analytics tables. Its a tradeoff between a highly available server for everyone, or choking the server with lots of expensive real-time aggregation requests.

Usually, this is not the “normal” workflow. People enter monthly data for instance once a month, and whether it gets aggregated one second or one hour after, should not make a difference. So whether they see the data the instant it was entered is not important. What is important is being able to serve lots of data to many people, so that is why the analytics solution exists. Data is pre-processed to make it quickly readable. The downside of this, there is a lag between when the data is entered and when it is actually available through the analytics resources.

Having said all of that, the developers are looking into ways to speed this process up, but for now, for instance by only aggregating what is actually needed (dirty data) along with offloading the analytics onto a separate server. But that is in the pipeline. For now, the best thing which you can do is to be sure you get a very fast database server with lots of RAM, and schedule the analytics run (simply by making a curl call to that API endpoint) once every X minutes. X minutes will depend on a number of factors, like how much data you have, how powerful your server is, etc. So, you will need to experiment a bit.

Regards,

Jason

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 12:51 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

Thanks Jason for your prompt response, It seems to be the api for using the manual export tables and the scheduling Analytics table externally; please correct me if am wrong.


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On Fri, Apr 22, 2016 at 10:40 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Jason

While this is happening, are reports still available or not?


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 5:31 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Bayoh,
Have a look here

http://dhis2.github.io/dhis2-docs/2.22/en/developer/html/ch01s33.html

You can acheive near-real time analytics by ensuring you have enough horsepower in your servers and only aggregating smaller pieces of the data (i.e. last year only).

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

On Fri, Apr 22, 2016 at 12:19 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

Bayoh.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Sure makes sense I think and it sounds like a good feature (aggregate only this leaf node and downwards). Blurprint?

Problem is, if you make analytics in one branch of the hierarchy, what about the ones above it?

Also, there are other dimensions other than the organiation unit to consider, like “dataset”. Maybe certain data sets should be aggregated more often than others. It would seem to make little sense to aggregate a yearly dataset once an hour, when it is only entered once a year. It might make sense to aggregate however a daily dataset a few times a day.

This is really about “dirty” analytics. Only aggregating processing what is needed. Right now, the process is not that efficient, as a lot of things get aggregated which have not changed. But determinering what is dirty is not that simple really.

At any rate, one solution is simply the brute-force approach. Get a very database fast server seperate from your application server with lots of RAM , and you can probably run analytics as often as you like, within reason.

···

On Fri, Apr 22, 2016 at 1:05 PM, Morten Olav Hansen morten@dhis2.org wrote:

Jason

From my perspective… what they want, is to see “aggregated” data from their leaf only… so there is not really anything to do, any leaf based analytics could (should) be done real-time right?

Basically what most people are asking for is:

  1. Enter data
  1. give me report

They don’t say… give me report for full country… they don’t care about that, it’s all about their leaf node…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:03 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi,
There may be a brief window of time when certain resources are not available. This happens when the analytics tables are switched. Lars can explain more about this. But it should be very brief.

This API endpoint simply triggers the analytics, and and allows more control over what is actually done. So, you might aggregate data once an hour throughout the day (near real time), then run a full analytics run at night per usual.

I think what you want, as many people have asked for, is to enter your data and immediately see it instantaneously in the Pivots It does not work like that, as the data has to be aggregated, and indexed to make it highly available. This is the reason that the pivots are usually very fast, because the data has been pre-processed from the raw data, and heavily indexed to speed up the query from the analytics tables. Its a tradeoff between a highly available server for everyone, or choking the server with lots of expensive real-time aggregation requests.

Usually, this is not the “normal” workflow. People enter monthly data for instance once a month, and whether it gets aggregated one second or one hour after, should not make a difference. So whether they see the data the instant it was entered is not important. What is important is being able to serve lots of data to many people, so that is why the analytics solution exists. Data is pre-processed to make it quickly readable. The downside of this, there is a lag between when the data is entered and when it is actually available through the analytics resources.

Having said all of that, the developers are looking into ways to speed this process up, but for now, for instance by only aggregating what is actually needed (dirty data) along with offloading the analytics onto a separate server. But that is in the pipeline. For now, the best thing which you can do is to be sure you get a very fast database server with lots of RAM, and schedule the analytics run (simply by making a curl call to that API endpoint) once every X minutes. X minutes will depend on a number of factors, like how much data you have, how powerful your server is, etc. So, you will need to experiment a bit.

Regards,

Jason

On Fri, Apr 22, 2016 at 12:51 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

Thanks Jason for your prompt response, It seems to be the api for using the manual export tables and the scheduling Analytics table externally; please correct me if am wrong.


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On Fri, Apr 22, 2016 at 10:40 AM, Morten Olav Hansen morten@dhis2.org wrote:

Hi Jason

While this is happening, are reports still available or not?


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 5:31 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Bayoh,
Have a look here

http://dhis2.github.io/dhis2-docs/2.22/en/developer/html/ch01s33.html

You can acheive near-real time analytics by ensuring you have enough horsepower in your servers and only aggregating smaller pieces of the data (i.e. last year only).

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

On Fri, Apr 22, 2016 at 12:19 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

Bayoh.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

No they are swapped now, so first they are built as temp tables, then swapped. So, this is the brief point in time in which things may not be available.

Every 10 minutes might be OK, depending on your server, loading, and amount of data. It just requires some experimentation.

Point is, “real time” analytics is not possible. Near-real time may be.

···

On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen morten@dhis2.org wrote:

But I thought all analytic tables was cleared out during re-generation? is this not true?


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring knutst@gmail.com wrote:

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

If they are swapped, and not dropped. I think that is good enough for most cases (but not for people entering data and want to print report immediately).

For this exact reason we have 2 APIs for events… one real-time, and one for analytics… we have the same for aggreaget, but I think today all reports are based on analytics? (unless doing api based /api/dataValueSets reports) of course…

···

On Fri, Apr 22, 2016 at 6:23 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

No they are swapped now, so first they are built as temp tables, then swapped. So, this is the brief point in time in which things may not be available.

Every 10 minutes might be OK, depending on your server, loading, and amount of data. It just requires some experimentation.

Point is, “real time” analytics is not possible. Near-real time may be.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen morten@dhis2.org wrote:

But I thought all analytic tables was cleared out during re-generation? is this not true?


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring knutst@gmail.com wrote:

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

@Jason,Knut and Morten, Having analytics table run every 10 mins sounds like a good place to start. but am kind of not sure how to implement this with the API calls and am sure this is not possible through the user interface. If you guys can point me in the right direction with an example of some sort that will greatly helpful and highly appreciated.

Thanks.

···

On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

No they are swapped now, so first they are built as temp tables, then swapped. So, this is the brief point in time in which things may not be available.

Every 10 minutes might be OK, depending on your server, loading, and amount of data. It just requires some experimentation.

Point is, “real time” analytics is not possible. Near-real time may be.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen morten@dhis2.org wrote:

But I thought all analytic tables was cleared out during re-generation? is this not true?


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring knutst@gmail.com wrote:

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

Well, it sounds like a bad solution for me :wink: if every 10 min works fine… why not every 5 min… why not every 1 min…

···

On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

@Jason,Knut and Morten, Having analytics table run every 10 mins sounds like a good place to start. but am kind of not sure how to implement this with the API calls and am sure this is not possible through the user interface. If you guys can point me in the right direction with an example of some sort that will greatly helpful and highly appreciated.

Thanks.

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

No they are swapped now, so first they are built as temp tables, then swapped. So, this is the brief point in time in which things may not be available.

Every 10 minutes might be OK, depending on your server, loading, and amount of data. It just requires some experimentation.

Point is, “real time” analytics is not possible. Near-real time may be.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen morten@dhis2.org wrote:

But I thought all analytic tables was cleared out during re-generation? is this not true?


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring knutst@gmail.com wrote:

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

I chose 10 min because the process took 6 min on my server

···

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

@Jason,Knut and Morten, Having analytics table run every 10 mins sounds like a good place to start. but am kind of not sure how to implement this with the API calls and am sure this is not possible through the user interface. If you guys can point me in the right direction with an example of some sort that will greatly helpful and highly appreciated.

Thanks.

On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

No they are swapped now, so first they are built as temp tables, then swapped. So, this is the brief point in time in which things may not be available.

Every 10 minutes might be OK, depending on your server, loading, and amount of data. It just requires some experimentation.

Point is, “real time” analytics is not possible. Near-real time may be.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen morten@dhis2.org wrote:

But I thought all analytic tables was cleared out during re-generation? is this not true?


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring knutst@gmail.com wrote:

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Why not ever 1 nanosecond? Its always going to take some amount of time, question is what is reasonable.

One could argue that real-time analytics in an aggregate data system is not needed. This is not real time data. Its not even close to it.

Look at Google Analytics. They provide reports once a day, and you do not see a whole lot of people complaning. Yes, you can get some limited real time information from this as well, but its limited. The data must be processed first, and that takes computational time. Same with DHIS2.

As for the call to the API, just create a Bash script and call it as frequently as you like with a cron task.

This is a very simple one, but you should really check for things like “Is analytics already running and should I trigger another run?”

#!/bin/sh

/usr/bin/curl “localhost:8080/api/resourceTables/analytics?skipResourceTables=true&lastYears=2” -X POST -u admin:district >/dev/null 2>&1

···

On Fri, Apr 22, 2016 at 1:34 PM, Morten Olav Hansen morten@dhis2.org wrote:

Well, it sounds like a bad solution for me :wink: if every 10 min works fine… why not every 5 min… why not every 1 min…

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

@Jason,Knut and Morten, Having analytics table run every 10 mins sounds like a good place to start. but am kind of not sure how to implement this with the API calls and am sure this is not possible through the user interface. If you guys can point me in the right direction with an example of some sort that will greatly helpful and highly appreciated.

Thanks.

On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

No they are swapped now, so first they are built as temp tables, then swapped. So, this is the brief point in time in which things may not be available.

Every 10 minutes might be OK, depending on your server, loading, and amount of data. It just requires some experimentation.

Point is, “real time” analytics is not possible. Near-real time may be.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen morten@dhis2.org wrote:

But I thought all analytic tables was cleared out during re-generation? is this not true?


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring knutst@gmail.com wrote:

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

1 ns is fine… but i depends on what the ramifications are… what if you start a new job every 1 ms? is that ok? what happens during table swap? is analytic blocked?

Maybe this is already documented… just curious :slight_smile:

···

On Fri, Apr 22, 2016 at 6:48 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Why not ever 1 nanosecond? Its always going to take some amount of time, question is what is reasonable.

One could argue that real-time analytics in an aggregate data system is not needed. This is not real time data. Its not even close to it.

Look at Google Analytics. They provide reports once a day, and you do not see a whole lot of people complaning. Yes, you can get some limited real time information from this as well, but its limited. The data must be processed first, and that takes computational time. Same with DHIS2.

From my experience, people think they need “real time analytics” when they really are just in a rush. Data takes time to review and analayze and whether its available now, 1 nanosecond from now, or 10 minutes from now, makes no difference in the end, as the amount of time which is required to digest that information is on a totally different time scale (hours days or weeks). Once an hour is probably easily achievable depending on the scale of the system however.

As for the call to the API, just create a Bash script and call it as frequently as you like with a cron task.

This is a very simple one, but you should really check for things like “Is analytics already running and should I trigger another run?”

#!/bin/sh

/usr/bin/curl “localhost:8080/api/resourceTables/analytics?skipResourceTables=true&lastYears=2” -X POST -u admin:district >/dev/null 2>&1

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 1:34 PM, Morten Olav Hansen morten@dhis2.org wrote:

Well, it sounds like a bad solution for me :wink: if every 10 min works fine… why not every 5 min… why not every 1 min…


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

@Jason,Knut and Morten, Having analytics table run every 10 mins sounds like a good place to start. but am kind of not sure how to implement this with the API calls and am sure this is not possible through the user interface. If you guys can point me in the right direction with an example of some sort that will greatly helpful and highly appreciated.

Thanks.

On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

No they are swapped now, so first they are built as temp tables, then swapped. So, this is the brief point in time in which things may not be available.

Every 10 minutes might be OK, depending on your server, loading, and amount of data. It just requires some experimentation.

Point is, “real time” analytics is not possible. Near-real time may be.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen morten@dhis2.org wrote:

But I thought all analytic tables was cleared out during re-generation? is this not true?


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring knutst@gmail.com wrote:

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason/Morten,

I agree that real time analytics would not be easily implemented. May be it is time to separate Aggregate and Tracker analytics as a start but it also depends on what the tracker is being used for e.g surveys may require no immediate analytics but IDSR/outbreaks may require instant update of analytics. We are looking at IDSR features and as I see, real time analytics will be a requirement. The IDSR requires real time analytics since people need to respond to outbreaks and handle outbreak responses and management in real time so that you can intervene and stop the outbreak.

Here is what I propose (just a thought from my discussion with Calle)

We set a configurable (checked) variable/attribute that indicates that if this variable is changed, the analytics process is started for example changing population data has a very big impact on indicators that depend on population,entering lab results or requests or patient updates for IDSR requires immediate analytics, etc. If we had this attribute/variable, then we would use it to identify the corresponding meta data that might need to be / may have changed and update (through temp tables) the only affected analytics tables. This way we can control the load required to run analytics in realtime since the system will only be updating affected changes.

Using the API to trigger analytics every after X minutes may be feasible but not sustainable as we do not control how many threads may be running as the API call through curl does not easily get feedback/update if the previous analytics process has been completed for the new one to start otherwise we can end up in a forever loop if the server resources are not enough.

Alex

···

On Fri, Apr 22, 2016 at 2:50 PM, Morten Olav Hansen morten@dhis2.org wrote:

1 ns is fine… but i depends on what the ramifications are… what if you start a new job every 1 ms? is that ok? what happens during table swap? is analytic blocked?

Maybe this is already documented… just curious :slight_smile:


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:48 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Why not ever 1 nanosecond? Its always going to take some amount of time, question is what is reasonable.

One could argue that real-time analytics in an aggregate data system is not needed. This is not real time data. Its not even close to it.

Look at Google Analytics. They provide reports once a day, and you do not see a whole lot of people complaning. Yes, you can get some limited real time information from this as well, but its limited. The data must be processed first, and that takes computational time. Same with DHIS2.

From my experience, people think they need “real time analytics” when they really are just in a rush. Data takes time to review and analayze and whether its available now, 1 nanosecond from now, or 10 minutes from now, makes no difference in the end, as the amount of time which is required to digest that information is on a totally different time scale (hours days or weeks). Once an hour is probably easily achievable depending on the scale of the system however.

As for the call to the API, just create a Bash script and call it as frequently as you like with a cron task.

This is a very simple one, but you should really check for things like “Is analytics already running and should I trigger another run?”

#!/bin/sh

/usr/bin/curl “localhost:8080/api/resourceTables/analytics?skipResourceTables=true&lastYears=2” -X POST -u admin:district >/dev/null 2>&1

On Fri, Apr 22, 2016 at 1:34 PM, Morten Olav Hansen morten@dhis2.org wrote:

Well, it sounds like a bad solution for me :wink: if every 10 min works fine… why not every 5 min… why not every 1 min…


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

@Jason,Knut and Morten, Having analytics table run every 10 mins sounds like a good place to start. but am kind of not sure how to implement this with the API calls and am sure this is not possible through the user interface. If you guys can point me in the right direction with an example of some sort that will greatly helpful and highly appreciated.

Thanks.

On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

No they are swapped now, so first they are built as temp tables, then swapped. So, this is the brief point in time in which things may not be available.

Every 10 minutes might be OK, depending on your server, loading, and amount of data. It just requires some experimentation.

Point is, “real time” analytics is not possible. Near-real time may be.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen morten@dhis2.org wrote:

But I thought all analytic tables was cleared out during re-generation? is this not true?


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring knutst@gmail.com wrote:

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET

Kampala

Uganda
+256 774149 775, + 256 759 800161

Skype ID: talexie

IT Consultant (Servers, Networks and Security, Health Information Systems - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant

"I don’t want to be anything other than what I have been - one tree hill "

Hi Alex - I agree that IDSR requires “immediate” response - but I we are usually talking hours, not minutes, right?

Knut

···

On Fri, Apr 22, 2016 at 2:23 PM, Alex Tumwesigye atumwesigye@gmail.com wrote:

Jason/Morten,

I agree that real time analytics would not be easily implemented. May be it is time to separate Aggregate and Tracker analytics as a start but it also depends on what the tracker is being used for e.g surveys may require no immediate analytics but IDSR/outbreaks may require instant update of analytics. We are looking at IDSR features and as I see, real time analytics will be a requirement. The IDSR requires real time analytics since people need to respond to outbreaks and handle outbreak responses and management in real time so that you can intervene and stop the outbreak.

Here is what I propose (just a thought from my discussion with Calle)

We set a configurable (checked) variable/attribute that indicates that if this variable is changed, the analytics process is started for example changing population data has a very big impact on indicators that depend on population,entering lab results or requests or patient updates for IDSR requires immediate analytics, etc. If we had this attribute/variable, then we would use it to identify the corresponding meta data that might need to be / may have changed and update (through temp tables) the only affected analytics tables. This way we can control the load required to run analytics in realtime since the system will only be updating affected changes.

Using the API to trigger analytics every after X minutes may be feasible but not sustainable as we do not control how many threads may be running as the API call through curl does not easily get feedback/update if the previous analytics process has been completed for the new one to start otherwise we can end up in a forever loop if the server resources are not enough.

Alex


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

On Fri, Apr 22, 2016 at 2:50 PM, Morten Olav Hansen morten@dhis2.org wrote:

1 ns is fine… but i depends on what the ramifications are… what if you start a new job every 1 ms? is that ok? what happens during table swap? is analytic blocked?

Maybe this is already documented… just curious :slight_smile:


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET

Kampala

Uganda
+256 774149 775, + 256 759 800161

Skype ID: talexie

IT Consultant (Servers, Networks and Security, Health Information Systems - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant

"I don’t want to be anything other than what I have been - one tree hill "

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:48 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Why not ever 1 nanosecond? Its always going to take some amount of time, question is what is reasonable.

One could argue that real-time analytics in an aggregate data system is not needed. This is not real time data. Its not even close to it.

Look at Google Analytics. They provide reports once a day, and you do not see a whole lot of people complaning. Yes, you can get some limited real time information from this as well, but its limited. The data must be processed first, and that takes computational time. Same with DHIS2.

From my experience, people think they need “real time analytics” when they really are just in a rush. Data takes time to review and analayze and whether its available now, 1 nanosecond from now, or 10 minutes from now, makes no difference in the end, as the amount of time which is required to digest that information is on a totally different time scale (hours days or weeks). Once an hour is probably easily achievable depending on the scale of the system however.

As for the call to the API, just create a Bash script and call it as frequently as you like with a cron task.

This is a very simple one, but you should really check for things like “Is analytics already running and should I trigger another run?”

#!/bin/sh

/usr/bin/curl “localhost:8080/api/resourceTables/analytics?skipResourceTables=true&lastYears=2” -X POST -u admin:district >/dev/null 2>&1

On Fri, Apr 22, 2016 at 1:34 PM, Morten Olav Hansen morten@dhis2.org wrote:

Well, it sounds like a bad solution for me :wink: if every 10 min works fine… why not every 5 min… why not every 1 min…


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

@Jason,Knut and Morten, Having analytics table run every 10 mins sounds like a good place to start. but am kind of not sure how to implement this with the API calls and am sure this is not possible through the user interface. If you guys can point me in the right direction with an example of some sort that will greatly helpful and highly appreciated.

Thanks.

On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

No they are swapped now, so first they are built as temp tables, then swapped. So, this is the brief point in time in which things may not be available.

Every 10 minutes might be OK, depending on your server, loading, and amount of data. It just requires some experimentation.

Point is, “real time” analytics is not possible. Near-real time may be.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen morten@dhis2.org wrote:

But I thought all analytic tables was cleared out during re-generation? is this not true?


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring knutst@gmail.com wrote:

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

Hi Knut,

We are not talking of hours here, dashboards needs to be updated in almost real time and outbreaks needed to be detected as data is entered (once thresholds are met). So 10 minutes as an option in the dropdown for analytics is ok for me and I would consider it ok. Adding it would make the system almost real time and this would cater for the IDSR requirement. A lag of 10 minutes for an update of dashboard on screen is ok.

I wish, it could be added to the system than doing it using the API given the reasons above.

Alex

···

On Fri, Apr 22, 2016 at 3:27 PM, Knut Staring knutst@gmail.com wrote:

Hi Alex - I agree that IDSR requires “immediate” response - but I we are usually talking hours, not minutes, right?

Knut

On Fri, Apr 22, 2016 at 2:23 PM, Alex Tumwesigye atumwesigye@gmail.com wrote:

Jason/Morten,

I agree that real time analytics would not be easily implemented. May be it is time to separate Aggregate and Tracker analytics as a start but it also depends on what the tracker is being used for e.g surveys may require no immediate analytics but IDSR/outbreaks may require instant update of analytics. We are looking at IDSR features and as I see, real time analytics will be a requirement. The IDSR requires real time analytics since people need to respond to outbreaks and handle outbreak responses and management in real time so that you can intervene and stop the outbreak.

Here is what I propose (just a thought from my discussion with Calle)

We set a configurable (checked) variable/attribute that indicates that if this variable is changed, the analytics process is started for example changing population data has a very big impact on indicators that depend on population,entering lab results or requests or patient updates for IDSR requires immediate analytics, etc. If we had this attribute/variable, then we would use it to identify the corresponding meta data that might need to be / may have changed and update (through temp tables) the only affected analytics tables. This way we can control the load required to run analytics in realtime since the system will only be updating affected changes.

Using the API to trigger analytics every after X minutes may be feasible but not sustainable as we do not control how many threads may be running as the API call through curl does not easily get feedback/update if the previous analytics process has been completed for the new one to start otherwise we can end up in a forever loop if the server resources are not enough.

Alex


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

On Fri, Apr 22, 2016 at 2:50 PM, Morten Olav Hansen morten@dhis2.org wrote:

1 ns is fine… but i depends on what the ramifications are… what if you start a new job every 1 ms? is that ok? what happens during table swap? is analytic blocked?

Maybe this is already documented… just curious :slight_smile:


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET

Kampala

Uganda
+256 774149 775, + 256 759 800161

Skype ID: talexie

IT Consultant (Servers, Networks and Security, Health Information Systems - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant

"I don’t want to be anything other than what I have been - one tree hill "

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:48 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Why not ever 1 nanosecond? Its always going to take some amount of time, question is what is reasonable.

One could argue that real-time analytics in an aggregate data system is not needed. This is not real time data. Its not even close to it.

Look at Google Analytics. They provide reports once a day, and you do not see a whole lot of people complaning. Yes, you can get some limited real time information from this as well, but its limited. The data must be processed first, and that takes computational time. Same with DHIS2.

From my experience, people think they need “real time analytics” when they really are just in a rush. Data takes time to review and analayze and whether its available now, 1 nanosecond from now, or 10 minutes from now, makes no difference in the end, as the amount of time which is required to digest that information is on a totally different time scale (hours days or weeks). Once an hour is probably easily achievable depending on the scale of the system however.

As for the call to the API, just create a Bash script and call it as frequently as you like with a cron task.

This is a very simple one, but you should really check for things like “Is analytics already running and should I trigger another run?”

#!/bin/sh

/usr/bin/curl “localhost:8080/api/resourceTables/analytics?skipResourceTables=true&lastYears=2” -X POST -u admin:district >/dev/null 2>&1

On Fri, Apr 22, 2016 at 1:34 PM, Morten Olav Hansen morten@dhis2.org wrote:

Well, it sounds like a bad solution for me :wink: if every 10 min works fine… why not every 5 min… why not every 1 min…


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

@Jason,Knut and Morten, Having analytics table run every 10 mins sounds like a good place to start. but am kind of not sure how to implement this with the API calls and am sure this is not possible through the user interface. If you guys can point me in the right direction with an example of some sort that will greatly helpful and highly appreciated.

Thanks.

On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

No they are swapped now, so first they are built as temp tables, then swapped. So, this is the brief point in time in which things may not be available.

Every 10 minutes might be OK, depending on your server, loading, and amount of data. It just requires some experimentation.

Point is, “real time” analytics is not possible. Near-real time may be.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen morten@dhis2.org wrote:

But I thought all analytic tables was cleared out during re-generation? is this not true?


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring knutst@gmail.com wrote:

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET

Kampala

Uganda
+256 774149 775, + 256 759 800161

Skype ID: talexie

IT Consultant (Servers, Networks and Security, Health Information Systems - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant

"I don’t want to be anything other than what I have been - one tree hill "

Hi All, I agree with Alex here, IDSR needs real time updates on indicators. Its was this very reason that triggered my email in the first place. While almost real time will be tolerated but fully real-time is ideal and recommended.

···

On Fri, Apr 22, 2016 at 12:43 PM, Alex Tumwesigye atumwesigye@gmail.com wrote:

Hi Knut,

We are not talking of hours here, dashboards needs to be updated in almost real time and outbreaks needed to be detected as data is entered (once thresholds are met). So 10 minutes as an option in the dropdown for analytics is ok for me and I would consider it ok. Adding it would make the system almost real time and this would cater for the IDSR requirement. A lag of 10 minutes for an update of dashboard on screen is ok.

I wish, it could be added to the system than doing it using the API given the reasons above.

Alex


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

On Fri, Apr 22, 2016 at 3:27 PM, Knut Staring knutst@gmail.com wrote:

Hi Alex - I agree that IDSR requires “immediate” response - but I we are usually talking hours, not minutes, right?

Knut


Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET

Kampala

Uganda
+256 774149 775, + 256 759 800161

Skype ID: talexie

IT Consultant (Servers, Networks and Security, Health Information Systems - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant

"I don’t want to be anything other than what I have been - one tree hill "

On Fri, Apr 22, 2016 at 2:23 PM, Alex Tumwesigye atumwesigye@gmail.com wrote:

Jason/Morten,

I agree that real time analytics would not be easily implemented. May be it is time to separate Aggregate and Tracker analytics as a start but it also depends on what the tracker is being used for e.g surveys may require no immediate analytics but IDSR/outbreaks may require instant update of analytics. We are looking at IDSR features and as I see, real time analytics will be a requirement. The IDSR requires real time analytics since people need to respond to outbreaks and handle outbreak responses and management in real time so that you can intervene and stop the outbreak.

Here is what I propose (just a thought from my discussion with Calle)

We set a configurable (checked) variable/attribute that indicates that if this variable is changed, the analytics process is started for example changing population data has a very big impact on indicators that depend on population,entering lab results or requests or patient updates for IDSR requires immediate analytics, etc. If we had this attribute/variable, then we would use it to identify the corresponding meta data that might need to be / may have changed and update (through temp tables) the only affected analytics tables. This way we can control the load required to run analytics in realtime since the system will only be updating affected changes.

Using the API to trigger analytics every after X minutes may be feasible but not sustainable as we do not control how many threads may be running as the API call through curl does not easily get feedback/update if the previous analytics process has been completed for the new one to start otherwise we can end up in a forever loop if the server resources are not enough.

Alex


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

On Fri, Apr 22, 2016 at 2:50 PM, Morten Olav Hansen morten@dhis2.org wrote:

1 ns is fine… but i depends on what the ramifications are… what if you start a new job every 1 ms? is that ok? what happens during table swap? is analytic blocked?

Maybe this is already documented… just curious :slight_smile:


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET

Kampala

Uganda
+256 774149 775, + 256 759 800161

Skype ID: talexie

IT Consultant (Servers, Networks and Security, Health Information Systems - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant

"I don’t want to be anything other than what I have been - one tree hill "

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:48 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Why not ever 1 nanosecond? Its always going to take some amount of time, question is what is reasonable.

One could argue that real-time analytics in an aggregate data system is not needed. This is not real time data. Its not even close to it.

Look at Google Analytics. They provide reports once a day, and you do not see a whole lot of people complaning. Yes, you can get some limited real time information from this as well, but its limited. The data must be processed first, and that takes computational time. Same with DHIS2.

From my experience, people think they need “real time analytics” when they really are just in a rush. Data takes time to review and analayze and whether its available now, 1 nanosecond from now, or 10 minutes from now, makes no difference in the end, as the amount of time which is required to digest that information is on a totally different time scale (hours days or weeks). Once an hour is probably easily achievable depending on the scale of the system however.

As for the call to the API, just create a Bash script and call it as frequently as you like with a cron task.

This is a very simple one, but you should really check for things like “Is analytics already running and should I trigger another run?”

#!/bin/sh

/usr/bin/curl “localhost:8080/api/resourceTables/analytics?skipResourceTables=true&lastYears=2” -X POST -u admin:district >/dev/null 2>&1

On Fri, Apr 22, 2016 at 1:34 PM, Morten Olav Hansen morten@dhis2.org wrote:

Well, it sounds like a bad solution for me :wink: if every 10 min works fine… why not every 5 min… why not every 1 min…


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh ibrahim.bayoh@sl.ehealthafrica.org wrote:

@Jason,Knut and Morten, Having analytics table run every 10 mins sounds like a good place to start. but am kind of not sure how to implement this with the API calls and am sure this is not possible through the user interface. If you guys can point me in the right direction with an example of some sort that will greatly helpful and highly appreciated.

Thanks.

On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

No they are swapped now, so first they are built as temp tables, then swapped. So, this is the brief point in time in which things may not be available.

Every 10 minutes might be OK, depending on your server, loading, and amount of data. It just requires some experimentation.

Point is, “real time” analytics is not possible. Near-real time may be.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/

On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen morten@dhis2.org wrote:

But I thought all analytic tables was cleared out during re-generation? is this not true?


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring knutst@gmail.com wrote:

Depends on the data. I trigger analytics every 10 min.

Hi All,
Initially i thought DHIS2 had fully or out-of-the-box realtime analytics and Dashboards. But after further inspection i realized that current data entry will not available for analysis or dashboards until the next day or by manually doing export tables. This is a real bottle neck to the intended implementation am working on. I have looked at Scheduling, but the options for analytics table is not ideal in my case. Is there way this can be reduced or removed to gain real-time dashboards and analytics?

Thanks,

Bayoh.


Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Ibrahim Rashid Bayoh
Information Systems Coordinator,

eHealth Africa(Sierra Leone)

117 Wilkinson Rd, Freetown, Sierra Leone

Mobile: +232 88-765-638

ibrahim.bayoh@sl.ehealthafrica.org

http://ehealthafrica.org/