DHIS2 - Versioning of data

Support - Assistance technique
1

Hi all,

Just wondering if there’s any way to implement a kind of versioning in DHIS2? With versioning, I mean having for instance a population census in 2010 of which the results are used in all indicator calculations between 2010 and 2012, i.e. until a new census is published. It’s then the same for the new census. So basically a way to store data with validity dates used for reporting and indicator calculation.

A possible solution might be to use data sets with a certain periodicity (e.g. 1 month) and copy-paste the previous value manually until an update takes place, but that’s not a very elegant solution. A simple “most recent value” filter for reporting and indicator calculation for instance would already do for me (although it wouldn’t give correct results for queries run on data from the past).

Thanks for any input,

Robin

2

Just wanted to relaunch this one (see below). Anyone has any ideas or suggestions?

Kind regards,

Robin

···

From: Dhis2-users [mailto:dhis2-users-bounces+martens=sher.be@lists.launchpad.net] On Behalf Of Robin Martens

Sent: 16 July 2014 10:16

To: dhis2-users@lists.launchpad.net

Subject: [Dhis2-users] DHIS2 - Versioning of data

Hi all,

Just wondering if there’s any way to implement a kind of versioning in DHIS2? With versioning, I mean having for instance a population census in 2010 of which the results are used in all indicator calculations between 2010 and 2012, i.e. until a new census is published. It’s then the same for the new census. So basically a way to store data with validity dates used for reporting and indicator calculation.

A possible solution might be to use data sets with a certain periodicity (e.g. 1 month) and copy-paste the previous value manually until an update takes place, but that’s not a very elegant solution. A simple “most recent value” filter for reporting and indicator calculation for instance would already do for me (although it wouldn’t give correct results for queries run on data from the past).

Thanks for any input,

Robin

3

Hi Robin,

its a good question and actually something that has been discussed before. You highlight some of the pros and cons in your email.

The reason why we haven’t done it so far is simply that we want to avoid the complexity that comes with validity dates instead of routine time periods and data aggregation. If this is used with care it would work well but the system must take handle the quirky cases as well. Example: if someone enters 10 data sets with completely arbitrary validity dates during a year, then what should be the yearly average?

So far we have been relying on scripts to move data over to new time periods. The typical use case is population data (like you say), where you have censuses being done every 3-10 years. In this case one would enter the data for a year (month is often too granular), then apply a SQL script to copy the data to coming years, taking a growth rate into account. This works pretty well.

Data with validity dates is also bordering to what we call “events”, or “surveys”, which you can capture through the “tracker” component. Here, events are capture for a timestamp, and you can capture any number of events during a timeline.

regards,

Lars

4

Hi devs and users,

I've been exploring the aggregation query builder which is effectively the link between tracker data and aggregated data. Now I'm a bit stuck on the following limits of the system, namely that a single query seems able to produce only a single data value, for any combination of orgunit and other dimensions there might be. On the other hand, it does calculate (and recalculate in case of past data?) this data value for each past period of the aggregated data element (monthly, quarterly, or any other). Can anyone confirm this is correct?

If so, imagine the following case: you have the aggregated data element "average patient age" which has 3 dimensions: orgunit, gender, and disease type. If you happen to have 20 orgunits, 2 genders, and 5 disease types, this makes 20x2x5=200 separate aggregation queries to be defined manually one by one! And that's only for a single aggregated data element!

Just wondering if I missed a functionality of the query builder somewhere (which I really hope I did).

Thanks for any feedback,

Robin

5

Hi Robin,
in the ‘DHIS world’ we distinguish between

  • data elements, (‘data variable’, ‘count’), which is may be regarded as an atomic unit / raw piece of data, such as in your case cases of patient encounters by diagnosed disease and by gender (gender / age group will typically be defined as dimension of the data element e.g. cases of measles)

and

  • indicators, which may be any further calculation based on the data elements, but typically percentages, rates, …

You define your indicators - as well as your data elements- in the DHIS meta data management module.

In your case you define the ‘five diseases’ as data elements and the ‘averages’ as indicators.

query builder only then used to aggregate the cases. The averages and other indicators are calculated as indicators by DHIS.

You can see the manual about indicators and data elements and meta data managment

Jørn

···

On Tue, Jul 22, 2014 at 5:39 PM, Robin Martens martens@sher.be wrote:

Hi devs and users,

I’ve been exploring the aggregation query builder which is effectively the link between tracker data and aggregated data. Now I’m a bit stuck on the following limits of the system, namely that a single query seems able to produce only a single data value, for any combination of orgunit and other dimensions there might be. On the other hand, it does calculate (and recalculate in case of past data?) this data value for each past period of the aggregated data element (monthly, quarterly, or any other). Can anyone confirm this is correct?

If so, imagine the following case: you have the aggregated data element “average patient age” which has 3 dimensions: orgunit, gender, and disease type. If you happen to have 20 orgunits, 2 genders, and 5 disease types, this makes 20x2x5=200 separate aggregation queries to be defined manually one by one! And that’s only for a single aggregated data element!

Just wondering if I missed a functionality of the query builder somewhere (which I really hope I did).

Thanks for any feedback,

Robin

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

6

Hi Robin,

in fact the organisation unit dimension should not be part of the
aggregation queries - you can set one aggregation query to run for any
number of org units. You could create the gender and disease type as
categories + category options. However you are right that to populate all
variations of this data element you would need 2 x 5 aggregation queries.

That said, in version 2.15 and 2.16 you have better support for doing
aggregated analysis without using aggregation queries. The "event reports"
app lets you create pivot tables and specify queries directly for
aggregated data, and you can similarly create charts through the "event
visualizer" app, which can then be included directly in the dashboard
together with the routine data charts. Have a look at those apps on the /dev
<DHIS2 App Hub; instance - the queries can be specified on the
left side panel e.g. under the "data" tab.

These apps are backed by the event analytics web api resource (docs
<https://www.dhis2.org/doc/snapshot/en/user/html/ch32s22.html&gt;\). Our
medium-term plan is to phase out the current aggregation queries function
by instead linking routine elements with such event analytics queries.

Now conceptually you would still have to define a query per such data
element + category option combination - it isn't really trivial to convert
events into aggregated data elements "auto-magically".

best regards,

Lars

7

Hi Robin,

I had the same issue as you said . I did it following way , but it may not be the correct way

  • build one aggregate query and save
  • go to database and find caseaggregationcondition table
  • identify how it identify DE
  • you can populate all query builders just in a excel sheet ( i will attach my sample excel sheet it contain 364 A.Q. Builders)
  • upload into database table caseaggregationcondition, as CSV file , It works for me well

some time this may help you.

@Lars answer i have some problem

  1. is it able to use attributes in event reports ?
  2. is it possible to save this values into a form in aggregate side (eg.Monthly return form) ?

Best Regards
Sumudu

TOTAL 364AQ.xlsx (47.1 KB)

···

On Wed, Jul 23, 2014 at 11:58 AM, Lars Helge Øverland larshelge@gmail.com wrote:

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Hi Robin,

in fact the organisation unit dimension should not be part of the aggregation queries - you can set one aggregation query to run for any number of org units. You could create the gender and disease type as categories + category options. However you are right that to populate all variations of this data element you would need 2 x 5 aggregation queries.

That said, in version 2.15 and 2.16 you have better support for doing aggregated analysis without using aggregation queries. The “event reports” app lets you create pivot tables and specify queries directly for aggregated data, and you can similarly create charts through the “event visualizer” app, which can then be included directly in the dashboard together with the routine data charts. Have a look at those apps on the /dev instance - the queries can be specified on the left side panel e.g. under the “data” tab.

These apps are backed by the event analytics web api resource (docs). Our medium-term plan is to phase out the current aggregation queries function by instead linking routine elements with such event analytics queries.

Now conceptually you would still have to define a query per such data element + category option combination - it isn’t really trivial to convert events into aggregated data elements “auto-magically”.

best regards,

Lars

8

Hi Sumudu,

Great solution, and I was actually looking in the same direction as well. It’s good to hear that it’s indeed possible. Thanks a lot!

Regards,

Robin

···

From: Dhis2-users [mailto:dhis2-users-bounces+martens=sher.be@lists.launchpad.net] On Behalf Of sumudu weerasinghe

Sent: 24 July 2014 02:43

To: Lars Helge Øverland

Cc: dhis2-users@lists.launchpad.net

Subject: Re: [Dhis2-users] DHIS2 - Tracker data aggregation

Hi Robin,

I had the same issue as you said . I did it following way , but it may not be the correct way

  • build one aggregate query and save

  • go to database and find caseaggregationcondition table

  • identify how it identify DE

  • you can populate all query builders just in a excel sheet ( i will attach my sample excel sheet it contain 364 A.Q. Builders)

  • upload into database table caseaggregationcondition, as CSV file , It works for me well

some time this may help you.

@Lars answer i have some problem

  1. is it able to use attributes in event reports ?

  2. is it possible to save this values into a form in aggregate side (eg.Monthly return form) ?

Best Regards

Sumudu

On Wed, Jul 23, 2014 at 11:58 AM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Robin,

in fact the organisation unit dimension should not be part of the aggregation queries - you can set one aggregation query to run for any number of org units. You could create the gender and disease type as categories + category options. However you are right that to populate all variations of this data element you would need 2 x 5 aggregation queries.

That said, in version 2.15 and 2.16 you have better support for doing aggregated analysis without using aggregation queries. The “event reports” app lets you create pivot tables and specify queries directly for aggregated data, and you can similarly create charts through the “event visualizer” app, which can then be included directly in the dashboard together with the routine data charts. Have a look at those apps on the /dev instance - the queries can be specified on the left side panel e.g. under the “data” tab.

These apps are backed by the event analytics web api resource (docs ). Our medium-term plan is to phase out the current aggregation queries function by instead linking routine elements with such event analytics queries.

Now conceptually you would still have to define a query per such data element + category option combination - it isn’t really trivial to convert events into aggregated data elements “auto-magically”.

best regards,

Lars

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

9

Dear all,

I didn't find so far any possibility for form printing (blank and filled) and event data approval procedures, as they are available for entries for aggregated data through data sets.

Are these functionalities that are soon planned to be implemented? It would be very useful as it would avoid double work (e.g. filling in paper forms and electronic forms) and allow data approval for decentralized data collection.

Thanks for the feedback,

Robin

10

Hi all,

Following my post of yesterday, here’s some ideas for future functionalities for the event capture and data entry modules:

  1.   Data entry approval for events

  2.   Data entry form print-out (blank and filled) for events

  3.   Adding attachments to events and data sets (e.g. pictures or documents)

What do you think of these? Are they somewhere on the “to be implemented” list or could we put them in a blueprint?

Kind regards,

Robin

···

From: Dhis2-users [mailto:dhis2-users-bounces+martens=sher.be@lists.launchpad.net] On Behalf Of Robin Martens

Sent: 30 July 2014 10:14

To: dhis2-users@lists.launchpad.net

Subject: [Dhis2-users] DHIS2 - Event capture forms printing & approving

Dear all,

I didn’t find so far any possibility for form printing (blank and filled) and event data approval procedures, as they are available for entries for aggregated data through data sets.

Are these functionalities that are soon planned to be implemented? It would be very useful as it would avoid double work (e.g. filling in paper forms and electronic forms) and allow data approval for decentralized data collection.

Thanks for the feedback,

Robin

11

Hi Robin,

Thank you for the feedback. As we pointed sometime back, we are in the process of overhauling the tracker module. In the process some features might be missing or not yet considered - the printing and approval you mentioned are two examples. I can quickly implement the printing feature and back-port it to 2.16. For approval and attachment, we need to take sometime to digest the requirement and come with a generic solution.

Thank you,

Abyot.

···

On Wed, Jul 30, 2014 at 10:14 AM, Robin Martens martens@sher.be wrote:

Dear all,

I didn’t find so far any possibility for form printing (blank and filled) and event data approval procedures, as they are available for entries for aggregated data through data sets.

Are these functionalities that are soon planned to be implemented? It would be very useful as it would avoid double work (e.g. filling in paper forms and electronic forms) and allow data approval for decentralized data collection.

Thanks for the feedback,

Robin

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

12

Hi Abyot,

Thanks for your reply. If it wouldn’t be too much of an effort, it would be great to have the printing option included somewhere soon (as it’s important for reducing workload for data collectors). The other ones are nice-to-have at the moment.

Regards,

Robin

···

From: Abyot Gizaw [mailto:abyota@gmail.com]

Sent: 01 August 2014 10:34

To: Robin Martens

Cc: dhis2-users@lists.launchpad.net

Subject: Re: [Dhis2-users] DHIS2 - Event capture forms printing & approving

Hi Robin,

Thank you for the feedback. As we pointed sometime back, we are in the process of overhauling the tracker module. In the process some features might be missing or not yet considered - the printing and approval you mentioned are two examples. I can quickly implement the printing feature and back-port it to 2.16. For approval and attachment, we need to take sometime to digest the requirement and come with a generic solution.

Thank you,

Abyot.

On Wed, Jul 30, 2014 at 10:14 AM, Robin Martens martens@sher.be wrote:

Dear all,

I didn’t find so far any possibility for form printing (blank and filled) and event data approval procedures, as they are available for entries for aggregated data through data sets.

Are these functionalities that are soon planned to be implemented? It would be very useful as it would avoid double work (e.g. filling in paper forms and electronic forms) and allow data approval for decentralized data collection.

Thanks for the feedback,

Robin

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

13

Hi Robin,

Yes, the printing feature is now implemented - both in trunk and 2.16.

Thank you,

Abyot.

···

On Fri, Aug 1, 2014 at 10:58 AM, Robin Martens martens@sher.be wrote:

Hi Abyot,

Thanks for your reply. If it wouldn’t be too much of an effort, it would be great to have the printing option included somewhere soon (as it’s important for reducing workload for data collectors). The other ones are nice-to-have at the moment.

Regards,

Robin

From: Abyot Gizaw [mailto:abyota@gmail.com]

Sent: 01 August 2014 10:34

To: Robin Martens

Cc: dhis2-users@lists.launchpad.net

Subject: Re: [Dhis2-users] DHIS2 - Event capture forms printing & approving

Hi Robin,

Thank you for the feedback. As we pointed sometime back, we are in the process of overhauling the tracker module. In the process some features might be missing or not yet considered - the printing and approval you mentioned are two examples. I can quickly implement the printing feature and back-port it to 2.16. For approval and attachment, we need to take sometime to digest the requirement and come with a generic solution.

Thank you,

Abyot.

On Wed, Jul 30, 2014 at 10:14 AM, Robin Martens martens@sher.be wrote:

Dear all,

I didn’t find so far any possibility for form printing (blank and filled) and event data approval procedures, as they are available for entries for aggregated data through data sets.

Are these functionalities that are soon planned to be implemented? It would be very useful as it would avoid double work (e.g. filling in paper forms and electronic forms) and allow data approval for decentralized data collection.

Thanks for the feedback,

Robin

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

14

Abyot: Thanks for the quick reaction, well done.

Robin: Your suggestions make sense and is something we have considered for a while. We have blueprints for image / document type data elements here:

https://blueprints.launchpad.net/dhis2/+spec/document-dataelement-attribute-type

https://blueprints.launchpad.net/dhis2/+spec/image-data-element-type

regards,

Lars

···

On Fri, Aug 1, 2014 at 11:14 AM, Abyot Gizaw abyota@gmail.com wrote:

Hi Robin,

Yes, the printing feature is now implemented - both in trunk and 2.16.

Thank you,

Abyot.

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

On Fri, Aug 1, 2014 at 10:58 AM, Robin Martens martens@sher.be wrote:

Hi Abyot,

Thanks for your reply. If it wouldn’t be too much of an effort, it would be great to have the printing option included somewhere soon (as it’s important for reducing workload for data collectors). The other ones are nice-to-have at the moment.

Regards,

Robin

From: Abyot Gizaw [mailto:abyota@gmail.com]

Sent: 01 August 2014 10:34

To: Robin Martens

Cc: dhis2-users@lists.launchpad.net

Subject: Re: [Dhis2-users] DHIS2 - Event capture forms printing & approving

Hi Robin,

Thank you for the feedback. As we pointed sometime back, we are in the process of overhauling the tracker module. In the process some features might be missing or not yet considered - the printing and approval you mentioned are two examples. I can quickly implement the printing feature and back-port it to 2.16. For approval and attachment, we need to take sometime to digest the requirement and come with a generic solution.

Thank you,

Abyot.

On Wed, Jul 30, 2014 at 10:14 AM, Robin Martens martens@sher.be wrote:

Dear all,

I didn’t find so far any possibility for form printing (blank and filled) and event data approval procedures, as they are available for entries for aggregated data through data sets.

Are these functionalities that are soon planned to be implemented? It would be very useful as it would avoid double work (e.g. filling in paper forms and electronic forms) and allow data approval for decentralized data collection.

Thanks for the feedback,

Robin

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

15

Thanks Lars, I’m looking forward to DHIS 2.17 then.

Regards,

Robin

···

From: Lars Helge Øverland [mailto:larshelge@gmail.com]

Sent: 05 August 2014 16:06

To: Abyot Gizaw

Cc: Robin Martens; dhis2-users@lists.launchpad.net

Subject: Re: [Dhis2-users] DHIS2 - Event capture forms printing & approving

Abyot: Thanks for the quick reaction, well done.

Robin: Your suggestions make sense and is something we have considered for a while. We have blueprints for image / document type data elements here:

https://blueprints.launchpad.net/dhis2/+spec/document-dataelement-attribute-type

https://blueprints.launchpad.net/dhis2/+spec/image-data-element-type

regards,

Lars

On Fri, Aug 1, 2014 at 11:14 AM, Abyot Gizaw abyota@gmail.com wrote:

Hi Robin,

Yes, the printing feature is now implemented - both in trunk and 2.16.

Thank you,

Abyot.

On Fri, Aug 1, 2014 at 10:58 AM, Robin Martens martens@sher.be wrote:

Hi Abyot,

Thanks for your reply. If it wouldn’t be too much of an effort, it would be great to have the printing option included somewhere soon (as it’s important for reducing workload for data collectors). The other ones are nice-to-have at the moment.

Regards,

Robin

From: Abyot Gizaw [mailto:abyota@gmail.com]

Sent: 01 August 2014 10:34

To: Robin Martens

Cc: dhis2-users@lists.launchpad.net

Subject: Re: [Dhis2-users] DHIS2 - Event capture forms printing & approving

Hi Robin,

Thank you for the feedback. As we pointed sometime back, we are in the process of overhauling the tracker module. In the process some features might be missing or not yet considered - the printing and approval you mentioned are two examples. I can quickly implement the printing feature and back-port it to 2.16. For approval and attachment, we need to take sometime to digest the requirement and come with a generic solution.

Thank you,

Abyot.

On Wed, Jul 30, 2014 at 10:14 AM, Robin Martens martens@sher.be wrote:

Dear all,

I didn’t find so far any possibility for form printing (blank and filled) and event data approval procedures, as they are available for entries for aggregated data through data sets.

Are these functionalities that are soon planned to be implemented? It would be very useful as it would avoid double work (e.g. filling in paper forms and electronic forms) and allow data approval for decentralized data collection.

Thanks for the feedback,

Robin

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp