Puzzling problem with datamart

Hi devs,

I am having some issues with some data elements not appearing in the data visualizer (and other modules which depend on datamart). Using the latest 2.11 file.

I get expected results from the analytics pivot tables, but nothing for the “Past twelve months” as a relative period in the DV.

Looked at the aggregateddata value table where this should appear, and see a some periods, but no monthly values (this is how the data is collected, by month).

SELECT startdate from period where periodid in (SELECT DISTINCT periodid from aggregateddatavalue where dataelementid = 80);

startdate | enddate

···

------------±-----------

2012-01-01 | 2012-12-31

2012-01-01 | 2012-03-31

2012-01-01 | 2012-06-30

2012-04-01 | 2012-06-30

2012-07-01 | 2012-09-30

2012-07-01 | 2012-12-31

2012-10-01 | 2012-12-31

2013-01-01 | 2013-06-30

2013-01-01 | 2013-03-31

2013-01-01 | 2013-12-31

(10 rows)

Looking at the data values table, just to be sure.

SELECT startdate,enddate from period where periodid in (SELECT DISTINCT periodid from datavalue where dataelementid = 80) ORDER BY startdate DESC;

startdate | enddate

------------±-----------

2013-03-01 | 2013-03-31

2013-02-01 | 2013-02-28

2013-01-01 | 2013-01-31

2012-12-01 | 2012-12-31

2012-11-01 | 2012-11-30

2012-10-01 | 2012-10-31

2012-09-01 | 2012-09-30

2012-08-01 | 2012-08-31

2012-07-01 | 2012-07-31

2012-06-01 | 2012-06-30

2012-05-01 | 2012-05-31

2012-04-01 | 2012-04-30

2012-03-01 | 2012-03-31

2012-02-01 | 2012-02-29

2012-01-01 | 2012-01-31

2011-12-01 | 2011-12-31

2011-11-01 | 2011-11-30

:

So, the raw data appears to be there.

I see no obvious errors in the logs. I did check the “Monthly” option for the data mart.

I find this really strange. Any ideas what may be going on? For other data elements collected in the same form, I do get monthly values.

Regards,

Jason

Hi Jason,

Are these data elements (with missing monthly vslues in datamart) also part of another dataset with e.g. quarterly collection frequency (period type)?

Only one period type per data element is supported in datamart and this is derived from the dataset’s period type. So if the data element is member of two data sets with different period types you will only get one of these period types in the datamart.

Ola

···

On 9 May 2013 18:02, “Jason Pickering” jason.p.pickering@gmail.com wrote:

Hi devs,

I am having some issues with some data elements not appearing in the data visualizer (and other modules which depend on datamart). Using the latest 2.11 file.

I get expected results from the analytics pivot tables, but nothing for the “Past twelve months” as a relative period in the DV.

Looked at the aggregateddata value table where this should appear, and see a some periods, but no monthly values (this is how the data is collected, by month).

SELECT startdate from period where periodid in (SELECT DISTINCT periodid from aggregateddatavalue where dataelementid = 80);

startdate | enddate

------------±-----------

2012-01-01 | 2012-12-31

2012-01-01 | 2012-03-31

2012-01-01 | 2012-06-30

2012-04-01 | 2012-06-30

2012-07-01 | 2012-09-30

2012-07-01 | 2012-12-31

2012-10-01 | 2012-12-31

2013-01-01 | 2013-06-30

2013-01-01 | 2013-03-31

2013-01-01 | 2013-12-31

(10 rows)

Looking at the data values table, just to be sure.

SELECT startdate,enddate from period where periodid in (SELECT DISTINCT periodid from datavalue where dataelementid = 80) ORDER BY startdate DESC;

startdate | enddate

------------±-----------

2013-03-01 | 2013-03-31

2013-02-01 | 2013-02-28

2013-01-01 | 2013-01-31

2012-12-01 | 2012-12-31

2012-11-01 | 2012-11-30

2012-10-01 | 2012-10-31

2012-09-01 | 2012-09-30

2012-08-01 | 2012-08-31

2012-07-01 | 2012-07-31

2012-06-01 | 2012-06-30

2012-05-01 | 2012-05-31

2012-04-01 | 2012-04-30

2012-03-01 | 2012-03-31

2012-02-01 | 2012-02-29

2012-01-01 | 2012-01-31

2011-12-01 | 2011-12-31

2011-11-01 | 2011-11-30

:

So, the raw data appears to be there.

I see no obvious errors in the logs. I did check the “Monthly” option for the data mart.

I find this really strange. Any ideas what may be going on? For other data elements collected in the same form, I do get monthly values.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Just to clarify:

You’ll get many aggregated period types in the datamart of course, but only one period type per data element can be used when looking/searching for raw data values to put into aggregation calculations during datamart export.

So your monthly data values might be ignored in the datamart export process if quarterly data values exist for the same data elements. And we use the dataset period type to know what data values to look for. At least that’s how I’ve understood this.

Lars, correct me if this is wrong.

Ola

···

On 11 May 2013 09:10, “Ola Hodne Titlestad” olati@ifi.uio.no wrote:

Hi Jason,

Are these data elements (with missing monthly vslues in datamart) also part of another dataset with e.g. quarterly collection frequency (period type)?

Only one period type per data element is supported in datamart and this is derived from the dataset’s period type. So if the data element is member of two data sets with different period types you will only get one of these period types in the datamart.

Ola


On 9 May 2013 18:02, “Jason Pickering” jason.p.pickering@gmail.com wrote:

Hi devs,

I am having some issues with some data elements not appearing in the data visualizer (and other modules which depend on datamart). Using the latest 2.11 file.

I get expected results from the analytics pivot tables, but nothing for the “Past twelve months” as a relative period in the DV.

Looked at the aggregateddata value table where this should appear, and see a some periods, but no monthly values (this is how the data is collected, by month).

SELECT startdate from period where periodid in (SELECT DISTINCT periodid from aggregateddatavalue where dataelementid = 80);

startdate | enddate

------------±-----------

2012-01-01 | 2012-12-31

2012-01-01 | 2012-03-31

2012-01-01 | 2012-06-30

2012-04-01 | 2012-06-30

2012-07-01 | 2012-09-30

2012-07-01 | 2012-12-31

2012-10-01 | 2012-12-31

2013-01-01 | 2013-06-30

2013-01-01 | 2013-03-31

2013-01-01 | 2013-12-31

(10 rows)

Looking at the data values table, just to be sure.

SELECT startdate,enddate from period where periodid in (SELECT DISTINCT periodid from datavalue where dataelementid = 80) ORDER BY startdate DESC;

startdate | enddate

------------±-----------

2013-03-01 | 2013-03-31

2013-02-01 | 2013-02-28

2013-01-01 | 2013-01-31

2012-12-01 | 2012-12-31

2012-11-01 | 2012-11-30

2012-10-01 | 2012-10-31

2012-09-01 | 2012-09-30

2012-08-01 | 2012-08-31

2012-07-01 | 2012-07-31

2012-06-01 | 2012-06-30

2012-05-01 | 2012-05-31

2012-04-01 | 2012-04-30

2012-03-01 | 2012-03-31

2012-02-01 | 2012-02-29

2012-01-01 | 2012-01-31

2011-12-01 | 2011-12-31

2011-11-01 | 2011-11-30

:

So, the raw data appears to be there.

I see no obvious errors in the logs. I did check the “Monthly” option for the data mart.

I find this really strange. Any ideas what may be going on? For other data elements collected in the same form, I do get monthly values.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

OK, so let me get this straight. Lets say I have a data element which belongs to a weekly and to a monthly dataset. Some orgunits report it weekly, others monthly. This is what is happening here in Zambia.

Which one wins? The weekly or the monthly?

It would seem that we would need to aggregate all of the data which was reported with a weekly period, to weekly, and to monthly, and all of the data which was reported monthly to monthly?

I will need to confirm whether or not this data element is part of another dataset.

What happens if the data element is not part of a data set at all?

Regards,

Jason

···

On Sat, May 11, 2013 at 9:17 AM, Ola Hodne Titlestad olati@ifi.uio.no wrote:

Just to clarify:

You’ll get many aggregated period types in the datamart of course, but only one period type per data element can be used when looking/searching for raw data values to put into aggregation calculations during datamart export.

So your monthly data values might be ignored in the datamart export process if quarterly data values exist for the same data elements. And we use the dataset period type to know what data values to look for. At least that’s how I’ve understood this.

Lars, correct me if this is wrong.

Ola


On 11 May 2013 09:10, “Ola Hodne Titlestad” olati@ifi.uio.no wrote:

Hi Jason,

Are these data elements (with missing monthly vslues in datamart) also part of another dataset with e.g. quarterly collection frequency (period type)?

Only one period type per data element is supported in datamart and this is derived from the dataset’s period type. So if the data element is member of two data sets with different period types you will only get one of these period types in the datamart.

Ola


On 9 May 2013 18:02, “Jason Pickering” jason.p.pickering@gmail.com wrote:

Hi devs,

I am having some issues with some data elements not appearing in the data visualizer (and other modules which depend on datamart). Using the latest 2.11 file.

I get expected results from the analytics pivot tables, but nothing for the “Past twelve months” as a relative period in the DV.

Looked at the aggregateddata value table where this should appear, and see a some periods, but no monthly values (this is how the data is collected, by month).

SELECT startdate from period where periodid in (SELECT DISTINCT periodid from aggregateddatavalue where dataelementid = 80);

startdate | enddate

------------±-----------

2012-01-01 | 2012-12-31

2012-01-01 | 2012-03-31

2012-01-01 | 2012-06-30

2012-04-01 | 2012-06-30

2012-07-01 | 2012-09-30

2012-07-01 | 2012-12-31

2012-10-01 | 2012-12-31

2013-01-01 | 2013-06-30

2013-01-01 | 2013-03-31

2013-01-01 | 2013-12-31

(10 rows)

Looking at the data values table, just to be sure.

SELECT startdate,enddate from period where periodid in (SELECT DISTINCT periodid from datavalue where dataelementid = 80) ORDER BY startdate DESC;

startdate | enddate

------------±-----------

2013-03-01 | 2013-03-31

2013-02-01 | 2013-02-28

2013-01-01 | 2013-01-31

2012-12-01 | 2012-12-31

2012-11-01 | 2012-11-30

2012-10-01 | 2012-10-31

2012-09-01 | 2012-09-30

2012-08-01 | 2012-08-31

2012-07-01 | 2012-07-31

2012-06-01 | 2012-06-30

2012-05-01 | 2012-05-31

2012-04-01 | 2012-04-30

2012-03-01 | 2012-03-31

2012-02-01 | 2012-02-29

2012-01-01 | 2012-01-31

2011-12-01 | 2011-12-31

2011-11-01 | 2011-11-30

:

So, the raw data appears to be there.

I see no obvious errors in the logs. I did check the “Monthly” option for the data mart.

I find this really strange. Any ideas what may be going on? For other data elements collected in the same form, I do get monthly values.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Jason,

the issue here is that “dis-aggregation” of data elements is not allowed. So for a “monthly” data element, weekly or daily data is ignored (data with higher frequency).

The frequency/period type of a data element is derived from the period type of the data sets which it is assigned to. When a data element is member of data sets with different period types, the behavior is undefined, ie. the “first” data set is just selected randomly. If there are no data sets, a yearly period type is assumed. We could of course change this behavior.

regards,

Lars

···

On Sat, May 11, 2013 at 2:15 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

OK, so let me get this straight. Lets say I have a data element which belongs to a weekly and to a monthly dataset. Some orgunits report it weekly, others monthly. This is what is happening here in Zambia.

Which one wins? The weekly or the monthly?

It would seem that we would need to aggregate all of the data which was reported with a weekly period, to weekly, and to monthly, and all of the data which was reported monthly to monthly?

I will need to confirm whether or not this data element is part of another dataset.

What happens if the data element is not part of a data set at all?

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Sat, May 11, 2013 at 9:17 AM, Ola Hodne Titlestad olati@ifi.uio.no wrote:

Just to clarify:

You’ll get many aggregated period types in the datamart of course, but only one period type per data element can be used when looking/searching for raw data values to put into aggregation calculations during datamart export.

So your monthly data values might be ignored in the datamart export process if quarterly data values exist for the same data elements. And we use the dataset period type to know what data values to look for. At least that’s how I’ve understood this.

Lars, correct me if this is wrong.

Ola


On 11 May 2013 09:10, “Ola Hodne Titlestad” olati@ifi.uio.no wrote:

Hi Jason,

Are these data elements (with missing monthly vslues in datamart) also part of another dataset with e.g. quarterly collection frequency (period type)?

Only one period type per data element is supported in datamart and this is derived from the dataset’s period type. So if the data element is member of two data sets with different period types you will only get one of these period types in the datamart.

Ola


On 9 May 2013 18:02, “Jason Pickering” jason.p.pickering@gmail.com wrote:

Hi devs,

I am having some issues with some data elements not appearing in the data visualizer (and other modules which depend on datamart). Using the latest 2.11 file.

I get expected results from the analytics pivot tables, but nothing for the “Past twelve months” as a relative period in the DV.

Looked at the aggregateddata value table where this should appear, and see a some periods, but no monthly values (this is how the data is collected, by month).

SELECT startdate from period where periodid in (SELECT DISTINCT periodid from aggregateddatavalue where dataelementid = 80);

startdate | enddate

------------±-----------

2012-01-01 | 2012-12-31

2012-01-01 | 2012-03-31

2012-01-01 | 2012-06-30

2012-04-01 | 2012-06-30

2012-07-01 | 2012-09-30

2012-07-01 | 2012-12-31

2012-10-01 | 2012-12-31

2013-01-01 | 2013-06-30

2013-01-01 | 2013-03-31

2013-01-01 | 2013-12-31

(10 rows)

Looking at the data values table, just to be sure.

SELECT startdate,enddate from period where periodid in (SELECT DISTINCT periodid from datavalue where dataelementid = 80) ORDER BY startdate DESC;

startdate | enddate

------------±-----------

2013-03-01 | 2013-03-31

2013-02-01 | 2013-02-28

2013-01-01 | 2013-01-31

2012-12-01 | 2012-12-31

2012-11-01 | 2012-11-30

2012-10-01 | 2012-10-31

2012-09-01 | 2012-09-30

2012-08-01 | 2012-08-31

2012-07-01 | 2012-07-31

2012-06-01 | 2012-06-30

2012-05-01 | 2012-05-31

2012-04-01 | 2012-04-30

2012-03-01 | 2012-03-31

2012-02-01 | 2012-02-29

2012-01-01 | 2012-01-31

2011-12-01 | 2011-12-31

2011-11-01 | 2011-11-30

:

So, the raw data appears to be there.

I see no obvious errors in the logs. I did check the “Monthly” option for the data mart.

I find this really strange. Any ideas what may be going on? For other data elements collected in the same form, I do get monthly values.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Lars,
This has never been an issue before. It has only recently ( I think in 2.11) been noticed. Perhaps it has just gone undetected until now.

I guess what is really puzzling is that it “seems” work fine for the new PivotTables. Anything which depends on the datamart does not work, so the logic does not seem to be consistent. What I mean my “seems”, is that I have not confirmed in the values which appear in the analytics actually sum to what they “should”, in this case that the weekly and monthly values should be summed. There of course could be other situations (similar to the “Aggregation levels”) in which weekly data (higher frequency) and monthly data (lower frequency) data should not be summed at all (which does not seem to be accounted for).

I guess what I am missing really is the relationship between a dataset and a data value, and perhaps a data element. One could think of a data warehousing situation when in fact there is not a dataset for a data element, unless this is implicitly required (as is perhaps indicated with the Data Integrity checks?) Perhaps it is never entered through the data entry screen at all, but simply imported from an external system. One could also think of situations where data elements are recorded at different frequencies. This is exactly what is happening here in Zambia, where certain data elements are reported weekly on some datasets, and monthly in others. In this case, they are for different orgunits, whereby certainly orgunits report weekly and others report monthly.

Regards,

Jason

···

On Sat, May 11, 2013 at 7:40 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Jason,

the issue here is that “dis-aggregation” of data elements is not allowed. So for a “monthly” data element, weekly or daily data is ignored (data with higher frequency).

The frequency/period type of a data element is derived from the period type of the data sets which it is assigned to. When a data element is member of data sets with different period types, the behavior is undefined, ie. the “first” data set is just selected randomly. If there are no data sets, a yearly period type is assumed. We could of course change this behavior.

regards,

Lars

On Sat, May 11, 2013 at 2:15 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

OK, so let me get this straight. Lets say I have a data element which belongs to a weekly and to a monthly dataset. Some orgunits report it weekly, others monthly. This is what is happening here in Zambia.

Which one wins? The weekly or the monthly?

It would seem that we would need to aggregate all of the data which was reported with a weekly period, to weekly, and to monthly, and all of the data which was reported monthly to monthly?

I will need to confirm whether or not this data element is part of another dataset.

What happens if the data element is not part of a data set at all?

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Sat, May 11, 2013 at 9:17 AM, Ola Hodne Titlestad olati@ifi.uio.no wrote:

Just to clarify:

You’ll get many aggregated period types in the datamart of course, but only one period type per data element can be used when looking/searching for raw data values to put into aggregation calculations during datamart export.

So your monthly data values might be ignored in the datamart export process if quarterly data values exist for the same data elements. And we use the dataset period type to know what data values to look for. At least that’s how I’ve understood this.

Lars, correct me if this is wrong.

Ola


On 11 May 2013 09:10, “Ola Hodne Titlestad” olati@ifi.uio.no wrote:

Hi Jason,

Are these data elements (with missing monthly vslues in datamart) also part of another dataset with e.g. quarterly collection frequency (period type)?

Only one period type per data element is supported in datamart and this is derived from the dataset’s period type. So if the data element is member of two data sets with different period types you will only get one of these period types in the datamart.

Ola


On 9 May 2013 18:02, “Jason Pickering” jason.p.pickering@gmail.com wrote:

Hi devs,

I am having some issues with some data elements not appearing in the data visualizer (and other modules which depend on datamart). Using the latest 2.11 file.

I get expected results from the analytics pivot tables, but nothing for the “Past twelve months” as a relative period in the DV.

Looked at the aggregateddata value table where this should appear, and see a some periods, but no monthly values (this is how the data is collected, by month).

SELECT startdate from period where periodid in (SELECT DISTINCT periodid from aggregateddatavalue where dataelementid = 80);

startdate | enddate

------------±-----------

2012-01-01 | 2012-12-31

2012-01-01 | 2012-03-31

2012-01-01 | 2012-06-30

2012-04-01 | 2012-06-30

2012-07-01 | 2012-09-30

2012-07-01 | 2012-12-31

2012-10-01 | 2012-12-31

2013-01-01 | 2013-06-30

2013-01-01 | 2013-03-31

2013-01-01 | 2013-12-31

(10 rows)

Looking at the data values table, just to be sure.

SELECT startdate,enddate from period where periodid in (SELECT DISTINCT periodid from datavalue where dataelementid = 80) ORDER BY startdate DESC;

startdate | enddate

------------±-----------

2013-03-01 | 2013-03-31

2013-02-01 | 2013-02-28

2013-01-01 | 2013-01-31

2012-12-01 | 2012-12-31

2012-11-01 | 2012-11-30

2012-10-01 | 2012-10-31

2012-09-01 | 2012-09-30

2012-08-01 | 2012-08-31

2012-07-01 | 2012-07-31

2012-06-01 | 2012-06-30

2012-05-01 | 2012-05-31

2012-04-01 | 2012-04-30

2012-03-01 | 2012-03-31

2012-02-01 | 2012-02-29

2012-01-01 | 2012-01-31

2011-12-01 | 2011-12-31

2011-11-01 | 2011-11-30

:

So, the raw data appears to be there.

I see no obvious errors in the logs. I did check the “Monthly” option for the data mart.

I find this really strange. Any ideas what may be going on? For other data elements collected in the same form, I do get monthly values.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Jason

···

On Sat, May 11, 2013 at 8:16 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Lars,
This has never been an issue before. It has only recently ( I think in 2.11) been noticed. Perhaps it has just gone undetected until now.

I guess what is really puzzling is that it “seems” work fine for the new PivotTables. Anything which depends on the datamart does not work, so the logic does not seem to be consistent. What I mean my “seems”, is that I have not confirmed in the values which appear in the analytics actually sum to what they “should”, in this case that the weekly and monthly values should be summed. There of course could be other situations (similar to the “Aggregation levels”) in which weekly data (higher frequency) and monthly data (lower frequency) data should not be summed at all (which does not seem to be accounted for).

I guess what I am missing really is the relationship between a dataset and a data value, and perhaps a data element. One could think of a data warehousing situation when in fact there is not a dataset for a data element, unless this is implicitly required (as is perhaps indicated with the Data Integrity checks?) Perhaps it is never entered through the data entry screen at all, but simply imported from an external system.

I remember there was a long discussion about this some years ago about whether to define collection frequency explicitly for each data element or implicitly through data set associations. The advantages of the latter are that data elements are not directly coupled with the frequency they are collected, and that this model itself allows for multiple capturing frequencies. The downside is like you point out that there has to be data sets for data elements, even if they are not collected through DHIS 2. So we went for the latter with these in mind.

One could also think of situations where data elements are recorded at different frequencies. This is exactly what is happening here in Zambia, where certain data elements are reported weekly on some datasets, and monthly in others. In this case, they are for different orgunits, whereby certainly orgunits report weekly and others report monthly.

Yes and we admittedly don’t support this well. Of course it could be done, but it would be complex. The issue is still that we cannot allow disaggregation. So if a branch of the org unit tree captures something weekly while the rest captures monthly, then we could produce weekly reports for org units within that branch, but not outside it. Something to consider.

Lars

Hi Lars,
Thanks for the answer. I suppose this is really something I never noticed. We have had data for a long time in the system for legacy data elements which was collected on a quarterly basis, and then later on on a monthly basis. It seemed to work fine, but this could have just been a fluke.

If this is the case, then it would seem to be the period type should actually be associated with the data element in this case. Attempting to derive it from it association with a data set would seem dubious. Not sure if a data integrity check is in place for this, but we should certainly warn the user that when a data elements belongs to two data sets of different frequencies, then weird stuff may happen.

Something just does not feel right about this though I must say. I had though all along that it was no problem to collect the data data element with different frequencies, similar to how it is possible to collect the same data element at differing orgunit levels. And it would seem that the analytics simply does not perform this check at all, which explains the difference I was noticing between the aggregated data from the datamart and the aggregated data from the analytics.

Regards,

Jason

···

On Tue, May 14, 2013 at 1:34 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Jason

On Sat, May 11, 2013 at 8:16 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Lars,
This has never been an issue before. It has only recently ( I think in 2.11) been noticed. Perhaps it has just gone undetected until now.

I guess what is really puzzling is that it “seems” work fine for the new PivotTables. Anything which depends on the datamart does not work, so the logic does not seem to be consistent. What I mean my “seems”, is that I have not confirmed in the values which appear in the analytics actually sum to what they “should”, in this case that the weekly and monthly values should be summed. There of course could be other situations (similar to the “Aggregation levels”) in which weekly data (higher frequency) and monthly data (lower frequency) data should not be summed at all (which does not seem to be accounted for).

I guess what I am missing really is the relationship between a dataset and a data value, and perhaps a data element. One could think of a data warehousing situation when in fact there is not a dataset for a data element, unless this is implicitly required (as is perhaps indicated with the Data Integrity checks?) Perhaps it is never entered through the data entry screen at all, but simply imported from an external system.

I remember there was a long discussion about this some years ago about whether to define collection frequency explicitly for each data element or implicitly through data set associations. The advantages of the latter are that data elements are not directly coupled with the frequency they are collected, and that this model itself allows for multiple capturing frequencies. The downside is like you point out that there has to be data sets for data elements, even if they are not collected through DHIS 2. So we went for the latter with these in mind.

One could also think of situations where data elements are recorded at different frequencies. This is exactly what is happening here in Zambia, where certain data elements are reported weekly on some datasets, and monthly in others. In this case, they are for different orgunits, whereby certainly orgunits report weekly and others report monthly.

Yes and we admittedly don’t support this well. Of course it could be done, but it would be complex. The issue is still that we cannot allow disaggregation. So if a branch of the org unit tree captures something weekly while the rest captures monthly, then we could produce weekly reports for org units within that branch, but not outside it. Something to consider.

Lars