Setting MinMax Values

Hi,

What i have observed, Min-max value is for a particular data element and for a particular org-unit.so each facility should have its own min-max values for each data element…In our case we want to set min max values for a particular data element to be applied to all facilities and periods even future periods. I don’t have to open the data entry form and double click in each field to set it …Is it possible to do this once not manually?? I have tried to use min max generation its not working…

Hi,
I think you could make a Validation rule for this, where you specify that the data element should be less than your maximum value.

left side: [data element]
operator: <
right side [max value]

Olav

···

19. sep. 2013 kl. 10:22 skrev conrad muyepa <cmuyepa@gmail.com>:

Hi,

What i have observed, Min-max value is for a particular data element and for a particular org-unit.so each facility should have its own min-max values for each data element...In our case we want to set min max values for a particular data element to be applied to all facilities and periods even future periods. I don't have to open the data entry form and double click in each field to set it ...Is it possible to do this once not manually?? I have tried to use min max generation its not working..

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

Olav’s approach might work, but would not allow you to conduct a min-max analysis (although you could do it through the validation rules analysis in this way). The advantage with the min/max is that it may be a bit more intuitive, especially with the data entry page and the graph which displays historical values.

Anyway, you could adapt this SQL query for your own use, which we have used in the past for a very similar situation.

DELETE FROM minmaxdataelement where dataelementid = 94442 and categoryoptioncomboid = 13 and sourceid IN (SELECT DISTINCT idlevel4 from _orgunitstructure where idlevel4 IS NOT NULL);

INSERT INTO minmaxdataelement

SELECT nextval(‘hibernate_sequence’::regclass) as minmaxdataelementid, a.organisationunitid as sourceid,

94442 as dataelementid, 13 as categoryoptioncomboid,

0 as minimumvalue, 10 as maximumvalue, TRUE as generated

FROM organisationunit a

where a.organisationunitid in ( SELECT DISTINCT idlevel4 from _orgunitstructure where idlevel4 IS NOT NULL);

The first query will get rid of all min/max values of the specified dataelement/categoryoptioncombo/source combination.

The second query will insert the values.

You should adjust the dataelementid/categoryoptioncomboids to suit your needs. The highlighted part of the query is going to return organisationunit IDs, and in this case, I took all level 4 orgunits. Of course, this might need to be adjusted to suit your needs (different orgunit level, restricting by membership in a dataset or orgunit group, or something similar).

Hope this helps.

Regards,

Jason

···

On Thu, Sep 19, 2013 at 10:26 AM, Olav Poppe olav.poppe@me.com wrote:

Hi,

I think you could make a Validation rule for this, where you specify that the data element should be less than your maximum value.

left side: [data element]

operator: <

right side [max value]

Olav

  1. sep. 2013 kl. 10:22 skrev conrad muyepa cmuyepa@gmail.com:

Hi,

What i have observed, Min-max value is for a particular data element and for a particular org-unit.so each facility should have its own min-max values for each data element…In our case we want to set min max values for a particular data element to be applied to all facilities and periods even future periods. I don’t have to open the data entry form and double click in each field to set it …Is it possible to do this once not manually?? I have tried to use min max generation its not working…


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars

I agree with Lars

Lungo

···

From: Lars Helge Øverland larshelge@gmail.com
To: Jason Pickering jason.p.pickering@gmail.com
Cc: dhis2-devs dhis2-devs@lists.launchpad.net
Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

···

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com
To: Jason Pickering <jason.p.pickering@gmail.com >
Cc: dhis2-devs dhis2-devs@lists.launchpad.net
Sent: Thursday, September 19, 2013 2:43 PM

Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

···

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering
Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs
Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com
To: Jason Pickering jason.p.pickering@gmail.com
Cc: dhis2-devs dhis2-devs@lists.launchpad.net
Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Hi Dapo,
If you were to generate the min-max values automatically, insert them into the database, and an authorized user was to change them, then they would be overwritten.

You can automatically generate these values through the data administration module by dataset and orgunit in Data Administration->Min-Max Value Generation . The problem is you often end up with a lot of dubious values, such as negative numbers. This is a result of the Gaussian distribution model which is applied (without taking any skewness into effect). I have attached a plot from a real DHIS2 database, which show sthat the distribution of many in DHIS2 are skewed left or right (more often to the right) . Use of an unbounded, normal distribution for calculation of the min-max is therefore not really appropriate. The standard deviation approach is also highly sensitive to outliers, which also creates problems for the outlier analysis if this approach is used. Anyway, I digress. Just wanted to highlight that the in-built functions for calculation of the min-max are rather primitive, and may not always lead to good results.

I am not so sure about the ability to restrict the time periods for which the mix-max is generated. I think you would end up with far too few data points to make a meaningful standard deviation. You are really looking for the extremes with the min-max, and restricting to such a a narrow time interval would probably not give you these extremes.

Regards,

Jason

image

···

On Thu, Sep 19, 2013 at 11:47 PM, Dapo Adejumo dapo_adejumo@yahoo.com wrote:

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering

Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs

Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com

To: Jason Pickering jason.p.pickering@gmail.com

Cc: dhis2-devs dhis2-devs@lists.launchpad.net

Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Thank you for your ideas and support… Many thanks to you all.

···

On Fri, Sep 20, 2013 at 7:30 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Dapo,
If you were to generate the min-max values automatically, insert them into the database, and an authorized user was to change them, then they would be overwritten.

You can automatically generate these values through the data administration module by dataset and orgunit in Data Administration->Min-Max Value Generation . The problem is you often end up with a lot of dubious values, such as negative numbers. This is a result of the Gaussian distribution model which is applied (without taking any skewness into effect). I have attached a plot from a real DHIS2 database, which show sthat the distribution of many in DHIS2 are skewed left or right (more often to the right) . Use of an unbounded, normal distribution for calculation of the min-max is therefore not really appropriate. The standard deviation approach is also highly sensitive to outliers, which also creates problems for the outlier analysis if this approach is used. Anyway, I digress. Just wanted to highlight that the in-built functions for calculation of the min-max are rather primitive, and may not always lead to good results.

I am not so sure about the ability to restrict the time periods for which the mix-max is generated. I think you would end up with far too few data points to make a meaningful standard deviation. You are really looking for the extremes with the min-max, and restricting to such a a narrow time interval would probably not give you these extremes.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Sep 19, 2013 at 11:47 PM, Dapo Adejumo dapo_adejumo@yahoo.com wrote:

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering

Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs

Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com

To: Jason Pickering jason.p.pickering@gmail.com

Cc: dhis2-devs dhis2-devs@lists.launchpad.net

Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Hi all,

To introduce a variation on this theme of generating min-max values: I expected that if you set data element number type to Zero of Positive Integer this should alter the distribution (in this case skew to the right as Jason put it) and avoid the negative minimum values. From the tests I have done, this is not happening.

The negative mim values are a real pain, anyone with a solution yet?

Regards,

Busoye

···

On Thu, Sep 19, 2013 at 11:47 PM, Dapo Adejumo dapo_adejumo@yahoo.com wrote:

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering

Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs

Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com

To: Jason Pickering jason.p.pickering@gmail.com

Cc: dhis2-devs dhis2-devs@lists.launchpad.net

Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Hi Dapo.,

This is just a design issue with DHIS2. There needs to be some better statistical methods employed to generate the min max, or a means to specify that the minimum should be zero for all values. There is no linkage between the value type and the method which the mix-max uses to generate its values. In our case, we use a combination of R and SQL to calculate “better” values on a regular basis. A bit of a hack, but the current methods are simply not working for us either.

Regards,

Jason

···

On Wed, Nov 6, 2013 at 12:23 AM, Adebusoye Anifalaje busoye@hisp.org wrote:

Hi all,

To introduce a variation on this theme of generating min-max values: I expected that if you set data element number type to Zero of Positive Integer this should alter the distribution (in this case skew to the right as Jason put it) and avoid the negative minimum values. From the tests I have done, this is not happening.

The negative mim values are a real pain, anyone with a solution yet?

Regards,

Busoye

On 31 May 2013, at 14:53, Gulam Khan gm-khan@criticalsoftware.com wrote:

Hi Dapo,
If you were to generate the min-max values automatically, insert them into the database, and an authorized user was to change them, then they would be overwritten.

You can automatically generate these values through the data administration module by dataset and orgunit in Data Administration->Min-Max Value Generation . The problem is you often end up with a lot of dubious values, such as negative numbers. This is a result of the Gaussian distribution model which is applied (without taking any skewness into effect). I have attached a plot from a real DHIS2 database, which show sthat the distribution of many in DHIS2 are skewed left or right (more often to the right) . Use of an unbounded, normal distribution for calculation of the min-max is therefore not really appropriate. The standard deviation approach is also highly sensitive to outliers, which also creates problems for the outlier analysis if this approach is used. Anyway, I digress. Just wanted to highlight that the in-built functions for calculation of the min-max are rather primitive, and may not always lead to good results.

I am not so sure about the ability to restrict the time periods for which the mix-max is generated. I think you would end up with far too few data points to make a meaningful standard deviation. You are really looking for the extremes with the min-max, and restricting to such a a narrow time interval would probably not give you these extremes.

Regards,

Jason

<skewness.png>_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Sep 19, 2013 at 11:47 PM, Dapo Adejumo dapo_adejumo@yahoo.com wrote:

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering

Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs

Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com

To: Jason Pickering jason.p.pickering@gmail.com

Cc: dhis2-devs dhis2-devs@lists.launchpad.net

Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Hi,

the issue with negative values occurring as min-values for data elements of type positive / zero-or-positive number, and positive values occurring as max-values for data elements of type negative number have now been fixed in trunk and 2.13.

regards,

Lars

···

On Wed, Nov 6, 2013 at 12:00 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Dapo.,

This is just a design issue with DHIS2. There needs to be some better statistical methods employed to generate the min max, or a means to specify that the minimum should be zero for all values. There is no linkage between the value type and the method which the mix-max uses to generate its values. In our case, we use a combination of R and SQL to calculate “better” values on a regular basis. A bit of a hack, but the current methods are simply not working for us either.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Wed, Nov 6, 2013 at 12:23 AM, Adebusoye Anifalaje busoye@hisp.org wrote:

Hi all,

To introduce a variation on this theme of generating min-max values: I expected that if you set data element number type to Zero of Positive Integer this should alter the distribution (in this case skew to the right as Jason put it) and avoid the negative minimum values. From the tests I have done, this is not happening.

The negative mim values are a real pain, anyone with a solution yet?

Regards,

Busoye

On 31 May 2013, at 14:53, Gulam Khan gm-khan@criticalsoftware.com wrote:

Hi Dapo,
If you were to generate the min-max values automatically, insert them into the database, and an authorized user was to change them, then they would be overwritten.

You can automatically generate these values through the data administration module by dataset and orgunit in Data Administration->Min-Max Value Generation . The problem is you often end up with a lot of dubious values, such as negative numbers. This is a result of the Gaussian distribution model which is applied (without taking any skewness into effect). I have attached a plot from a real DHIS2 database, which show sthat the distribution of many in DHIS2 are skewed left or right (more often to the right) . Use of an unbounded, normal distribution for calculation of the min-max is therefore not really appropriate. The standard deviation approach is also highly sensitive to outliers, which also creates problems for the outlier analysis if this approach is used. Anyway, I digress. Just wanted to highlight that the in-built functions for calculation of the min-max are rather primitive, and may not always lead to good results.

I am not so sure about the ability to restrict the time periods for which the mix-max is generated. I think you would end up with far too few data points to make a meaningful standard deviation. You are really looking for the extremes with the min-max, and restricting to such a a narrow time interval would probably not give you these extremes.

Regards,

Jason

<skewness.png>_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Sep 19, 2013 at 11:47 PM, Dapo Adejumo dapo_adejumo@yahoo.com wrote:

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering

Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs

Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com

To: Jason Pickering jason.p.pickering@gmail.com

Cc: dhis2-devs dhis2-devs@lists.launchpad.net

Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Just tried on Version: 2.13 Build revision: 12888. Still generating negative values! Deleted all the values from the minmaxdataelement table and ran the min-max value generation again but negative values still coming through for some minimum numbers (example attached).

Am I missing something?

Busoye

···

On Wed, Nov 6, 2013 at 12:00 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Dapo.,

This is just a design issue with DHIS2. There needs to be some better statistical methods employed to generate the min max, or a means to specify that the minimum should be zero for all values. There is no linkage between the value type and the method which the mix-max uses to generate its values. In our case, we use a combination of R and SQL to calculate “better” values on a regular basis. A bit of a hack, but the current methods are simply not working for us either.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Wed, Nov 6, 2013 at 12:23 AM, Adebusoye Anifalaje busoye@hisp.org wrote:

Hi all,

To introduce a variation on this theme of generating min-max values: I expected that if you set data element number type to Zero of Positive Integer this should alter the distribution (in this case skew to the right as Jason put it) and avoid the negative minimum values. From the tests I have done, this is not happening.

The negative mim values are a real pain, anyone with a solution yet?

Regards,

Busoye

On 31 May 2013, at 14:53, Gulam Khan gm-khan@criticalsoftware.com wrote:

Hi Dapo,
If you were to generate the min-max values automatically, insert them into the database, and an authorized user was to change them, then they would be overwritten.

You can automatically generate these values through the data administration module by dataset and orgunit in Data Administration->Min-Max Value Generation . The problem is you often end up with a lot of dubious values, such as negative numbers. This is a result of the Gaussian distribution model which is applied (without taking any skewness into effect). I have attached a plot from a real DHIS2 database, which show sthat the distribution of many in DHIS2 are skewed left or right (more often to the right) . Use of an unbounded, normal distribution for calculation of the min-max is therefore not really appropriate. The standard deviation approach is also highly sensitive to outliers, which also creates problems for the outlier analysis if this approach is used. Anyway, I digress. Just wanted to highlight that the in-built functions for calculation of the min-max are rather primitive, and may not always lead to good results.

I am not so sure about the ability to restrict the time periods for which the mix-max is generated. I think you would end up with far too few data points to make a meaningful standard deviation. You are really looking for the extremes with the min-max, and restricting to such a a narrow time interval would probably not give you these extremes.

Regards,

Jason

<skewness.png>_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Sep 19, 2013 at 11:47 PM, Dapo Adejumo dapo_adejumo@yahoo.com wrote:

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering

Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs

Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com

To: Jason Pickering jason.p.pickering@gmail.com

Cc: dhis2-devs dhis2-devs@lists.launchpad.net

Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Hi,

Just checking, did you remember to actually set number type to positive/zero or positive number?

Min-Max Negative Values.tiff (155 KB)

···

On Wed, Nov 6, 2013 at 12:00 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Dapo.,

This is just a design issue with DHIS2. There needs to be some better statistical methods employed to generate the min max, or a means to specify that the minimum should be zero for all values. There is no linkage between the value type and the method which the mix-max uses to generate its values. In our case, we use a combination of R and SQL to calculate “better” values on a regular basis. A bit of a hack, but the current methods are simply not working for us either.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Wed, Nov 6, 2013 at 12:23 AM, Adebusoye Anifalaje busoye@hisp.org wrote:

Hi all,

To introduce a variation on this theme of generating min-max values: I expected that if you set data element number type to Zero of Positive Integer this should alter the distribution (in this case skew to the right as Jason put it) and avoid the negative minimum values. From the tests I have done, this is not happening.

The negative mim values are a real pain, anyone with a solution yet?

Regards,

Busoye

On 31 May 2013, at 14:53, Gulam Khan gm-khan@criticalsoftware.com wrote:

Hi Dapo,
If you were to generate the min-max values automatically, insert them into the database, and an authorized user was to change them, then they would be overwritten.

You can automatically generate these values through the data administration module by dataset and orgunit in Data Administration->Min-Max Value Generation . The problem is you often end up with a lot of dubious values, such as negative numbers. This is a result of the Gaussian distribution model which is applied (without taking any skewness into effect). I have attached a plot from a real DHIS2 database, which show sthat the distribution of many in DHIS2 are skewed left or right (more often to the right) . Use of an unbounded, normal distribution for calculation of the min-max is therefore not really appropriate. The standard deviation approach is also highly sensitive to outliers, which also creates problems for the outlier analysis if this approach is used. Anyway, I digress. Just wanted to highlight that the in-built functions for calculation of the min-max are rather primitive, and may not always lead to good results.

I am not so sure about the ability to restrict the time periods for which the mix-max is generated. I think you would end up with far too few data points to make a meaningful standard deviation. You are really looking for the extremes with the min-max, and restricting to such a a narrow time interval would probably not give you these extremes.

Regards,

Jason

<skewness.png>_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Sep 19, 2013 at 11:47 PM, Dapo Adejumo dapo_adejumo@yahoo.com wrote:

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering

Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs

Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com

To: Jason Pickering jason.p.pickering@gmail.com

Cc: dhis2-devs dhis2-devs@lists.launchpad.net

Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Yes.

Min-Max Negative Values.tiff (155 KB)

···

On Wed, Nov 6, 2013 at 12:00 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Dapo.,

This is just a design issue with DHIS2. There needs to be some better statistical methods employed to generate the min max, or a means to specify that the minimum should be zero for all values. There is no linkage between the value type and the method which the mix-max uses to generate its values. In our case, we use a combination of R and SQL to calculate “better” values on a regular basis. A bit of a hack, but the current methods are simply not working for us either.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Wed, Nov 6, 2013 at 12:23 AM, Adebusoye Anifalaje busoye@hisp.org wrote:

Hi all,

To introduce a variation on this theme of generating min-max values: I expected that if you set data element number type to Zero of Positive Integer this should alter the distribution (in this case skew to the right as Jason put it) and avoid the negative minimum values. From the tests I have done, this is not happening.

The negative mim values are a real pain, anyone with a solution yet?

Regards,

Busoye

On 31 May 2013, at 14:53, Gulam Khan gm-khan@criticalsoftware.com wrote:

Hi Dapo,
If you were to generate the min-max values automatically, insert them into the database, and an authorized user was to change them, then they would be overwritten.

You can automatically generate these values through the data administration module by dataset and orgunit in Data Administration->Min-Max Value Generation . The problem is you often end up with a lot of dubious values, such as negative numbers. This is a result of the Gaussian distribution model which is applied (without taking any skewness into effect). I have attached a plot from a real DHIS2 database, which show sthat the distribution of many in DHIS2 are skewed left or right (more often to the right) . Use of an unbounded, normal distribution for calculation of the min-max is therefore not really appropriate. The standard deviation approach is also highly sensitive to outliers, which also creates problems for the outlier analysis if this approach is used. Anyway, I digress. Just wanted to highlight that the in-built functions for calculation of the min-max are rather primitive, and may not always lead to good results.

I am not so sure about the ability to restrict the time periods for which the mix-max is generated. I think you would end up with far too few data points to make a meaningful standard deviation. You are really looking for the extremes with the min-max, and restricting to such a a narrow time interval would probably not give you these extremes.

Regards,

Jason

<skewness.png>_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Sep 19, 2013 at 11:47 PM, Dapo Adejumo dapo_adejumo@yahoo.com wrote:

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering

Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs

Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com

To: Jason Pickering jason.p.pickering@gmail.com

Cc: dhis2-devs dhis2-devs@lists.launchpad.net

Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Yes, set the data elements to Zero or positive integer

Min-Max Negative Values.tiff (155 KB)

···

On Wed, Nov 6, 2013 at 12:00 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Dapo.,

This is just a design issue with DHIS2. There needs to be some better statistical methods employed to generate the min max, or a means to specify that the minimum should be zero for all values. There is no linkage between the value type and the method which the mix-max uses to generate its values. In our case, we use a combination of R and SQL to calculate “better” values on a regular basis. A bit of a hack, but the current methods are simply not working for us either.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Wed, Nov 6, 2013 at 12:23 AM, Adebusoye Anifalaje busoye@hisp.org wrote:

Hi all,

To introduce a variation on this theme of generating min-max values: I expected that if you set data element number type to Zero of Positive Integer this should alter the distribution (in this case skew to the right as Jason put it) and avoid the negative minimum values. From the tests I have done, this is not happening.

The negative mim values are a real pain, anyone with a solution yet?

Regards,

Busoye

On 31 May 2013, at 14:53, Gulam Khan gm-khan@criticalsoftware.com wrote:

Hi Dapo,
If you were to generate the min-max values automatically, insert them into the database, and an authorized user was to change them, then they would be overwritten.

You can automatically generate these values through the data administration module by dataset and orgunit in Data Administration->Min-Max Value Generation . The problem is you often end up with a lot of dubious values, such as negative numbers. This is a result of the Gaussian distribution model which is applied (without taking any skewness into effect). I have attached a plot from a real DHIS2 database, which show sthat the distribution of many in DHIS2 are skewed left or right (more often to the right) . Use of an unbounded, normal distribution for calculation of the min-max is therefore not really appropriate. The standard deviation approach is also highly sensitive to outliers, which also creates problems for the outlier analysis if this approach is used. Anyway, I digress. Just wanted to highlight that the in-built functions for calculation of the min-max are rather primitive, and may not always lead to good results.

I am not so sure about the ability to restrict the time periods for which the mix-max is generated. I think you would end up with far too few data points to make a meaningful standard deviation. You are really looking for the extremes with the min-max, and restricting to such a a narrow time interval would probably not give you these extremes.

Regards,

Jason

<skewness.png>_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Sep 19, 2013 at 11:47 PM, Dapo Adejumo dapo_adejumo@yahoo.com wrote:

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering

Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs

Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com

To: Jason Pickering jason.p.pickering@gmail.com

Cc: dhis2-devs dhis2-devs@lists.launchpad.net

Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Ops sorry back-port didn’t go through, fixed it now, download again in 30 min.

Lars

Min-Max Negative Values.tiff (155 KB)

···

On Tue, Dec 3, 2013 at 10:08 AM, Busoye Anifalaje busoye@hisp.org wrote:

Yes, set the data elements to Zero or positive integer

On Dec 3, 2013 7:47 AM, “Lars Helge Øverland” larshelge@gmail.com wrote:

Hi,

Just checking, did you remember to actually set number type to positive/zero or positive number?

On Dec 3, 2013 12:08 AM, “Adebusoye Anifalaje” busoye@hisp.org wrote:

Just tried on Version: 2.13 Build revision: 12888. Still generating negative values! Deleted all the values from the minmaxdataelement table and ran the min-max value generation again but negative values still coming through for some minimum numbers (example attached).

Am I missing something?

Busoye

On 2 Dec 2013, at 18:53, Lars Helge Øverland larshelge@gmail.com wrote:

Hi,

the issue with negative values occurring as min-values for data elements of type positive / zero-or-positive number, and positive values occurring as max-values for data elements of type negative number have now been fixed in trunk and 2.13.

regards,

Lars

On Wed, Nov 6, 2013 at 12:00 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Dapo.,

This is just a design issue with DHIS2. There needs to be some better statistical methods employed to generate the min max, or a means to specify that the minimum should be zero for all values. There is no linkage between the value type and the method which the mix-max uses to generate its values. In our case, we use a combination of R and SQL to calculate “better” values on a regular basis. A bit of a hack, but the current methods are simply not working for us either.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Wed, Nov 6, 2013 at 12:23 AM, Adebusoye Anifalaje busoye@hisp.org wrote:

Hi all,

To introduce a variation on this theme of generating min-max values: I expected that if you set data element number type to Zero of Positive Integer this should alter the distribution (in this case skew to the right as Jason put it) and avoid the negative minimum values. From the tests I have done, this is not happening.

The negative mim values are a real pain, anyone with a solution yet?

Regards,

Busoye

On 31 May 2013, at 14:53, Gulam Khan gm-khan@criticalsoftware.com wrote:

Hi Dapo,
If you were to generate the min-max values automatically, insert them into the database, and an authorized user was to change them, then they would be overwritten.

You can automatically generate these values through the data administration module by dataset and orgunit in Data Administration->Min-Max Value Generation . The problem is you often end up with a lot of dubious values, such as negative numbers. This is a result of the Gaussian distribution model which is applied (without taking any skewness into effect). I have attached a plot from a real DHIS2 database, which show sthat the distribution of many in DHIS2 are skewed left or right (more often to the right) . Use of an unbounded, normal distribution for calculation of the min-max is therefore not really appropriate. The standard deviation approach is also highly sensitive to outliers, which also creates problems for the outlier analysis if this approach is used. Anyway, I digress. Just wanted to highlight that the in-built functions for calculation of the min-max are rather primitive, and may not always lead to good results.

I am not so sure about the ability to restrict the time periods for which the mix-max is generated. I think you would end up with far too few data points to make a meaningful standard deviation. You are really looking for the extremes with the min-max, and restricting to such a a narrow time interval would probably not give you these extremes.

Regards,

Jason

<skewness.png>_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Sep 19, 2013 at 11:47 PM, Dapo Adejumo dapo_adejumo@yahoo.com wrote:

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering

Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs

Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com

To: Jason Pickering jason.p.pickering@gmail.com

Cc: dhis2-devs dhis2-devs@lists.launchpad.net

Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Thanks Lars,

Works without the negative minimum values.

Busoye

···

On Tue, Dec 3, 2013 at 10:08 AM, Busoye Anifalaje busoye@hisp.org wrote:

Yes, set the data elements to Zero or positive integer

On Dec 3, 2013 7:47 AM, “Lars Helge Øverland” larshelge@gmail.com wrote:

Hi,

Just checking, did you remember to actually set number type to positive/zero or positive number?

On Dec 3, 2013 12:08 AM, “Adebusoye Anifalaje” busoye@hisp.org wrote:

Just tried on Version: 2.13 Build revision: 12888. Still generating negative values! Deleted all the values from the minmaxdataelement table and ran the min-max value generation again but negative values still coming through for some minimum numbers (example attached).

Am I missing something?

Busoye

On 2 Dec 2013, at 18:53, Lars Helge Øverland larshelge@gmail.com wrote:

Hi,

the issue with negative values occurring as min-values for data elements of type positive / zero-or-positive number, and positive values occurring as max-values for data elements of type negative number have now been fixed in trunk and 2.13.

regards,

Lars

On Wed, Nov 6, 2013 at 12:00 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Dapo.,

This is just a design issue with DHIS2. There needs to be some better statistical methods employed to generate the min max, or a means to specify that the minimum should be zero for all values. There is no linkage between the value type and the method which the mix-max uses to generate its values. In our case, we use a combination of R and SQL to calculate “better” values on a regular basis. A bit of a hack, but the current methods are simply not working for us either.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Wed, Nov 6, 2013 at 12:23 AM, Adebusoye Anifalaje busoye@hisp.org wrote:

Hi all,

To introduce a variation on this theme of generating min-max values: I expected that if you set data element number type to Zero of Positive Integer this should alter the distribution (in this case skew to the right as Jason put it) and avoid the negative minimum values. From the tests I have done, this is not happening.

The negative mim values are a real pain, anyone with a solution yet?

Regards,

Busoye

On 31 May 2013, at 14:53, Gulam Khan gm-khan@criticalsoftware.com wrote:

Hi Dapo,
If you were to generate the min-max values automatically, insert them into the database, and an authorized user was to change them, then they would be overwritten.

You can automatically generate these values through the data administration module by dataset and orgunit in Data Administration->Min-Max Value Generation . The problem is you often end up with a lot of dubious values, such as negative numbers. This is a result of the Gaussian distribution model which is applied (without taking any skewness into effect). I have attached a plot from a real DHIS2 database, which show sthat the distribution of many in DHIS2 are skewed left or right (more often to the right) . Use of an unbounded, normal distribution for calculation of the min-max is therefore not really appropriate. The standard deviation approach is also highly sensitive to outliers, which also creates problems for the outlier analysis if this approach is used. Anyway, I digress. Just wanted to highlight that the in-built functions for calculation of the min-max are rather primitive, and may not always lead to good results.

I am not so sure about the ability to restrict the time periods for which the mix-max is generated. I think you would end up with far too few data points to make a meaningful standard deviation. You are really looking for the extremes with the min-max, and restricting to such a a narrow time interval would probably not give you these extremes.

Regards,

Jason

<skewness.png>_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Sep 19, 2013 at 11:47 PM, Dapo Adejumo dapo_adejumo@yahoo.com wrote:

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering

Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs

Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com

To: Jason Pickering jason.p.pickering@gmail.com

Cc: dhis2-devs dhis2-devs@lists.launchpad.net

Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Great.

···

On Tue, Dec 3, 2013 at 12:27 PM, Adebusoye Anifalaje busoye@hisp.org wrote:

Thanks Lars,

Works without the negative minimum values.

Busoye

On 3 Dec 2013, at 09:18, Lars Helge Øverland larshelge@gmail.com wrote:

Ops sorry back-port didn’t go through, fixed it now, download again in 30 min.

Lars

On Tue, Dec 3, 2013 at 10:08 AM, Busoye Anifalaje busoye@hisp.org wrote:

Yes, set the data elements to Zero or positive integer

On Dec 3, 2013 7:47 AM, “Lars Helge Øverland” larshelge@gmail.com wrote:

Hi,

Just checking, did you remember to actually set number type to positive/zero or positive number?

On Dec 3, 2013 12:08 AM, “Adebusoye Anifalaje” busoye@hisp.org wrote:

Just tried on Version: 2.13 Build revision: 12888. Still generating negative values! Deleted all the values from the minmaxdataelement table and ran the min-max value generation again but negative values still coming through for some minimum numbers (example attached).

Am I missing something?

Busoye

On 2 Dec 2013, at 18:53, Lars Helge Øverland larshelge@gmail.com wrote:

Hi,

the issue with negative values occurring as min-values for data elements of type positive / zero-or-positive number, and positive values occurring as max-values for data elements of type negative number have now been fixed in trunk and 2.13.

regards,

Lars

On Wed, Nov 6, 2013 at 12:00 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Dapo.,

This is just a design issue with DHIS2. There needs to be some better statistical methods employed to generate the min max, or a means to specify that the minimum should be zero for all values. There is no linkage between the value type and the method which the mix-max uses to generate its values. In our case, we use a combination of R and SQL to calculate “better” values on a regular basis. A bit of a hack, but the current methods are simply not working for us either.

Regards,

Jason


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Wed, Nov 6, 2013 at 12:23 AM, Adebusoye Anifalaje busoye@hisp.org wrote:

Hi all,

To introduce a variation on this theme of generating min-max values: I expected that if you set data element number type to Zero of Positive Integer this should alter the distribution (in this case skew to the right as Jason put it) and avoid the negative minimum values. From the tests I have done, this is not happening.

The negative mim values are a real pain, anyone with a solution yet?

Regards,

Busoye

On 31 May 2013, at 14:53, Gulam Khan gm-khan@criticalsoftware.com wrote:

Hi Dapo,
If you were to generate the min-max values automatically, insert them into the database, and an authorized user was to change them, then they would be overwritten.

You can automatically generate these values through the data administration module by dataset and orgunit in Data Administration->Min-Max Value Generation . The problem is you often end up with a lot of dubious values, such as negative numbers. This is a result of the Gaussian distribution model which is applied (without taking any skewness into effect). I have attached a plot from a real DHIS2 database, which show sthat the distribution of many in DHIS2 are skewed left or right (more often to the right) . Use of an unbounded, normal distribution for calculation of the min-max is therefore not really appropriate. The standard deviation approach is also highly sensitive to outliers, which also creates problems for the outlier analysis if this approach is used. Anyway, I digress. Just wanted to highlight that the in-built functions for calculation of the min-max are rather primitive, and may not always lead to good results.

I am not so sure about the ability to restrict the time periods for which the mix-max is generated. I think you would end up with far too few data points to make a meaningful standard deviation. You are really looking for the extremes with the min-max, and restricting to such a a narrow time interval would probably not give you these extremes.

Regards,

Jason

<skewness.png>_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Sep 19, 2013 at 11:47 PM, Dapo Adejumo dapo_adejumo@yahoo.com wrote:

Hi,

It will be great if the generation of min-max values can be automated based on a global setting ( ability to select last 3,6 or 12 months data) . I am curious what happens to these “hard coded” min/max values when the database is set to generate min/max values from the frontend – I will assume they are discarded since they reside in the same table? Or which takes precedence ?

Thanks.

Regards,

Dapo Adejumo

+234803363677

skype : dapojorge

From: Dhis2-devs [mailto:dhis2-devs-bounces+dapo_adejumo=yahoo.com@lists.launchpad.net] On Behalf Of Jason Pickering

Sent: Thursday, September 19, 2013 6:28 PM
To: Juma Lungo
Cc: dhis2-devs

Subject: Re: [Dhis2-devs] Setting MinMax Values

I think I agree with Lars as well. Our use case was we wanted to restrict values between 0 and 31, and not do it with a validation rule, to provide visual clues during data entry.

As an aside, one may also want to calculate these values and save them to the database. The current algorithm uses a standard deviation approach, which is problematic, in that in many cases the data which is typically observed does not follow a normal Gaussian distribution. Other statistical models often provide better estimates of the mix/max, but currently, cannot be calculated automatically by the system. If these values are calculated externally in some statistical package, they can be inserted back into the system (or even automated).

Regards,

Jason

On Thu, Sep 19, 2013 at 6:25 PM, Juma Lungo jlungo@yahoo.com wrote:

I agree with Lars

Lungo


From: Lars Helge Øverland larshelge@gmail.com

To: Jason Pickering jason.p.pickering@gmail.com

Cc: dhis2-devs dhis2-devs@lists.launchpad.net

Sent: Thursday, September 19, 2013 2:43 PM
Subject: Re: [Dhis2-devs] Setting MinMax Values

Hi Conrad,

the min-max data element is per org unit and data element like you say, so it applies to all current and future periods by definition. I don’t know your use-case… but in general one might want to think twice before setting the same max value for all facilities, since they usually have very different catchment population and hence natural variations in numbers.

Lars


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp