[Branch ~dhis2-devs-core/dhis2/trunk] Rev 4611: added proper validation for real/natural numbers (including support for e-notation)

Support - Assistance technique
1

revision-diff.txt (5.88 KB)

2

Hello Morten,

I agree with these validation on field but why don’t you apply the “Math methods” - which are in commons - into them. The “Math methods” are working on regex that I think it is very very robust in validating.

Btw, the name of method “positive/negative_natural_number” is quite sensitive to me.

···

On Mon, Sep 19, 2011 at 3:41 AM, noreply@launchpad.net wrote:

revno: 4611

committer: Morten Olav Hansen mortenoh@gmail.com

branch nick: dhis2

timestamp: Sun 2011-09-18 22:39:31 +0200

message:

added proper validation for real/natural numbers (including support for e-notation)

modified:

dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/i18n/messages.vm

dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/javascripts/jQuery/jquery.validate.ext.js

dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/macros.vm

dhis-2/dhis-web/dhis-web-commons/src/main/resources/i18n_global.properties

–

lp:dhis2

https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk

Your team DHIS 2 developers is subscribed to branch lp:dhis2.

To unsubscribe from this branch go to https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription

=== modified file ‘dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/i18n/messages.vm’

— dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/i18n/messages.vm 2011-09-12 06:19:59 +0000

+++ dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/i18n/messages.vm 2011-09-18 20:39:31 +0000

@@ -68,5 +68,8 @@

            ,greaterDate: '$encoder.jsEscape($i18n.getString('closedDate_should_be_greater_than_openDate' ) , "'")'

            ,unicodechars: '$encoder.jsEscape($i18n.getString('please_unicode_chars_only' ) , "'")'

            ,unrecognizedcoordinatestring: '$encoder.jsEscape($i18n.getString('unrecognized_coordinate_string' ) , "'")'

  •           ,real_number: '$encoder.jsEscape($i18n.getString('please_enter_valid_real_number' ) , "'")'

  •           ,natural_number: '$encoder.jsEscape($i18n.getString('please_enter_valid_natural_number' ) , "'")'

  •           ,positive_natural_number: '$encoder.jsEscape($i18n.getString('please_enter_valid_positive_natural_number' ) , "'")'

  •           ,negative_natural_number: '$encoder.jsEscape($i18n.getString('please_enter_valid_negative_natural_number' ) , "'")'

};

=== modified file ‘dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/javascripts/jQuery/jquery.validate.ext.js’

— dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/javascripts/jQuery/jquery.validate.ext.js 2011-08-15 14:42:49 +0000

+++ dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/javascripts/jQuery/jquery.validate.ext.js 2011-09-18 20:39:31 +0000

@@ -336,9 +336,52 @@

 params[0] = (params[0] == '') ? new RegExp(params[0]) : params[0];



 return this.optional(element) || params[0].test(value);

});

+jQuery.validator.addMethod(“real_number”, function(value, element, param) {

  • value = +value;

  • if( isNaN( value ) )

  • {

  •    return false;

  • }

  • return true;

+}, “Please enter a valid real number.”);

+jQuery.validator.addMethod(“natural_number”, function(value, element, param) {

  • value = +value;

  • if( isNaN( value ) || (“”+value).indexOf(‘.’) != -1 )

  • {

  •    return false;

  • }

  • return true;

+}, “Please enter a valid natural number.”);

+jQuery.validator.addMethod(“positive_natural_number”, function(value, element, param) {

  • value = +value;

  • if( isNaN( value ) || (“”+value).indexOf(‘.’) != -1 )

  • {

  •    return false;

  • }

  • return value >= 0;

+}, “Please enter a valid positive natural number.”);

+jQuery.validator.addMethod(“negative_natural_number”, function(value, element, param) {

  • value = +value;

  • if( isNaN( value ) || (“”+value).indexOf(‘.’) != -1 )

  • {

  •    return false;

  • }

  • return value <= 0;

+}, “Please enter a valid negative natural number.”);

// Support method for date

//Parse a string and convert it to a Date object.

//If string cannot be parsed, return null.

=== modified file ‘dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/macros.vm’

— dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/macros.vm 2011-09-18 19:46:53 +0000

+++ dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/macros.vm 2011-09-18 20:39:31 +0000

@@ -240,7 +240,7 @@

    <td style="width: 200px;">$!args.text</td>

    <td>

  •           <input type="text" id="$![args.id](http://args.id)" name="$!name" value="$!args.value" class="{validate:{required:$args.mandatory,rangelength:[2,230],number:true}}" style="width: 240px;" />

  •           <input type="text" id="$![args.id](http://args.id)" name="$!name" value="$!args.value" class="{validate:{required:$args.mandatory,real_number:true}}" style="width: 240px;" />




  </td>

#end

@@ -264,7 +264,7 @@

    <td style="width: 200px;">$!args.text</td>

    <td>

  •           <input type="text" id="$![args.id](http://args.id)" name="$!name" value="$!args.value" class="{validate:{required:$args.mandatory,rangelength:[2,230],digits:true}}" style="width: 240px;" />

  •           <input type="text" id="$![args.id](http://args.id)" name="$!name" value="$!args.value" class="{validate:{required:$args.mandatory,natural_number:true}}" style="width: 240px;" />




  </td>

#end

@@ -288,7 +288,7 @@

    <td style="width: 200px;">$!args.text</td>

    <td>

  •           <input type="text" id="$![args.id](http://args.id)" name="$!name" value="$!args.value" class="{validate:{required:$args.mandatory,rangelength:[2,230],digits:true}}" style="width: 240px;" />

  •           <input type="text" id="$![args.id](http://args.id)" name="$!name" value="$!args.value" class="{validate:{required:$args.mandatory,positive_natural_number:true}}" style="width: 240px;" />




  </td>

#end

@@ -312,7 +312,7 @@

    <td style="width: 200px;">$!args.text</td>

    <td>

  •           <input type="text" id="$![args.id](http://args.id)" name="$!name" value="$!args.value" class="{validate:{required:$args.mandatory,rangelength:[2,230],digits:true}}" style="width: 240px;" />

  •           <input type="text" id="$![args.id](http://args.id)" name="$!name" value="$!args.value" class="{validate:{required:$args.mandatory,negative_natural_number:true}}" style="width: 240px;" />




  </td>

#end

=== modified file ‘dhis-2/dhis-web/dhis-web-commons/src/main/resources/i18n_global.properties’

— dhis-2/dhis-web/dhis-web-commons/src/main/resources/i18n_global.properties 2011-09-16 07:53:59 +0000

+++ dhis-2/dhis-web/dhis-web-commons/src/main/resources/i18n_global.properties 2011-09-18 20:39:31 +0000

@@ -376,6 +376,11 @@

unrecognized_coordinate_string = Unrecognized coordinate string

please_enter_a_letters_or_digits = Please enter letters or digits

+please_enter_valid_real_number = Please enter a valid real number.

+please_enter_valid_natural_number = Please enter a valid natural number.

+please_enter_valid_positive_natural_number = Please enter a valid positive natural number.

+please_enter_valid_negative_natural_number = Please enter a valid negative natural number.

please_enter_name = Please enter name!

please_select_period = Please select period!

please_select_dataset = Please select dataset!

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–
“Expert By Chance”

3

Hi

I agree with these validation on field but why don't you apply the "Math
methods" - which are in commons - into them. The "Math methods" are working
on regex that I think it is very very robust in validating.

Ok, sounds good. What are they named? I never found any validations
for positive/negative numbers..

Btw, the name of method "positive/negative_natural_number" is quite
sensitive to me.

The naming here is a bit off. I need suggestions here, even whole
numbers is not defined always mean the set of all integers.

···

--
Morten

On Mon, Sep 19, 2011 at 3:41 AM, <noreply@launchpad.net> wrote:

------------------------------------------------------------
revno: 4611
committer: Morten Olav Hansen <mortenoh@gmail.com>
branch nick: dhis2
timestamp: Sun 2011-09-18 22:39:31 +0200
message:
added proper validation for real/natural numbers (including support for
e-notation)
modified:

dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/i18n/messages.vm

dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/javascripts/jQuery/jquery.validate.ext.js
dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/macros.vm

dhis-2/dhis-web/dhis-web-commons/src/main/resources/i18n_global.properties

--
lp:dhis2
trunk : Code : DHIS

Your team DHIS 2 developers is subscribed to branch lp:dhis2.
To unsubscribe from this branch go to
OpenID transaction in progress

=== modified file
'dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/i18n/messages.vm'
---
dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/i18n/messages.vm
2011-09-12 06:19:59 +0000
+++
dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/i18n/messages.vm
2011-09-18 20:39:31 +0000
@@ -68,5 +68,8 @@
,greaterDate:
'$encoder.jsEscape($i18n.getString('closedDate_should_be_greater_than_openDate'
) , "'")'
,unicodechars:
'$encoder.jsEscape($i18n.getString('please_unicode_chars_only' ) , "'")'
,unrecognizedcoordinatestring:
'$encoder.jsEscape($i18n.getString('unrecognized_coordinate_string' ) ,
"'")'
-
+ ,real_number:
'$encoder.jsEscape($i18n.getString('please_enter_valid_real_number' ) ,
"'")'
+ ,natural_number:
'$encoder.jsEscape($i18n.getString('please_enter_valid_natural_number' ) ,
"'")'
+ ,positive_natural_number:
'$encoder.jsEscape($i18n.getString('please_enter_valid_positive_natural_number'
) , "'")'
+ ,negative_natural_number:
'$encoder.jsEscape($i18n.getString('please_enter_valid_negative_natural_number'
) , "'")'
};

=== modified file
'dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/javascripts/jQuery/jquery.validate.ext.js'
---
dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/javascripts/jQuery/jquery.validate.ext.js
2011-08-15 14:42:49 +0000
+++
dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/javascripts/jQuery/jquery.validate.ext.js
2011-09-18 20:39:31 +0000
@@ -336,9 +336,52 @@
params[0] = (params[0] == '') ? new RegExp(params[0]) : params[0];

return this\.optional\(element\) || params\[0\]\.test\(value\);

-
});

+jQuery.validator.addMethod("real_number", function(value, element, param)
{
+ value = +value;
+
+ if( isNaN( value ) )
+ {
+ return false;
+ }
+
+ return true;
+}, "Please enter a valid real number.");
+
+jQuery.validator.addMethod("natural_number", function(value, element,
param) {
+ value = +value;
+
+ if( isNaN( value ) || (""+value).indexOf('.') != -1 )
+ {
+ return false;
+ }
+
+ return true;
+}, "Please enter a valid natural number.");
+
+jQuery.validator.addMethod("positive_natural_number", function(value,
element, param) {
+ value = +value;
+
+ if( isNaN( value ) || (""+value).indexOf('.') != -1 )
+ {
+ return false;
+ }
+
+ return value >= 0;
+}, "Please enter a valid positive natural number.");
+
+jQuery.validator.addMethod("negative_natural_number", function(value,
element, param) {
+ value = +value;
+
+ if( isNaN( value ) || (""+value).indexOf('.') != -1 )
+ {
+ return false;
+ }
+
+ return value <= 0;
+}, "Please enter a valid negative natural number.");
+
// Support method for date
//Parse a string and convert it to a Date object.
//If string cannot be parsed, return null.

=== modified file
'dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/macros.vm'
--- dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/macros.vm
2011-09-18 19:46:53 +0000
+++ dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/macros.vm
2011-09-18 20:39:31 +0000
@@ -240,7 +240,7 @@
<tr>
<td style="width: 200px;">$!args.text</td>
<td>
- <input type="text" id="$!args.id" name="$!name"
value="$!args.value"
class="{validate:{required:$args.mandatory,rangelength:[2,230],number:true}}"
style="width: 240px;" />
+ <input type="text" id="$!args.id" name="$!name"
value="$!args.value"
class="{validate:{required:$args.mandatory,real_number:true}}" style="width:
240px;" />
</td>
</tr>
#end
@@ -264,7 +264,7 @@
<tr>
<td style="width: 200px;">$!args.text</td>
<td>
- <input type="text" id="$!args.id" name="$!name"
value="$!args.value"
class="{validate:{required:$args.mandatory,rangelength:[2,230],digits:true}}"
style="width: 240px;" />
+ <input type="text" id="$!args.id" name="$!name"
value="$!args.value"
class="{validate:{required:$args.mandatory,natural_number:true}}"
style="width: 240px;" />
</td>
</tr>
#end
@@ -288,7 +288,7 @@
<tr>
<td style="width: 200px;">$!args.text</td>
<td>
- <input type="text" id="$!args.id" name="$!name"
value="$!args.value"
class="{validate:{required:$args.mandatory,rangelength:[2,230],digits:true}}"
style="width: 240px;" />
+ <input type="text" id="$!args.id" name="$!name"
value="$!args.value"
class="{validate:{required:$args.mandatory,positive_natural_number:true}}"
style="width: 240px;" />
</td>
</tr>
#end
@@ -312,7 +312,7 @@
<tr>
<td style="width: 200px;">$!args.text</td>
<td>
- <input type="text" id="$!args.id" name="$!name"
value="$!args.value"
class="{validate:{required:$args.mandatory,rangelength:[2,230],digits:true}}"
style="width: 240px;" />
+ <input type="text" id="$!args.id" name="$!name"
value="$!args.value"
class="{validate:{required:$args.mandatory,negative_natural_number:true}}"
style="width: 240px;" />
</td>
</tr>
#end

=== modified file
'dhis-2/dhis-web/dhis-web-commons/src/main/resources/i18n_global.properties'
---
dhis-2/dhis-web/dhis-web-commons/src/main/resources/i18n_global.properties
2011-09-16 07:53:59 +0000
+++
dhis-2/dhis-web/dhis-web-commons/src/main/resources/i18n_global.properties
2011-09-18 20:39:31 +0000
@@ -376,6 +376,11 @@
unrecognized_coordinate_string = Unrecognized coordinate string
please_enter_a_letters_or_digits = Please enter letters or digits

+please_enter_valid_real_number = Please enter a valid real number.
+please_enter_valid_natural_number = Please enter a valid natural number.
+please_enter_valid_positive_natural_number = Please enter a valid
positive natural number.
+please_enter_valid_negative_natural_number = Please enter a valid
negative natural number.
+
please_enter_name = Please enter name!
please_select_period = Please select period!
please_select_dataset = Please select dataset!

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

--
"Expert By Chance"

4

Hi

I agree with these validation on field but why don’t you apply the "Math

methods" - which are in commons - into them. The “Math methods” are working

on regex that I think it is very very robust in validating.

Ok, sounds good. What are they named? I never found any validations

for positive/negative numbers…

Yes, they are isRealNumber() which is including in positive/negative numbers, isNegativeInt(), and isPositiveInt().

Btw, the name of method “positive/negative_natural_number” is quite

sensitive to me.

The naming here is a bit off. I need suggestions here, even whole

numbers is not defined always mean the set of all integers.

–

Morten

About this you could also ask Jason for more suggestion.

···

On Mon, Sep 19, 2011 at 1:42 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

–

“Expert By Chance”

5

Hi Morten,

We currently have four number types.

Number. This is any real number, with an option single sign (-), a
single decimal point and no thousand separators. Scientific notation
is not allowed (but could be if there is a real need for it).
Integer: Any whole number, including zero.
Positive integers: Any whole number greater than zero.
Negative integers: Any whole number less than zero.

We had problems with some of the Javascript validations, so had to
implement more robust ones (see rev 3921 for details). Things like
"00" were valid, but should not be allowed.

As to whether users know what integers are, I am not sure. The current
syntax we have is a bit technical, but adding more number types is
likely to confuse things further It is detailed in the user manual,
Data element chapter. I am not sure they necessarily know what a
natural number is either but I am not opposed to the change of name if
it is more clear.. Regardless, there are no negative natural numbers.
:slight_smile:

As for the support for scientific notation, I am not necessarily
opposed to it, but think we need to stick to the more strict regex
validation.

Something like

^-?(0|[1-9]\d*)(\.\d+)?(?:[eE][-+]?[0-9]+)?$

would allow scientifically noted numbers, with an optional negative
sign, optional decimal, and optional trailing scientific notation. No
leading or trailing spaces allowed.

Have a look in
dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/javascripts/commons.js
for details of the different regex checks we developed to support the
current four number types.

Hope this helps.
Jason

···

On Mon, Sep 19, 2011 at 8:48 AM, Hieu Dang Duy <hieu.hispvietnam@gmail.com> wrote:

On Mon, Sep 19, 2011 at 1:42 PM, Morten Olav Hansen <mortenoh@gmail.com> > wrote:

Hi

> I agree with these validation on field but why don't you apply the "Math
> methods" - which are in commons - into them. The "Math methods" are
> working
> on regex that I think it is very very robust in validating.

Ok, sounds good. What are they named? I never found any validations
for positive/negative numbers..

Yes, they are isRealNumber() which is including in positive/negative
numbers, isNegativeInt(), and isPositiveInt().

> Btw, the name of method "positive/negative_natural_number" is quite
> sensitive to me.

The naming here is a bit off. I need suggestions here, even whole
numbers is not defined always mean the set of all integers.

--
Morten

About this you could also ask Jason for more suggestion.

--
"Expert By Chance"

6

Number. This is any real number, with an option single sign (-), a
single decimal point and no thousand separators. Scientific notation
is not allowed (but could be if there is a real need for it).

We don't really support real numbers.. we support integers, and
decimal approximations of rational numbers.. I think the name decimal
would be better here.

Integer: Any whole number, including zero.
Positive integers: Any whole number greater than zero.
Negative integers: Any whole number less than zero.

These are good.

As to whether users know what integers are, I am not sure. The current
syntax we have is a bit technical, but adding more number types is
likely to confuse things further It is detailed in the user manual,
Data element chapter. I am not sure they necessarily know what a
natural number is either but I am not opposed to the change of name if
it is more clear.. Regardless, there are no negative natural numbers.
:slight_smile:

Yes, I think maybe integer is the way to go, since whole numbers is
not clearly defined. But I'm not a native speaker, so I'm not really
sure whats best.

As for the support for scientific notation, I am not necessarily
opposed to it, but think we need to stick to the more strict regex
validation.

I think maybe supporting e-notation but restricting it might be more
confusing.. ? not sure here... maybe just plain decimals is OK.

Have a look in
dhis-2/dhis-web/dhis-web-commons-resources/src/main/webapp/dhis-web-commons/javascripts/commons.js

Yes, looking there now.

···

--
Morten

for details of the different regex checks we developed to support the
current four number types.

Hope this helps.
Jason

On Mon, Sep 19, 2011 at 8:48 AM, Hieu Dang Duy > <hieu.hispvietnam@gmail.com> wrote:

On Mon, Sep 19, 2011 at 1:42 PM, Morten Olav Hansen <mortenoh@gmail.com> >> wrote:

Hi

> I agree with these validation on field but why don't you apply the "Math
> methods" - which are in commons - into them. The "Math methods" are
> working
> on regex that I think it is very very robust in validating.

Ok, sounds good. What are they named? I never found any validations
for positive/negative numbers..

Yes, they are isRealNumber() which is including in positive/negative
numbers, isNegativeInt(), and isPositiveInt().

> Btw, the name of method "positive/negative_natural_number" is quite
> sensitive to me.

The naming here is a bit off. I need suggestions here, even whole
numbers is not defined always mean the set of all integers.

--
Morten

About this you could also ask Jason for more suggestion.

--
"Expert By Chance"

7

We don't really support real numbers.. we support integers, and
decimal approximations of rational numbers.. I think the name decimal
would be better here.

Good point and I support the change in naming. We should have seen
this earlier. However, we do also support decimal representation of
irrational/transcendental numbers. I think you mean "finite decimal
approximations of real numbers". We do not support complex numbers
(and I do not see a real use case for them!).

Not knowing the details of how Java handles floating point numbers,
but I also assume that this is a 32 bit decimal representation, which
further restricts the length of the number which the regex does not
really account for.

Yes, I think maybe integer is the way to go, since whole numbers is
not clearly defined. But I'm not a native speaker, so I'm not really
sure whats best.

Natural numbers are simply positive integers but whether they include
zero is somewhat of a mathematical fine point. Not being an
authoritative source on this, I think we should stick with positive
and negative integers and be sure they are explained well in the help.
I think if we need a set of numbers, which include 0 and positive
integers, we should use "Natural numbers", or allow zero to be
included in "Positive and negative integers", but be sure it is well
explained in the manual.

As for the support for scientific notation, I am not necessarily
opposed to it, but think we need to stick to the more strict regex
validation.

I think maybe supporting e-notation but restricting it might be more
confusing.. ? not sure here... maybe just plain decimals is OK.

Well, if we support it, we need to be sure it is valid and I
personally do not trust !NaN. I see no real need for it and unless
people are asking for it, I suggest we simply not allow it.

Best regards,
Jason

8

Good point and I support the change in naming. We should have seen
this earlier. However, we do also support decimal representation of
irrational/transcendental numbers. I think you mean "finite decimal
approximations of real numbers". We do not support complex numbers
(and I do not see a real use case for them!).

Yes, the point was that we don't support real numbers.

Not knowing the details of how Java handles floating point numbers,
but I also assume that this is a 32 bit decimal representation, which
further restricts the length of the number which the regex does not
really account for.

This is not a problem, we store them as strings. There are 64 bit
value types that can be used, and there are also libraries that would
support any number of bits.

Natural numbers are simply positive integers but whether they include
zero is somewhat of a mathematical fine point. Not being an
authoritative source on this, I think we should stick with positive
and negative integers and be sure they are explained well in the help.
I think if we need a set of numbers, which include 0 and positive
integers, we should use "Natural numbers", or allow zero to be
included in "Positive and negative integers", but be sure it is well
explained in the manual.

I opted for simplicity, and simply used what we have.

Numbers (uses isRealnumber())
Integer (uses isInt())
Positive Integer (uses isPositiveInt())
Negative Integer (uses isNegativeInt())

At some point I think number should be renamed, but this is not the
time for that. :slight_smile:

···

--
Morten

Best regards,
Jason

9

This is not a problem, we store them as strings. There are 64 bit
value types that can be used, and there are also libraries that would
support any number of bits.

Yes, this is my point. I am sure (without knowing the details) that
there are restrictions on what would be a valid exponent and fraction
for a decimal representation of a real number. If a number with 255
digits is stored as text, and whether the values are handled as a
double (I think that all values are treated as doubles regardless of
whether they are integers or not), this places different restrictions
on the number length which we should allow. So, if someone types in an
exponent with 200 numbers and 55 decimal points (which we could store
as text), would be be a valid double value?

10

Yes, this is my point. I am sure (without knowing the details) that
there are restrictions on what would be a valid exponent and fraction
for a decimal representation of a real number. If a number with 255
digits is stored as text, and whether the values are handled as a
double (I think that all values are treated as doubles regardless of
whether they are integers or not), this places different restrictions
on the number length which we should allow. So, if someone types in an
exponent with 200 numbers and 55 decimal points (which we could store
as text), would be be a valid double value?

The range of double should be -1.79769313486231570E+308 to
1.79769313486231570E+308 (if using 64 bit java I assume..).

There is also BigInteger / BigDecial that could be used, that supports
even bigger numbers.

That said, this is just what Java has to offer, what DHIS2 supports I
do not know.

···

--
Morten

11

Perhaps what is more important than being able to support a count of
the number of stars in the galaxy, is how we treat these numbers once
we have them captured at whatever that precision this might be.

Currently I think our notion of precision is a bit weak. In the
datamart service for indicators, for example, we seem to have a fixed
notion of precision which is based on decimal places - from my reading
of the code, it seems we store accurate to 1 decimal place.

What we probably should be doing is maintaining some confidence level
of significant figures. This becomes quite obvious if we start
inputting and storing values in scientific notation - those of us old
enough to have used a slide rule will be familiar with this :-).

So if I have a numerator (eg malaria cases) of 5436 and a denominator
(eg population) of 155000, then what can I say about the indicator
value? Well if I calculate on my calculator I get:
0.035070968

but obviously I am not confident in all those digits. But if my
numerator is accurate to 4 significant figures and my denominator is
accurate to 3, then I can be be confident to 2 significant figures in
my result; ie I can report the value as:
0.035

I am not sure what the best strategy of managing precision in dhis
should be, but it does strike me, for a system concerned with
aggregation, we should attempt to attack it a bit more rigorously than
we do. What this probably requires, at the point of capture, is to
capture the precision of the number, particularly where we know we are
capturing an estimate eg. as a result of rounding. This is done
implicitly when using scientific notation. The problem is more
visible when we capture a string like "155000". How precise is that?
Well we don't actually know. Intuitively we suspect its not accurate
to 6 significant figure, and that its accurate to at least 3. But it
could be 4 (eg. 1.550E5)..

Maybe its just me that worries a bit about these things. Does anyone
else have a sense that it is important to be able to indicate the
precision of calculated indicator values?

Bob

PS. Storing natural number 'counts' as a floating point number
introduces some untidyness here, but one that can be dealt with as we
"know" the numberType of the datalement value.

PPS. this is is a very similar issue with an earlier discussion re
rounding of coordinates during GML import. The number of decimal
places should always be an outcome rather a target of specifying
precision.

···

On 19 September 2011 09:16, Morten Olav Hansen <mortenoh@gmail.com> wrote:

Yes, this is my point. I am sure (without knowing the details) that
there are restrictions on what would be a valid exponent and fraction
for a decimal representation of a real number. If a number with 255
digits is stored as text, and whether the values are handled as a
double (I think that all values are treated as doubles regardless of
whether they are integers or not), this places different restrictions
on the number length which we should allow. So, if someone types in an
exponent with 200 numbers and 55 decimal points (which we could store
as text), would be be a valid double value?

The range of double should be -1.79769313486231570E+308 to
1.79769313486231570E+308 (if using 64 bit java I assume..).

There is also BigInteger / BigDecial that could be used, that supports
even bigger numbers.

That said, this is just what Java has to offer, what DHIS2 supports I
do not know.

--
Morten

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

12

We have sort of wondered away from the original discussion, but of
course, you raise a good point Bob. We sometimes get somewhat
unexpected results, with particularly low values..i.e. 0.04 which in
DHIS is the same as 0.0. So, we maybe can use a "factor" to present
the number as 0.04*1000 but it seems the epi folks have particular
ways of representing certain indicators, i.e. they should be a
straight rate, or per 100,000 and the problem is, we do not always
know, prima facie, so that we can set the indicator to have the
correct "factor". So, we have to double check what the data mart gives
us, to be sure that the number really is a zero. The difference
between 0.0 and 0.04 seems to be significant to folks who are looking
closely.

Anyway, my original point here was that I think having numbers stored
as text is fine, just as long as we are very robust with how we store
them. Not that I think anyone is going to input a number greater than
1e308 (we could not store it anyway since we only allow 255
characters) but it is better to check and be sure that it does not
happen.

Regards,
Jason

···

On Mon, Sep 19, 2011 at 11:31 AM, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

Perhaps what is more important than being able to support a count of
the number of stars in the galaxy, is how we treat these numbers once
we have them captured at whatever that precision this might be.

Currently I think our notion of precision is a bit weak. In the
datamart service for indicators, for example, we seem to have a fixed
notion of precision which is based on decimal places - from my reading
of the code, it seems we store accurate to 1 decimal place.

What we probably should be doing is maintaining some confidence level
of significant figures. This becomes quite obvious if we start
inputting and storing values in scientific notation - those of us old
enough to have used a slide rule will be familiar with this :-).

So if I have a numerator (eg malaria cases) of 5436 and a denominator
(eg population) of 155000, then what can I say about the indicator
value? Well if I calculate on my calculator I get:
0.035070968

but obviously I am not confident in all those digits. But if my
numerator is accurate to 4 significant figures and my denominator is
accurate to 3, then I can be be confident to 2 significant figures in
my result; ie I can report the value as:
0.035

I am not sure what the best strategy of managing precision in dhis
should be, but it does strike me, for a system concerned with
aggregation, we should attempt to attack it a bit more rigorously than
we do. What this probably requires, at the point of capture, is to
capture the precision of the number, particularly where we know we are
capturing an estimate eg. as a result of rounding. This is done
implicitly when using scientific notation. The problem is more
visible when we capture a string like "155000". How precise is that?
Well we don't actually know. Intuitively we suspect its not accurate
to 6 significant figure, and that its accurate to at least 3. But it
could be 4 (eg. 1.550E5)..

Maybe its just me that worries a bit about these things. Does anyone
else have a sense that it is important to be able to indicate the
precision of calculated indicator values?

Bob

PS. Storing natural number 'counts' as a floating point number
introduces some untidyness here, but one that can be dealt with as we
"know" the numberType of the datalement value.

PPS. this is is a very similar issue with an earlier discussion re
rounding of coordinates during GML import. The number of decimal
places should always be an outcome rather a target of specifying
precision.

On 19 September 2011 09:16, Morten Olav Hansen <mortenoh@gmail.com> wrote:

Yes, this is my point. I am sure (without knowing the details) that
there are restrictions on what would be a valid exponent and fraction
for a decimal representation of a real number. If a number with 255
digits is stored as text, and whether the values are handled as a
double (I think that all values are treated as doubles regardless of
whether they are integers or not), this places different restrictions
on the number length which we should allow. So, if someone types in an
exponent with 200 numbers and 55 decimal points (which we could store
as text), would be be a valid double value?

The range of double should be -1.79769313486231570E+308 to
1.79769313486231570E+308 (if using 64 bit java I assume..).

There is also BigInteger / BigDecial that could be used, that supports
even bigger numbers.

That said, this is just what Java has to offer, what DHIS2 supports I
do not know.

--
Morten

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help