Many program rules not firing in 2.26

Hi Devs,

I’m finding that a number of program rules aren’t firing in 2.26 – this is happening with both Event Capture and Tracker Capture.

This includes the sample ‘Hide Program Stage’ rule which has been added to the WHO RMNCH Tracker for the 2.26 release. To reproduce in Play demo:

  1.   add a new registration to the WHO RMNCH Tracker with date 2017-01-01
    
  2.   complete the initial visit, and click ‘cancel’ to prevent creation of a second antenatal care visit
    
  3.   click ‘+’ to create a ‘Care at birth’ event on 2017-02-01
    
  4.   for WHOMCH pregnancy outcome, select ‘Stillbirth’, and complete the event.
    

The rule which should hide the postnatal program stage has not fired. In the browser console, the following two errors are recorded are:

· Expression #{currentProgranancyOutcome} != ‘Live birth’ contains variable currentProgranancyOutcome - but this variable is not defined.

· * Expression with id rule:xOm49QX4Nsc could not be run. Original condition was: #{currentProgranancyOutcome} != ‘Live birth’ - Evaluation ended up as:#{currentProgranancyOutcome} != ‘Live birth’ - error message:SyntaxError: Invalid or unexpected token*

The important error seems to be the second one, as in many cases this is appearing in isolation (without the first one), and is still breaking program rules. For example, I’ve loaded the Cameroon EquityTool into the Play Demo instance, and any attempt to enter new Event Capture events for this program keeps generating similar error messages in the browser console, eg:

· * Expression with id rule:gXwzJFfRn5m could not be run. Original condition was: #{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - Evaluation ended up as:#{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - error message:SyntaxError: Invalid or unexpected token*

These program rules have been extensively tested on earlier releases, and worked well – where users used to be prevented from entering invalid surveys, they can now save invalid surveys.

This is very serious, as it means that in 2.26, users can now bypass many program rules, and save incomplete or incorrect data. But I’m also aware that there have recently been build issues with 2.25 and 2.26 releases – before I log a bug on Jira, could this be related to those build issues?

Regards, Sam.

Hey Sam!
Thanks for reporting.

I tested WHO RMNCH and 'Hide program stage' on Play, and found that it the problem is in fact what is indicated by the console message in point 4 below. The source field #{currentProganancyOutcome} did not exist. Seems someone has renamed it to
#{Current Progranancy Outcome} - see below to the left:

Such a name change should ideally clean up the expressions using the source field, or at least flag the ones that has become illegal as a result of the name change. At the moment however you would have to manually update the expressions after a name change. Updated the expression now on Play, and the expression runs fine:
Expression with id rule:xOm49QX4Nsc was successfully run. Original condition was: #{current Progranancy Outcome} != 'Live birth' - Evaluation ended up as:'' != 'Live birth' - Result of evaluation was:true

In general your rules hiding program stages should work fine, let me know if you find any other problems here.

Looking into the second part of the mail regarding Event Capture - this is a different problem: There is a dash in the source field name. Registered an improvement in Jira based on this: https://jira.dhis2.org/browse/DHIS2-1449 - but avoiding dashes is a possible workaround.

Best regards,
Markus

···

19. apr. 2017 kl. 12.43 skrev Sam Johnson <samuel.johnson@qebo.co.uk>:

Hi Devs,

I’m finding that a number of program rules aren’t firing in 2.26 – this is happening with both Event Capture and Tracker Capture.

This includes the sample ‘Hide Program Stage’ rule which has been added to the WHO RMNCH Tracker for the 2.26 release. To reproduce in Play demo:
1. add a new registration to the WHO RMNCH Tracker with date 2017-01-01
2. complete the initial visit, and click ‘cancel’ to prevent creation of a second antenatal care visit
3. click ‘+’ to create a ‘Care at birth’ event on 2017-02-01
4. for WHOMCH pregnancy outcome, select ‘Stillbirth’, and complete the event.
The rule which should hide the postnatal program stage has not fired. In the browser console, the following two errors are recorded are:
· Expression #{currentProgranancyOutcome} != 'Live birth' contains variable currentProgranancyOutcome - but this variable is not defined.
· Expression with id rule:xOm49QX4Nsc could not be run. Original condition was: #{currentProgranancyOutcome} != 'Live birth' - Evaluation ended up as:#{currentProgranancyOutcome} != 'Live birth' - error message:SyntaxError: Invalid or unexpected token

The important error seems to be the second one, as in many cases this is appearing in isolation (without the first one), and is still breaking program rules. For example, I’ve loaded the Cameroon EquityTool into the Play Demo instance, and any attempt to enter new Event Capture events for this program keeps generating similar error messages in the browser console, eg:
· Expression with id rule:gXwzJFfRn5m could not be run. Original condition was: #{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - Evaluation ended up as:#{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - error message:SyntaxError: Invalid or unexpected token
These program rules have been extensively tested on earlier releases, and worked well – where users used to be prevented from entering invalid surveys, they can now save invalid surveys.

This is very serious, as it means that in 2.26, users can now bypass many program rules, and save incomplete or incorrect data. But I’m also aware that there have recently been build issues with 2.25 and 2.26 releases – before I log a bug on Jira, could this be related to those build issues?

Regards, Sam.

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net <mailto:dhis2-devs@lists.launchpad.net>
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Hi Markus (and other devs),

Many thanks for diagnosing this so quickly. You’re spot-on, it’s the dash in the source name that’s causing the rules to fail – if I remove dashes from all source names, the rules fire without problems. In fact, this seems wider than just dashes – program rules that were failing in another instance seem to now work if I similarly remove brackets and spaces from source names.

Would it be at all possible to get this treated as a bug and fixed in 2.26, rather than treated as an improvement for a future version of DHIS2? These rules were all working perfectly when we did intensive testing on our programs in 2.25 (eg they work properly in 2.25 da6a2df / 2017-02-02 06:43), so this limitation seems to have been recently introduced (very likely by accident, since other element names allow dashes), and it’s breaking configurations that have worked well up until now.

The reason we’re so keen on this is that Metrics for Management has published XMLs for 29 EquityTool programs, and these have now been downloaded and installed in a wide range of DHIS2 instances. It will be a huge job to not only update the source names in all of these 29 programs and re-publish them, but also contact all existing users and ask them to reinstall the corrected programs. If there’s any chance we could get the program rules in 2.26 working as they did before, this would save us an enormous amount of work…

Many thanks for any help you might be able to give us with this!

Regards, Sam.

···

From: Markus Bekken markus@dhis2.org

Date: Wednesday, 19 April 2017 at 14:55

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: DHIS2 Developers dhis2-devs@lists.launchpad.net, Kenzo Fry kenzo.fry.consultant@gmail.com

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Hey Sam!

Thanks for reporting.

I tested WHO RMNCH and ‘Hide program stage’ on Play, and found that it the problem is in fact what is indicated by the console message in point 4 below. The source field #{currentProganancyOutcome} did not exist. Seems someone has renamed it to

#{Current Progranancy Outcome} - see below to the left:

Such a name change should ideally clean up the expressions using the source field, or at least flag the ones that has become illegal as a result of the name change. At the moment however you would have to manually update the expressions after a name change. Updated the expression now on Play, and the expression runs fine:

Expression with id rule:xOm49QX4Nsc was successfully run. Original condition was: #{current Progranancy Outcome} != ‘Live birth’ - Evaluation ended up as:’’ != ‘Live birth’ - Result of evaluation was:true

In general your rules hiding program stages should work fine, let me know if you find any other problems here.

Looking into the second part of the mail regarding Event Capture - this is a different problem: There is a dash in the source field name. Registered an improvement in Jira based on this: https://jira.dhis2.org/browse/DHIS2-1449 - but avoiding dashes is a possible workaround.

Best regards,

Markus

  1. apr. 2017 kl. 12.43 skrev Sam Johnson samuel.johnson@qebo.co.uk:

Hi Devs,

I’m finding that a number of program rules aren’t firing in 2.26 – this is happening with both Event Capture and Tracker Capture.

This includes the sample ‘Hide Program Stage’ rule which has been added to the WHO RMNCH Tracker for the 2.26 release. To reproduce in Play demo:

  1.    add a new registration to the WHO RMNCH Tracker with date 2017-01-01
    
  1.    complete the initial visit, and click ‘cancel’ to prevent creation of a second antenatal care visit
    
  1.    click ‘+’ to create a ‘Care at birth’ event on 2017-02-01
    
  1.    for WHOMCH pregnancy outcome, select ‘Stillbirth’, and complete the event.
    

The rule which should hide the postnatal program stage has not fired. In the browser console, the following two errors are recorded are:

· * Expression #{currentProgranancyOutcome} != ‘Live birth’ contains variable currentProgranancyOutcome - but this variable is not defined.*

· * Expression with id rule:xOm49QX4Nsc could not be run. Original condition was: #{currentProgranancyOutcome} != ‘Live birth’ - Evaluation ended up as:#{currentProgranancyOutcome} != ‘Live birth’ - error message:SyntaxError: Invalid or unexpected token*

The important error seems to be the second one, as in many cases this is appearing in isolation (without the first one), and is still breaking program rules. For example, I’ve loaded the Cameroon EquityTool into the Play Demo instance, and any attempt to enter new Event Capture events for this program keeps generating similar error messages in the browser console, eg:

· * Expression with id rule:gXwzJFfRn5m could not be run. Original condition was: #{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - Evaluation ended up as:#{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - error message:SyntaxError: Invalid or unexpected token*

These program rules have been extensively tested on earlier releases, and worked well – where users used to be prevented from entering invalid surveys, they can now save invalid surveys.

This is very serious, as it means that in 2.26, users can now bypass many program rules, and save incomplete or incorrect data. But I’m also aware that there have recently been build issues with 2.25 and 2.26 releases – before I log a bug on Jira, could this be related to those build issues?

Regards, Sam.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Markus,

Just following up my question below – is there any chance we could get this treated as a bug, and fixed in 2.26? This issue has crippled 30-odd programs that Metrics for Management (M4M) has already published and deployed, but I’m guessing that the fix is probably just be a very simple string-handling change…

Many thanks for any help you might be able to give!

Cheers, Sam.

···

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Sam Johnson samuel.johnson@qebo.co.uk

Date: Friday, 21 April 2017 at 17:58

To: Markus Bekken markus@dhis2.org

Cc: Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett andrea@m4mgmt.org, DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing

Feedback

Hi Markus (and other devs),

Many thanks for diagnosing this so quickly. You’re spot-on, it’s the dash in the source name that’s causing the rules to fail – if I remove dashes from all source names, the rules fire without problems. In fact, this seems wider than just dashes – program rules that were failing in another instance seem to now work if I similarly remove brackets and spaces from source names.

Would it be at all possible to get this treated as a bug and fixed in 2.26, rather than treated as an improvement for a future version of DHIS2? These rules were all working perfectly when we did intensive testing on our programs in 2.25 (eg they work properly in 2.25 da6a2df / 2017-02-02 06:43), so this limitation seems to have been recently introduced (very likely by accident, since other element names allow dashes), and it’s breaking configurations that have worked well up until now.

The reason we’re so keen on this is that Metrics for Management has published XMLs for 29 EquityTool programs, and these have now been downloaded and installed in a wide range of DHIS2 instances. It will be a huge job to not only update the source names in all of these 29 programs and re-publish them, but also contact all existing users and ask them to reinstall the corrected programs. If there’s any chance we could get the program rules in 2.26 working as they did before, this would save us an enormous amount of work…

Many thanks for any help you might be able to give us with this!

Regards, Sam.

From: Markus Bekken markus@dhis2.org

Date: Wednesday, 19 April 2017 at 14:55

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: DHIS2 Developers dhis2-devs@lists.launchpad.net, Kenzo Fry kenzo.fry.consultant@gmail.com

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Hey Sam!

Thanks for reporting.

I tested WHO RMNCH and ‘Hide program stage’ on Play, and found that it the problem is in fact what is indicated by the console message in point 4 below. The source field #{currentProganancyOutcome} did not exist. Seems someone has renamed it to

#{Current Progranancy Outcome} - see below to the left:

Such a name change should ideally clean up the expressions using the source field, or at least flag the ones that has become illegal as a result of the name change. At the moment however you would have to manually update the expressions after a name change. Updated the expression now on Play, and the expression runs fine:

Expression with id rule:xOm49QX4Nsc was successfully run. Original condition was: #{current Progranancy Outcome} != ‘Live birth’ - Evaluation ended up as:’’ != ‘Live birth’ - Result of evaluation was:true

In general your rules hiding program stages should work fine, let me know if you find any other problems here.

Looking into the second part of the mail regarding Event Capture - this is a different problem: There is a dash in the source field name. Registered an improvement in Jira based on this: https://jira.dhis2.org/browse/DHIS2-1449 - but avoiding dashes is a possible workaround.

Best regards,

Markus

  1. apr. 2017 kl. 12.43 skrev Sam Johnson samuel.johnson@qebo.co.uk:

Hi Devs,

I’m finding that a number of program rules aren’t firing in 2.26 – this is happening with both Event Capture and Tracker Capture.

This includes the sample ‘Hide Program Stage’ rule which has been added to the WHO RMNCH Tracker for the 2.26 release. To reproduce in Play demo:

  1.    add a new registration to the WHO RMNCH Tracker with date 2017-01-01
    
  1.    complete the initial visit, and click ‘cancel’ to prevent creation of a second antenatal care visit
    
  1.    click ‘+’ to create a ‘Care at birth’ event on 2017-02-01
    
  1.    for WHOMCH pregnancy outcome, select ‘Stillbirth’, and complete the event.
    

The rule which should hide the postnatal program stage has not fired. In the browser console, the following two errors are recorded are:

· * Expression #{currentProgranancyOutcome} != ‘Live birth’ contains variable currentProgranancyOutcome - but this variable is not defined.*

· * Expression with id rule:xOm49QX4Nsc could not be run. Original condition was: #{currentProgranancyOutcome} != ‘Live birth’ - Evaluation ended up as:#{currentProgranancyOutcome} != ‘Live birth’ - error message:SyntaxError: Invalid or unexpected token*

The important error seems to be the second one, as in many cases this is appearing in isolation (without the first one), and is still breaking program rules. For example, I’ve loaded the Cameroon EquityTool into the Play Demo instance, and any attempt to enter new Event Capture events for this program keeps generating similar error messages in the browser console, eg:

· * Expression with id rule:gXwzJFfRn5m could not be run. Original condition was: #{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - Evaluation ended up as:#{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - error message:SyntaxError: Invalid or unexpected token*

These program rules have been extensively tested on earlier releases, and worked well – where users used to be prevented from entering invalid surveys, they can now save invalid surveys.

This is very serious, as it means that in 2.26, users can now bypass many program rules, and save incomplete or incorrect data. But I’m also aware that there have recently been build issues with 2.25 and 2.26 releases – before I log a bug on Jira, could this be related to those build issues?

Regards, Sam.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Sam,

I understand your frustration with already published and deployed programs, and that you hope for some software fix that will make them work.

For everybody else, I would strongly recommend the naming practice that I’ve been following (and beating into all my colleagues in HISP-South Africa) since DHIS version 1.0 beta was released in May 1997: NEVER, repeat NEVER, use any non-alphanumeric character except underscore for any meta-data or variable name if you can avoid it. The problem is that ampersands, asterisks, forward and backslashes, equal signs, hashes - almost every special character like that is a restricted character in SOME context. Yes, we have escape sequences and all that - but look through the dev list over the last few years, and you see these kind of issues coming up again and again.

In 1.4 we blocked the most common problems (double spaces, leading spaces, all these special characters, em-dash instead of hyphen, etc etc) in the database itself, so users could not add them in the first place, but that method was dropped off when DHIS 1.4 was “ported” to DHIS2.

Regards

Calle

···

On 28 April 2017 at 00:32, Sam Johnson samuel.johnson@qebo.co.uk wrote:

Hi Markus,

Just following up my question below – is there any chance we could get this treated as a bug, and fixed in 2.26? This issue has crippled 30-odd programs that Metrics for Management (M4M) has already published and deployed, but I’m guessing that the fix is probably just be a very simple string-handling change…

Many thanks for any help you might be able to give!

Cheers, Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Sam Johnson samuel.johnson@qebo.co.uk

Date: Friday, 21 April 2017 at 17:58

To: Markus Bekken markus@dhis2.org

Cc: Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett andrea@m4mgmt.org, DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing

Feedback

Hi Markus (and other devs),

Many thanks for diagnosing this so quickly. You’re spot-on, it’s the dash in the source name that’s causing the rules to fail – if I remove dashes from all source names, the rules fire without problems. In fact, this seems wider than just dashes – program rules that were failing in another instance seem to now work if I similarly remove brackets and spaces from source names.

Would it be at all possible to get this treated as a bug and fixed in 2.26, rather than treated as an improvement for a future version of DHIS2? These rules were all working perfectly when we did intensive testing on our programs in 2.25 (eg they work properly in 2.25 da6a2df / 2017-02-02 06:43), so this limitation seems to have been recently introduced (very likely by accident, since other element names allow dashes), and it’s breaking configurations that have worked well up until now.

The reason we’re so keen on this is that Metrics for Management has published XMLs for 29 EquityTool programs, and these have now been downloaded and installed in a wide range of DHIS2 instances. It will be a huge job to not only update the source names in all of these 29 programs and re-publish them, but also contact all existing users and ask them to reinstall the corrected programs. If there’s any chance we could get the program rules in 2.26 working as they did before, this would save us an enormous amount of work…

Many thanks for any help you might be able to give us with this!

Regards, Sam.

From: Markus Bekken markus@dhis2.org

Date: Wednesday, 19 April 2017 at 14:55

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: DHIS2 Developers dhis2-devs@lists.launchpad.net, Kenzo Fry kenzo.fry.consultant@gmail.com

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Hey Sam!

Thanks for reporting.

I tested WHO RMNCH and ‘Hide program stage’ on Play, and found that it the problem is in fact what is indicated by the console message in point 4 below. The source field #{currentProganancyOutcome} did not exist. Seems someone has renamed it to

#{Current Progranancy Outcome} - see below to the left:

Such a name change should ideally clean up the expressions using the source field, or at least flag the ones that has become illegal as a result of the name change. At the moment however you would have to manually update the expressions after a name change. Updated the expression now on Play, and the expression runs fine:

Expression with id rule:xOm49QX4Nsc was successfully run. Original condition was: #{current Progranancy Outcome} != ‘Live birth’ - Evaluation ended up as:’’ != ‘Live birth’ - Result of evaluation was:true

In general your rules hiding program stages should work fine, let me know if you find any other problems here.

Looking into the second part of the mail regarding Event Capture - this is a different problem: There is a dash in the source field name. Registered an improvement in Jira based on this: https://jira.dhis2.org/browse/DHIS2-1449 - but avoiding dashes is a possible workaround.

Best regards,

Markus

  1. apr. 2017 kl. 12.43 skrev Sam Johnson samuel.johnson@qebo.co.uk:

Hi Devs,

I’m finding that a number of program rules aren’t firing in 2.26 – this is happening with both Event Capture and Tracker Capture.

This includes the sample ‘Hide Program Stage’ rule which has been added to the WHO RMNCH Tracker for the 2.26 release. To reproduce in Play demo:

  1.    add a new registration to the WHO RMNCH Tracker with date 2017-01-01
    
  1.    complete the initial visit, and click ‘cancel’ to prevent creation of a second antenatal care visit
    
  1.    click ‘+’ to create a ‘Care at birth’ event on 2017-02-01
    
  1.    for WHOMCH pregnancy outcome, select ‘Stillbirth’, and complete the event.
    

The rule which should hide the postnatal program stage has not fired. In the browser console, the following two errors are recorded are:

· * Expression #{currentProgranancyOutcome} != ‘Live birth’ contains variable currentProgranancyOutcome - but this variable is not defined.*

· * Expression with id rule:xOm49QX4Nsc could not be run. Original condition was: #{currentProgranancyOutcome} != ‘Live birth’ - Evaluation ended up as:#{currentProgranancyOutcome} != ‘Live birth’ - error message:SyntaxError: Invalid or unexpected token*

The important error seems to be the second one, as in many cases this is appearing in isolation (without the first one), and is still breaking program rules. For example, I’ve loaded the Cameroon EquityTool into the Play Demo instance, and any attempt to enter new Event Capture events for this program keeps generating similar error messages in the browser console, eg:

· * Expression with id rule:gXwzJFfRn5m could not be run. Original condition was: #{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - Evaluation ended up as:#{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - error message:SyntaxError: Invalid or unexpected token*

These program rules have been extensively tested on earlier releases, and worked well – where users used to be prevented from entering invalid surveys, they can now save invalid surveys.

This is very serious, as it means that in 2.26, users can now bypass many program rules, and save incomplete or incorrect data. But I’m also aware that there have recently been build issues with 2.25 and 2.26 releases – before I log a bug on Jira, could this be related to those build issues?

Regards, Sam.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Hi Calle,

I heartily agree, and I do actually use camelCase or similar (without spaces) for all codes. However, the one departure from strict Camel Case that I quite frequently use is to partition codes with dashes (hyphens), because we use very structured naming conventions, and they can quickly become illegible without proper partitioning – eg in ‘EQT-CM11-Q06a’ it’s very easy to immediately pick out the country, year and question number, whereas it’s much more difficult in ‘EQTCM11Q06a’. (It sounds like you add underscores for the same purpose.)

Markus has noted in his ticket that both dashes and underscores should be supported in DHIS2 codes, and I’ve never had problems with dashes before, but given your very long history with DHIS2, I might draw from your experience, and shift to using underscores for future projects… :wink:

In the meantime, at Markus’ suggestion I’ve just created a new bug ticket for this, linked to the longer-term validation ‘improvement’ ticket that he created:

https://jira.dhis2.org/browse/DHIS2-1503

Cheers, Sam.

···

From: Calle Hedberg calle.hedberg@gmail.com

Reply-To: "calle.hedberg@gmail.com" calle.hedberg@gmail.com

Date: Friday, 28 April 2017 at 06:37

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: Markus Bekken markus@dhis2.org, Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett andrea@m4mgmt.org, DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Sam,

I understand your frustration with already published and deployed programs, and that you hope for some software fix that will make them work.

For everybody else, I would strongly recommend the naming practice that I’ve been following (and beating into all my colleagues in HISP-South Africa) since DHIS version 1.0 beta was released in May 1997: NEVER, repeat NEVER, use any non-alphanumeric character except underscore for any meta-data or variable name if you can avoid it. The problem is that ampersands, asterisks, forward and backslashes, equal signs, hashes - almost every special character like that is a restricted character in SOME context. Yes, we have escape sequences and all that - but look through the dev list over the last few years, and you see these kind of issues coming up again and again.

In 1.4 we blocked the most common problems (double spaces, leading spaces, all these special characters, em-dash instead of hyphen, etc etc) in the database itself, so users could not add them in the first place, but that method was dropped off when DHIS 1.4 was “ported” to DHIS2.

Regards

Calle

On 28 April 2017 at 00:32, Sam Johnson samuel.johnson@qebo.co.uk wrote:

Hi Markus,

Just following up my question below – is there any chance we could get this treated as a bug, and fixed in 2.26? This issue has crippled 30-odd programs that Metrics for Management (M4M) has already published and deployed, but I’m guessing that the fix is probably just be a very simple string-handling change…

Many thanks for any help you might be able to give!

Cheers, Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Sam Johnson samuel.johnson@qebo.co.uk

Date: Friday, 21 April 2017 at 17:58

To: Markus Bekken markus@dhis2.org

Cc: Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett andrea@m4mgmt.org, DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing

Feedback

Hi Markus (and other devs),

Many thanks for diagnosing this so quickly. You’re spot-on, it’s the dash in the source name that’s causing the rules to fail – if I remove dashes from all source names, the rules fire without problems. In fact, this seems wider than just dashes – program rules that were failing in another instance seem to now work if I similarly remove brackets and spaces from source names.

** Would it be at all possible to get this treated as a bug and fixed in 2.26, rather than treated as an improvement for a future version of DHIS2?** These rules were all working perfectly when we did intensive testing on our programs in 2.25 (eg they work properly in 2.25 da6a2df / 2017-02-02 06:43), so this limitation seems to have been recently introduced (very likely by accident, since other element names allow dashes), and it’s breaking configurations that have worked well up until now.

The reason we’re so keen on this is that Metrics for Management has published XMLs for 29 EquityTool programs, and these have now been downloaded and installed in a wide range of DHIS2 instances. It will be a huge job to not only update the source names in all of these 29 programs and re-publish them, but also contact all existing users and ask them to reinstall the corrected programs. If there’s any chance we could get the program rules in 2.26 working as they did before, this would save us an enormous amount of work…

Many thanks for any help you might be able to give us with this!

Regards, Sam.

From: Markus Bekken markus@dhis2.org

Date: Wednesday, 19 April 2017 at 14:55

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: DHIS2 Developers dhis2-devs@lists.launchpad.net, Kenzo Fry kenzo.fry.consultant@gmail.com

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Hey Sam!

Thanks for reporting.

I tested WHO RMNCH and ‘Hide program stage’ on Play, and found that it the problem is in fact what is indicated by the console message in point 4 below. The source field #{currentProganancyOutcome} did not exist. Seems someone has renamed it to

#{Current Progranancy Outcome} - see below to the left:

Such a name change should ideally clean up the expressions using the source field, or at least flag the ones that has become illegal as a result of the name change. At the moment however you would have to manually update the expressions after a name change. Updated the expression now on Play, and the expression runs fine:

  • Expression with id rule:xOm49QX4Nsc was successfully run. Original condition was: #{current Progranancy Outcome} != ‘Live birth’ - Evaluation ended up as:’’ != ‘Live birth’ - Result of evaluation was:true*

In general your rules hiding program stages should work fine, let me know if you find any other problems here.

Looking into the second part of the mail regarding Event Capture - this is a different problem: There is a dash in the source field name. Registered an improvement in Jira based on this: https://jira.dhis2.org/browse/DHIS2-1449 - but avoiding dashes is a possible workaround.

Best regards,

Markus

  1. apr. 2017 kl. 12.43 skrev Sam Johnson samuel.johnson@qebo.co.uk:

Hi Devs,

I’m finding that a number of program rules aren’t firing in 2.26 – this is happening with both Event Capture and Tracker Capture.

This includes the sample ‘Hide Program Stage’ rule which has been added to the WHO RMNCH Tracker for the 2.26 release. To reproduce in Play demo:

  1.    add a new registration to the WHO RMNCH Tracker with date 2017-01-01
    
  1.    complete the initial visit, and click ‘cancel’ to prevent creation of a second antenatal care visit
    
  1.    click ‘+’ to create a ‘Care at birth’ event on 2017-02-01
    
  1.    for WHOMCH pregnancy outcome, select ‘Stillbirth’, and complete the event.
    

The rule which should hide the postnatal program stage has not fired. In the browser console, the following two errors are recorded are:

· * Expression #{currentProgranancyOutcome} != ‘Live birth’ contains variable currentProgranancyOutcome - but this variable is not defined.*

· * Expression with id rule:xOm49QX4Nsc could not be run. Original condition was: #{currentProgranancyOutcome} != ‘Live birth’ - Evaluation ended up as:#{currentProgranancyOutcome} != ‘Live birth’ - error message:SyntaxError: Invalid or unexpected token*

The important error seems to be the second one, as in many cases this is appearing in isolation (without the first one), and is still breaking program rules. For example, I’ve loaded the Cameroon EquityTool into the Play Demo instance, and any attempt to enter new Event Capture events for this program keeps generating similar error messages in the browser console, eg:

· * Expression with id rule:gXwzJFfRn5m could not be run. Original condition was: #{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - Evaluation ended up as:#{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - error message:SyntaxError: Invalid or unexpected token*

These program rules have been extensively tested on earlier releases, and worked well – where users used to be prevented from entering invalid surveys, they can now save invalid surveys.

This is very serious, as it means that in 2.26, users can now bypass many program rules, and save incomplete or incorrect data. But I’m also aware that there have recently been build issues with 2.25 and 2.26 releases – before I log a bug on Jira, could this be related to those build issues?

Regards, Sam.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Sam

My main issue with hyphens are usually related the source of variables or data elements or indicators etc being in a MS Word document, where Word tend to automatically replace hyphens with em-dash (the “long” hyphen). Underscore on the other hand has been common as a delimiter for time immemorial, and I’ve never seen it being used as a special character in any type of syntax.

BTW, also keep an eye on spelling - I see in your source fields that you have “chronich ypertension” and “current Progranancy outcome”. Spelling mistakes often results in people adding multiple variables for the same thing but with different spelling. You cannot really set up check algorithms for spelling (beyond what’s in the system by default) the same way you can set them up to block special characters, so it requires a keen human eye.

My 2c worth

Calle

···

On 28 April 2017 at 12:47, Sam Johnson samuel.johnson@qebo.co.uk wrote:

Hi Calle,

I heartily agree, and I do actually use camelCase or similar (without spaces) for all codes. However, the one departure from strict Camel Case that I quite frequently use is to partition codes with dashes (hyphens), because we use very structured naming conventions, and they can quickly become illegible without proper partitioning – eg in ‘EQT-CM11-Q06a’ it’s very easy to immediately pick out the country, year and question number, whereas it’s much more difficult in ‘EQTCM11Q06a’. (It sounds like you add underscores for the same purpose.)

Markus has noted in his ticket that both dashes and underscores should be supported in DHIS2 codes, and I’ve never had problems with dashes before, but given your very long history with DHIS2, I might draw from your experience, and shift to using underscores for future projects… :wink:

In the meantime, at Markus’ suggestion I’ve just created a new bug ticket for this, linked to the longer-term validation ‘improvement’ ticket that he created:

https://jira.dhis2.org/browse/DHIS2-1503

Cheers, Sam.

From: Calle Hedberg calle.hedberg@gmail.com

Reply-To: "calle.hedberg@gmail.com" calle.hedberg@gmail.com

Date: Friday, 28 April 2017 at 06:37

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: Markus Bekken markus@dhis2.org, Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett andrea@m4mgmt.org, DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Sam,

I understand your frustration with already published and deployed programs, and that you hope for some software fix that will make them work.

For everybody else, I would strongly recommend the naming practice that I’ve been following (and beating into all my colleagues in HISP-South Africa) since DHIS version 1.0 beta was released in May 1997: NEVER, repeat NEVER, use any non-alphanumeric character except underscore for any meta-data or variable name if you can avoid it. The problem is that ampersands, asterisks, forward and backslashes, equal signs, hashes - almost every special character like that is a restricted character in SOME context. Yes, we have escape sequences and all that - but look through the dev list over the last few years, and you see these kind of issues coming up again and again.

In 1.4 we blocked the most common problems (double spaces, leading spaces, all these special characters, em-dash instead of hyphen, etc etc) in the database itself, so users could not add them in the first place, but that method was dropped off when DHIS 1.4 was “ported” to DHIS2.

Regards

Calle

On 28 April 2017 at 00:32, Sam Johnson samuel.johnson@qebo.co.uk wrote:

Hi Markus,

Just following up my question below – is there any chance we could get this treated as a bug, and fixed in 2.26? This issue has crippled 30-odd programs that Metrics for Management (M4M) has already published and deployed, but I’m guessing that the fix is probably just be a very simple string-handling change…

Many thanks for any help you might be able to give!

Cheers, Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Sam Johnson samuel.johnson@qebo.co.uk

Date: Friday, 21 April 2017 at 17:58

To: Markus Bekken markus@dhis2.org

Cc: Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett andrea@m4mgmt.org, DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing

Feedback

Hi Markus (and other devs),

Many thanks for diagnosing this so quickly. You’re spot-on, it’s the dash in the source name that’s causing the rules to fail – if I remove dashes from all source names, the rules fire without problems. In fact, this seems wider than just dashes – program rules that were failing in another instance seem to now work if I similarly remove brackets and spaces from source names.

** Would it be at all possible to get this treated as a bug and fixed in 2.26, rather than treated as an improvement for a future version of DHIS2?** These rules were all working perfectly when we did intensive testing on our programs in 2.25 (eg they work properly in 2.25 da6a2df / 2017-02-02 06:43), so this limitation seems to have been recently introduced (very likely by accident, since other element names allow dashes), and it’s breaking configurations that have worked well up until now.

The reason we’re so keen on this is that Metrics for Management has published XMLs for 29 EquityTool programs, and these have now been downloaded and installed in a wide range of DHIS2 instances. It will be a huge job to not only update the source names in all of these 29 programs and re-publish them, but also contact all existing users and ask them to reinstall the corrected programs. If there’s any chance we could get the program rules in 2.26 working as they did before, this would save us an enormous amount of work…

Many thanks for any help you might be able to give us with this!

Regards, Sam.

From: Markus Bekken markus@dhis2.org

Date: Wednesday, 19 April 2017 at 14:55

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: DHIS2 Developers dhis2-devs@lists.launchpad.net, Kenzo Fry kenzo.fry.consultant@gmail.com

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Hey Sam!

Thanks for reporting.

I tested WHO RMNCH and ‘Hide program stage’ on Play, and found that it the problem is in fact what is indicated by the console message in point 4 below. The source field #{currentProganancyOutcome} did not exist. Seems someone has renamed it to

#{Current Progranancy Outcome} - see below to the left:

Such a name change should ideally clean up the expressions using the source field, or at least flag the ones that has become illegal as a result of the name change. At the moment however you would have to manually update the expressions after a name change. Updated the expression now on Play, and the expression runs fine:

  • Expression with id rule:xOm49QX4Nsc was successfully run. Original condition was: #{current Progranancy Outcome} != ‘Live birth’ - Evaluation ended up as:’’ != ‘Live birth’ - Result of evaluation was:true*

In general your rules hiding program stages should work fine, let me know if you find any other problems here.

Looking into the second part of the mail regarding Event Capture - this is a different problem: There is a dash in the source field name. Registered an improvement in Jira based on this: https://jira.dhis2.org/browse/DHIS2-1449 - but avoiding dashes is a possible workaround.

Best regards,

Markus

  1. apr. 2017 kl. 12.43 skrev Sam Johnson samuel.johnson@qebo.co.uk:

Hi Devs,

I’m finding that a number of program rules aren’t firing in 2.26 – this is happening with both Event Capture and Tracker Capture.

This includes the sample ‘Hide Program Stage’ rule which has been added to the WHO RMNCH Tracker for the 2.26 release. To reproduce in Play demo:

  1.    add a new registration to the WHO RMNCH Tracker with date 2017-01-01
    
  1.    complete the initial visit, and click ‘cancel’ to prevent creation of a second antenatal care visit
    
  1.    click ‘+’ to create a ‘Care at birth’ event on 2017-02-01
    
  1.    for WHOMCH pregnancy outcome, select ‘Stillbirth’, and complete the event.
    

The rule which should hide the postnatal program stage has not fired. In the browser console, the following two errors are recorded are:

· * Expression #{currentProgranancyOutcome} != ‘Live birth’ contains variable currentProgranancyOutcome - but this variable is not defined.*

· * Expression with id rule:xOm49QX4Nsc could not be run. Original condition was: #{currentProgranancyOutcome} != ‘Live birth’ - Evaluation ended up as:#{currentProgranancyOutcome} != ‘Live birth’ - error message:SyntaxError: Invalid or unexpected token*

The important error seems to be the second one, as in many cases this is appearing in isolation (without the first one), and is still breaking program rules. For example, I’ve loaded the Cameroon EquityTool into the Play Demo instance, and any attempt to enter new Event Capture events for this program keeps generating similar error messages in the browser console, eg:

· * Expression with id rule:gXwzJFfRn5m could not be run. Original condition was: #{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - Evaluation ended up as:#{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - error message:SyntaxError: Invalid or unexpected token*

These program rules have been extensively tested on earlier releases, and worked well – where users used to be prevented from entering invalid surveys, they can now save invalid surveys.

This is very serious, as it means that in 2.26, users can now bypass many program rules, and save incomplete or incorrect data. But I’m also aware that there have recently been build issues with 2.25 and 2.26 releases – before I log a bug on Jira, could this be related to those build issues?

Regards, Sam.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Hi Calle,

BTW, also keep an eye on spelling - I see in your source fields that you have “chronich ypertension” and “current Progranancy outcome”.

That RMNCH Tracker isn’t our program, I’ve just used it to replicate the bugs I found in our own programs. (We always try to replicate any bugs we find in Oslo’s own Play demo site, so the developers can easily see what we’re talking about.)

The programs I was talking about are the EquityTool surveys (http://www.equitytool.org/dhis2/ ), and all of the codes and names are automatically-generated, to ensure they’re consistent across countries – plus we rigorously test and check everything before release. I’ll buy you lunch if you find a spelling mistake in one of our programs. :wink:

Cheers, Sam.

···

From: Calle Hedberg calle.hedberg@gmail.com

Reply-To: "calle.hedberg@gmail.com" calle.hedberg@gmail.com

Date: Sunday, 30 April 2017 at 09:38

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: DHIS2 Developers dhis2-devs@lists.launchpad.net, Markus Bekken markus@dhis2.org, Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett andrea@m4mgmt.org

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Sam

My main issue with hyphens are usually related the source of variables or data elements or indicators etc being in a MS Word document, where Word tend to automatically replace hyphens with em-dash (the “long” hyphen). Underscore on the other hand has been common as a delimiter for time immemorial, and I’ve never seen it being used as a special character in any type of syntax.

BTW, also keep an eye on spelling - I see in your source fields that you have “chronich ypertension” and “current Progranancy outcome”. Spelling mistakes often results in people adding multiple variables for the same thing but with different spelling. You cannot really set up check algorithms for spelling (beyond what’s in the system by default) the same way you can set them up to block special characters, so it requires a keen human eye.

My 2c worth

Calle

On 28 April 2017 at 12:47, Sam Johnson samuel.johnson@qebo.co.uk wrote:

Hi Calle,

I heartily agree, and I do actually use camelCase or similar (without spaces) for all codes. However, the one departure from strict Camel Case that I quite frequently use is to partition codes with dashes (hyphens), because we use very structured naming conventions, and they can quickly become illegible without proper partitioning – eg in ‘EQT-CM11-Q06a’ it’s very easy to immediately pick out the country, year and question number, whereas it’s much more difficult in ‘EQTCM11Q06a’. (It sounds like you add underscores for the same purpose.)

Markus has noted in his ticket that both dashes and underscores should be supported in DHIS2 codes, and I’ve never had problems with dashes before, but given your very long history with DHIS2, I might draw from your experience, and shift to using underscores for future projects… :wink:

In the meantime, at Markus’ suggestion I’ve just created a new bug ticket for this, linked to the longer-term validation ‘improvement’ ticket that he created:

https://jira.dhis2.org/browse/DHIS2-1503

Cheers, Sam.

From: Calle Hedberg calle.hedberg@gmail.com

Reply-To: "calle.hedberg@gmail.com" calle.hedberg@gmail.com

Date: Friday, 28 April 2017 at 06:37

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: Markus Bekken markus@dhis2.org, Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett <andrea@m4mgmt.org >, DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Sam,

I understand your frustration with already published and deployed programs, and that you hope for some software fix that will make them work.

For everybody else, I would strongly recommend the naming practice that I’ve been following (and beating into all my colleagues in HISP-South Africa) since DHIS version 1.0 beta was released in May 1997: NEVER, repeat NEVER, use any non-alphanumeric character except underscore for any meta-data or variable name if you can avoid it. The problem is that ampersands, asterisks, forward and backslashes, equal signs, hashes - almost every special character like that is a restricted character in SOME context. Yes, we have escape sequences and all that - but look through the dev list over the last few years, and you see these kind of issues coming up again and again.

In 1.4 we blocked the most common problems (double spaces, leading spaces, all these special characters, em-dash instead of hyphen, etc etc) in the database itself, so users could not add them in the first place, but that method was dropped off when DHIS 1.4 was “ported” to DHIS2.

Regards

Calle

On 28 April 2017 at 00:32, Sam Johnson samuel.johnson@qebo.co.uk wrote:

Hi Markus,

Just following up my question below – is there any chance we could get this treated as a bug, and fixed in 2.26? This issue has crippled 30-odd programs that Metrics for Management (M4M) has already published and deployed, but I’m guessing that the fix is probably just be a very simple string-handling change…

Many thanks for any help you might be able to give!

Cheers, Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Sam Johnson samuel.johnson@qebo.co.uk

Date: Friday, 21 April 2017 at 17:58

To: Markus Bekken markus@dhis2.org

Cc: Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett andrea@m4mgmt.org, DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing

Feedback

Hi Markus (and other devs),

Many thanks for diagnosing this so quickly. You’re spot-on, it’s the dash in the source name that’s causing the rules to fail – if I remove dashes from all source names, the rules fire without problems. In fact, this seems wider than just dashes – program rules that were failing in another instance seem to now work if I similarly remove brackets and spaces from source names.

** Would it be at all possible to get this treated as a bug and fixed in 2.26, rather than treated as an improvement for a future version of DHIS2?** These rules were all working perfectly when we did intensive testing on our programs in 2.25 (eg they work properly in 2.25 da6a2df / 2017-02-02 06:43), so this limitation seems to have been recently introduced (very likely by accident, since other element names allow dashes), and it’s breaking configurations that have worked well up until now.

The reason we’re so keen on this is that Metrics for Management has published XMLs for 29 EquityTool programs, and these have now been downloaded and installed in a wide range of DHIS2 instances. It will be a huge job to not only update the source names in all of these 29 programs and re-publish them, but also contact all existing users and ask them to reinstall the corrected programs. If there’s any chance we could get the program rules in 2.26 working as they did before, this would save us an enormous amount of work…

Many thanks for any help you might be able to give us with this!

Regards, Sam.

From: Markus Bekken markus@dhis2.org

Date: Wednesday, 19 April 2017 at 14:55

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: DHIS2 Developers dhis2-devs@lists.launchpad.net, Kenzo Fry kenzo.fry.consultant@gmail.com

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Hey Sam!

Thanks for reporting.

I tested WHO RMNCH and ‘Hide program stage’ on Play, and found that it the problem is in fact what is indicated by the console message in point 4 below. The source field #{currentProganancyOutcome} did not exist. Seems someone has renamed it to

#{Current Progranancy Outcome} - see below to the left:

Such a name change should ideally clean up the expressions using the source field, or at least flag the ones that has become illegal as a result of the name change. At the moment however you would have to manually update the expressions after a name change. Updated the expression now on Play, and the expression runs fine:

  • Expression with id rule:xOm49QX4Nsc was successfully run. Original condition was: #{current Progranancy Outcome} != ‘Live birth’ - Evaluation ended up as:’’ != ‘Live birth’ - Result of evaluation was:true*

In general your rules hiding program stages should work fine, let me know if you find any other problems here.

Looking into the second part of the mail regarding Event Capture - this is a different problem: There is a dash in the source field name. Registered an improvement in Jira based on this: https://jira.dhis2.org/browse/DHIS2-1449 - but avoiding dashes is a possible workaround.

Best regards,

Markus

  1. apr. 2017 kl. 12.43 skrev Sam Johnson samuel.johnson@qebo.co.uk:

Hi Devs,

I’m finding that a number of program rules aren’t firing in 2.26 – this is happening with both Event Capture and Tracker Capture.

This includes the sample ‘Hide Program Stage’ rule which has been added to the WHO RMNCH Tracker for the 2.26 release. To reproduce in Play demo:

  1.    add a new registration to the WHO RMNCH Tracker with date 2017-01-01
    
  1.    complete the initial visit, and click ‘cancel’ to prevent creation of a second antenatal care visit
    
  1.    click ‘+’ to create a ‘Care at birth’ event on 2017-02-01
    
  1.    for WHOMCH pregnancy outcome, select ‘Stillbirth’, and complete the event.
    

The rule which should hide the postnatal program stage has not fired. In the browser console, the following two errors are recorded are:

· * Expression #{currentProgranancyOutcome} != ‘Live birth’ contains variable currentProgranancyOutcome - but this variable is not defined.*

· * Expression with id rule:xOm49QX4Nsc could not be run. Original condition was: #{currentProgranancyOutcome} != ‘Live birth’ - Evaluation ended up as:#{currentProgranancyOutcome} != ‘Live birth’ - error message:SyntaxError: Invalid or unexpected token*

The important error seems to be the second one, as in many cases this is appearing in isolation (without the first one), and is still breaking program rules. For example, I’ve loaded the Cameroon EquityTool into the Play Demo instance, and any attempt to enter new Event Capture events for this program keeps generating similar error messages in the browser console, eg:

· * Expression with id rule:gXwzJFfRn5m could not be run. Original condition was: #{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - Evaluation ended up as:#{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - error message:SyntaxError: Invalid or unexpected token*

These program rules have been extensively tested on earlier releases, and worked well – where users used to be prevented from entering invalid surveys, they can now save invalid surveys.

This is very serious, as it means that in 2.26, users can now bypass many program rules, and save incomplete or incorrect data. But I’m also aware that there have recently been build issues with 2.25 and 2.26 releases – before I log a bug on Jira, could this be related to those build issues?

Regards, Sam.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


:wink:

OK, noted.

In any case, my comment was general, not for you specifically - I see a lot of sloppy data element and indicator design, and not paying attention to spelling and intuitive clarity are the two main culprits…

Regards

Calle

···

On 30 April 2017 at 12:05, Sam Johnson samuel.johnson@qebo.co.uk wrote:

Hi Calle,

BTW, also keep an eye on spelling - I see in your source fields that you have “chronich ypertension” and “current Progranancy outcome”.

That RMNCH Tracker isn’t our program, I’ve just used it to replicate the bugs I found in our own programs. (We always try to replicate any bugs we find in Oslo’s own Play demo site, so the developers can easily see what we’re talking about.)

The programs I was talking about are the EquityTool surveys (http://www.equitytool.org/dhis2/ ), and all of the codes and names are automatically-generated, to ensure they’re consistent across countries – plus we rigorously test and check everything before release. I’ll buy you lunch if you find a spelling mistake in one of our programs. :wink:

Cheers, Sam.

From: Calle Hedberg calle.hedberg@gmail.com

Reply-To: "calle.hedberg@gmail.com" calle.hedberg@gmail.com

Date: Sunday, 30 April 2017 at 09:38

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: DHIS2 Developers dhis2-devs@lists.launchpad.net, Markus Bekken markus@dhis2.org, Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett andrea@m4mgmt.org

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Sam

My main issue with hyphens are usually related the source of variables or data elements or indicators etc being in a MS Word document, where Word tend to automatically replace hyphens with em-dash (the “long” hyphen). Underscore on the other hand has been common as a delimiter for time immemorial, and I’ve never seen it being used as a special character in any type of syntax.

BTW, also keep an eye on spelling - I see in your source fields that you have “chronich ypertension” and “current Progranancy outcome”. Spelling mistakes often results in people adding multiple variables for the same thing but with different spelling. You cannot really set up check algorithms for spelling (beyond what’s in the system by default) the same way you can set them up to block special characters, so it requires a keen human eye.

My 2c worth

Calle

On 28 April 2017 at 12:47, Sam Johnson samuel.johnson@qebo.co.uk wrote:

Hi Calle,

I heartily agree, and I do actually use camelCase or similar (without spaces) for all codes. However, the one departure from strict Camel Case that I quite frequently use is to partition codes with dashes (hyphens), because we use very structured naming conventions, and they can quickly become illegible without proper partitioning – eg in ‘EQT-CM11-Q06a’ it’s very easy to immediately pick out the country, year and question number, whereas it’s much more difficult in ‘EQTCM11Q06a’. (It sounds like you add underscores for the same purpose.)

Markus has noted in his ticket that both dashes and underscores should be supported in DHIS2 codes, and I’ve never had problems with dashes before, but given your very long history with DHIS2, I might draw from your experience, and shift to using underscores for future projects… :wink:

In the meantime, at Markus’ suggestion I’ve just created a new bug ticket for this, linked to the longer-term validation ‘improvement’ ticket that he created:

https://jira.dhis2.org/browse/DHIS2-1503

Cheers, Sam.

From: Calle Hedberg calle.hedberg@gmail.com

Reply-To: "calle.hedberg@gmail.com" calle.hedberg@gmail.com

Date: Friday, 28 April 2017 at 06:37

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: Markus Bekken markus@dhis2.org, Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett <andrea@m4mgmt.org >, DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Sam,

I understand your frustration with already published and deployed programs, and that you hope for some software fix that will make them work.

For everybody else, I would strongly recommend the naming practice that I’ve been following (and beating into all my colleagues in HISP-South Africa) since DHIS version 1.0 beta was released in May 1997: NEVER, repeat NEVER, use any non-alphanumeric character except underscore for any meta-data or variable name if you can avoid it. The problem is that ampersands, asterisks, forward and backslashes, equal signs, hashes - almost every special character like that is a restricted character in SOME context. Yes, we have escape sequences and all that - but look through the dev list over the last few years, and you see these kind of issues coming up again and again.

In 1.4 we blocked the most common problems (double spaces, leading spaces, all these special characters, em-dash instead of hyphen, etc etc) in the database itself, so users could not add them in the first place, but that method was dropped off when DHIS 1.4 was “ported” to DHIS2.

Regards

Calle

On 28 April 2017 at 00:32, Sam Johnson samuel.johnson@qebo.co.uk wrote:

Hi Markus,

Just following up my question below – is there any chance we could get this treated as a bug, and fixed in 2.26? This issue has crippled 30-odd programs that Metrics for Management (M4M) has already published and deployed, but I’m guessing that the fix is probably just be a very simple string-handling change…

Many thanks for any help you might be able to give!

Cheers, Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Sam Johnson samuel.johnson@qebo.co.uk

Date: Friday, 21 April 2017 at 17:58

To: Markus Bekken markus@dhis2.org

Cc: Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett andrea@m4mgmt.org, DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing

Feedback

Hi Markus (and other devs),

Many thanks for diagnosing this so quickly. You’re spot-on, it’s the dash in the source name that’s causing the rules to fail – if I remove dashes from all source names, the rules fire without problems. In fact, this seems wider than just dashes – program rules that were failing in another instance seem to now work if I similarly remove brackets and spaces from source names.

** Would it be at all possible to get this treated as a bug and fixed in 2.26, rather than treated as an improvement for a future version of DHIS2?** These rules were all working perfectly when we did intensive testing on our programs in 2.25 (eg they work properly in 2.25 da6a2df / 2017-02-02 06:43), so this limitation seems to have been recently introduced (very likely by accident, since other element names allow dashes), and it’s breaking configurations that have worked well up until now.

The reason we’re so keen on this is that Metrics for Management has published XMLs for 29 EquityTool programs, and these have now been downloaded and installed in a wide range of DHIS2 instances. It will be a huge job to not only update the source names in all of these 29 programs and re-publish them, but also contact all existing users and ask them to reinstall the corrected programs. If there’s any chance we could get the program rules in 2.26 working as they did before, this would save us an enormous amount of work…

Many thanks for any help you might be able to give us with this!

Regards, Sam.

From: Markus Bekken markus@dhis2.org

Date: Wednesday, 19 April 2017 at 14:55

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: DHIS2 Developers dhis2-devs@lists.launchpad.net, Kenzo Fry kenzo.fry.consultant@gmail.com

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Hey Sam!

Thanks for reporting.

I tested WHO RMNCH and ‘Hide program stage’ on Play, and found that it the problem is in fact what is indicated by the console message in point 4 below. The source field #{currentProganancyOutcome} did not exist. Seems someone has renamed it to

#{Current Progranancy Outcome} - see below to the left:

Such a name change should ideally clean up the expressions using the source field, or at least flag the ones that has become illegal as a result of the name change. At the moment however you would have to manually update the expressions after a name change. Updated the expression now on Play, and the expression runs fine:

  • Expression with id rule:xOm49QX4Nsc was successfully run. Original condition was: #{current Progranancy Outcome} != ‘Live birth’ - Evaluation ended up as:’’ != ‘Live birth’ - Result of evaluation was:true*

In general your rules hiding program stages should work fine, let me know if you find any other problems here.

Looking into the second part of the mail regarding Event Capture - this is a different problem: There is a dash in the source field name. Registered an improvement in Jira based on this: https://jira.dhis2.org/browse/DHIS2-1449 - but avoiding dashes is a possible workaround.

Best regards,

Markus

  1. apr. 2017 kl. 12.43 skrev Sam Johnson samuel.johnson@qebo.co.uk:

Hi Devs,

I’m finding that a number of program rules aren’t firing in 2.26 – this is happening with both Event Capture and Tracker Capture.

This includes the sample ‘Hide Program Stage’ rule which has been added to the WHO RMNCH Tracker for the 2.26 release. To reproduce in Play demo:

  1.    add a new registration to the WHO RMNCH Tracker with date 2017-01-01
    
  1.    complete the initial visit, and click ‘cancel’ to prevent creation of a second antenatal care visit
    
  1.    click ‘+’ to create a ‘Care at birth’ event on 2017-02-01
    
  1.    for WHOMCH pregnancy outcome, select ‘Stillbirth’, and complete the event.
    

The rule which should hide the postnatal program stage has not fired. In the browser console, the following two errors are recorded are:

· * Expression #{currentProgranancyOutcome} != ‘Live birth’ contains variable currentProgranancyOutcome - but this variable is not defined.*

· * Expression with id rule:xOm49QX4Nsc could not be run. Original condition was: #{currentProgranancyOutcome} != ‘Live birth’ - Evaluation ended up as:#{currentProgranancyOutcome} != ‘Live birth’ - error message:SyntaxError: Invalid or unexpected token*

The important error seems to be the second one, as in many cases this is appearing in isolation (without the first one), and is still breaking program rules. For example, I’ve loaded the Cameroon EquityTool into the Play Demo instance, and any attempt to enter new Event Capture events for this program keeps generating similar error messages in the browser console, eg:

· * Expression with id rule:gXwzJFfRn5m could not be run. Original condition was: #{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - Evaluation ended up as:#{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - error message:SyntaxError: Invalid or unexpected token*

These program rules have been extensively tested on earlier releases, and worked well – where users used to be prevented from entering invalid surveys, they can now save invalid surveys.

This is very serious, as it means that in 2.26, users can now bypass many program rules, and save incomplete or incorrect data. But I’m also aware that there have recently been build issues with 2.25 and 2.26 releases – before I log a bug on Jira, could this be related to those build issues?

Regards, Sam.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Hi Markus,

Many thanks for your fix for this issue (https://jira.dhis2.org/browse/DHIS2-1503 ). However, I think it has only partially worked.

Some of our program rules now fire correctly, while others do not, and it’s definitely still related to dashes in the variables. For example, !(d2:hasValue(‘EQT-CM11Q05’)) now works well with a dash in the variable name; but the following rule failed to fire (this is the browser console error) when it had a dash in the name:

Expression with id rule:uvfGwNfRwu2 could not be run. Original condition was:

#{EQT-CM11Q06a} + #{EQT-CM11Q06b} == 0 && d2:hasValue(‘EQT-CM11Q06a’) && d2:hasValue(‘EQT-CM11Q06b’)

  • Evaluation ended up as:

#{EQT-CM11Q06a} + #{EQT-CM11Q06b} == 0 && d2:hasValue(‘EQT-CM11Q06a’) && d2:hasValue(‘EQT-CM11Q06b’)

  • error message:SyntaxError: Invalid or unexpected token

After removing the dash from the name, this second rule also fired correctly.

I’ve spent quite a bit of time trying to find the pattern here – ie why some rules with dashes in the variable name work, and others don’t – and the only possible pattern I can see is that variables with dashes are handled properly within functions, but not when they are used directly within expressions… Could this be the issue?

Many thanks for any further help you can provide with this (persistent!) bug.

Regard, Sam.

···

From: Sam Johnson samuel.johnson@qebo.co.uk

Date: Thursday, 27 April 2017 at 23:32

To: Markus Bekken markus@dhis2.org

Cc: Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett andrea@m4mgmt.org, DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Hi Markus,

Just following up my question below – is there any chance we could get this treated as a bug, and fixed in 2.26? This issue has crippled 30-odd programs that Metrics for Management (M4M) has already published and deployed, but I’m guessing that the fix is probably just be a very simple string-handling change…

Many thanks for any help you might be able to give!

Cheers, Sam.

From: Dhis2-devs dhis2-devs-bounces+samuel.johnson=qebo.co.uk@lists.launchpad.net on behalf of Sam Johnson samuel.johnson@qebo.co.uk

Date: Friday, 21 April 2017 at 17:58

To: Markus Bekken markus@dhis2.org

Cc: Kenzo Fry kenzo.fry.consultant@gmail.com, Andrea Sprockett andrea@m4mgmt.org, DHIS2 Developers dhis2-devs@lists.launchpad.net

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing

Feedback

Hi Markus (and other devs),

Many thanks for diagnosing this so quickly. You’re spot-on, it’s the dash in the source name that’s causing the rules to fail – if I remove dashes from all source names, the rules fire without problems. In fact, this seems wider than just dashes – program rules that were failing in another instance seem to now work if I similarly remove brackets and spaces from source names.

Would it be at all possible to get this treated as a bug and fixed in 2.26, rather than treated as an improvement for a future version of DHIS2? These rules were all working perfectly when we did intensive testing on our programs in 2.25 (eg they work properly in 2.25 da6a2df / 2017-02-02 06:43), so this limitation seems to have been recently introduced (very likely by accident, since other element names allow dashes), and it’s breaking configurations that have worked well up until now.

The reason we’re so keen on this is that Metrics for Management has published XMLs for 29 EquityTool programs, and these have now been downloaded and installed in a wide range of DHIS2 instances. It will be a huge job to not only update the source names in all of these 29 programs and re-publish them, but also contact all existing users and ask them to reinstall the corrected programs. If there’s any chance we could get the program rules in 2.26 working as they did before, this would save us an enormous amount of work…

Many thanks for any help you might be able to give us with this!

Regards, Sam.

From: Markus Bekken markus@dhis2.org

Date: Wednesday, 19 April 2017 at 14:55

To: Sam Johnson samuel.johnson@qebo.co.uk

Cc: DHIS2 Developers dhis2-devs@lists.launchpad.net, Kenzo Fry kenzo.fry.consultant@gmail.com

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Hey Sam!

Thanks for reporting.

I tested WHO RMNCH and ‘Hide program stage’ on Play, and found that it the problem is in fact what is indicated by the console message in point 4 below. The source field #{currentProganancyOutcome} did not exist. Seems someone has renamed it to

#{Current Progranancy Outcome} - see below to the left:

Such a name change should ideally clean up the expressions using the source field, or at least flag the ones that has become illegal as a result of the name change. At the moment however you would have to manually update the expressions after a name change. Updated the expression now on Play, and the expression runs fine:

Expression with id rule:xOm49QX4Nsc was successfully run. Original condition was: #{current Progranancy Outcome} != ‘Live birth’ - Evaluation ended up as:’’ != ‘Live birth’ - Result of evaluation was:true

In general your rules hiding program stages should work fine, let me know if you find any other problems here.

Looking into the second part of the mail regarding Event Capture - this is a different problem: There is a dash in the source field name. Registered an improvement in Jira based on this: https://jira.dhis2.org/browse/DHIS2-1449 - but avoiding dashes is a possible workaround.

Best regards,

Markus

  1. apr. 2017 kl. 12.43 skrev Sam Johnson samuel.johnson@qebo.co.uk:

Hi Devs,

I’m finding that a number of program rules aren’t firing in 2.26 – this is happening with both Event Capture and Tracker Capture.

This includes the sample ‘Hide Program Stage’ rule which has been added to the WHO RMNCH Tracker for the 2.26 release. To reproduce in Play demo:

  1.    add a new registration to the WHO RMNCH Tracker with date 2017-01-01
    
  1.    complete the initial visit, and click ‘cancel’ to prevent creation of a second antenatal care visit
    
  1.    click ‘+’ to create a ‘Care at birth’ event on 2017-02-01
    
  1.    for WHOMCH pregnancy outcome, select ‘Stillbirth’, and complete the event.
    

The rule which should hide the postnatal program stage has not fired. In the browser console, the following two errors are recorded are:

· * Expression #{currentProgranancyOutcome} != ‘Live birth’ contains variable currentProgranancyOutcome - but this variable is not defined.*

· * Expression with id rule:xOm49QX4Nsc could not be run. Original condition was: #{currentProgranancyOutcome} != ‘Live birth’ - Evaluation ended up as:#{currentProgranancyOutcome} != ‘Live birth’ - error message:SyntaxError: Invalid or unexpected token*

The important error seems to be the second one, as in many cases this is appearing in isolation (without the first one), and is still breaking program rules. For example, I’ve loaded the Cameroon EquityTool into the Play Demo instance, and any attempt to enter new Event Capture events for this program keeps generating similar error messages in the browser console, eg:

· * Expression with id rule:gXwzJFfRn5m could not be run. Original condition was: #{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - Evaluation ended up as:#{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - error message:SyntaxError: Invalid or unexpected token*

These program rules have been extensively tested on earlier releases, and worked well – where users used to be prevented from entering invalid surveys, they can now save invalid surveys.

This is very serious, as it means that in 2.26, users can now bypass many program rules, and save incomplete or incorrect data. But I’m also aware that there have recently been build issues with 2.25 and 2.26 releases – before I log a bug on Jira, could this be related to those build issues?

Regards, Sam.


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp