the latest builds of tomcat (the servlet container mostly used with DHIS 2) has tightened up validation of characters in URLs, so that only characters defined as safe per RFC 1738 are allowed. Our apps had some cases of un-escaped use of the pipe character which was causing tomcat to occasionally return 400 bad request.
We have patched this now in 2.24, 2.25 and master.
Bottom line: If you plan to upgrade to very latest Tomcat 7, 8 or 8.5 builds on your server, make sure to upgrade to latest 2.24 or 2.25 of DHIS 2.
On Sat, Jan 7, 2017 at 6:26 PM, Lars Helge Øverland lars@dhis2.org wrote:
Hi all,
the latest builds of tomcat (the servlet container mostly used with DHIS 2) has tightened up validation of characters in URLs, so that only characters defined as safe per RFC 1738 are allowed. Our apps had some cases of un-escaped use of the pipe character which was causing tomcat to occasionally return 400 bad request.
We have patched this now in 2.24, 2.25 and master.
Bottom line: If you plan to upgrade to very latest Tomcat 7, 8 or 8.5 builds on your server, make sure to upgrade to latest 2.24 or 2.25 of DHIS 2.
Ministry of Health, Nutrition and Indigenous Medicine,
Sri Lanka
Confidentiality Notice: the information contained in this email and any attachments may be legally privileged and confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachments for any purpose, nor disclose all or any part of the contents to any other person.
I can see this is going to cause quite a bit of chaos with large country installations where they are not able to be too agile with upgrading.
Do you have more precise info on the exact tomcat version numbers? We just saw in Zim (DHIS 2.22) that the package manager automatically upgraded to 7.0.52 and they started seeing these problems. So maybe it is that version?
They will have to try and come up with a process of downgrading tomcat and holding that version via the package manager as a short term measure while they plan any dhis2 upgrade process.
So getting the exact tomcat versions where the URL checking was introduced will be helpful if you have them.
···
On 7 January 2017 at 12:56, Lars Helge Øverland lars@dhis2.org wrote:
Hi all,
the latest builds of tomcat (the servlet container mostly used with DHIS 2) has tightened up validation of characters in URLs, so that only characters defined as safe per RFC 1738 are allowed. Our apps had some cases of un-escaped use of the pipe character which was causing tomcat to occasionally return 400 bad request.
We have patched this now in 2.24, 2.25 and master.
Bottom line: If you plan to upgrade to very latest Tomcat 7, 8 or 8.5 builds on your server, make sure to upgrade to latest 2.24 or 2.25 of DHIS 2.
is known to work in this situation for me. Lars suggested this version and it worked for us.
We had the exact same thing happen on another instance, which basically “broke” dhis2-tools, so for the time being, we are using this specific version of Tomcat as a local install to work around the problem until that instance can be upgraded.
Specifically, it was this commit (thanks to BAO for finding it)
which introduced this, which seems to be Tomcat 7.0.73, so something earlier than that should work as well. I am not sure which commit this was in Tomcat 8.
I can see this is going to cause quite a bit of chaos with large country installations where they are not able to be too agile with upgrading.
Do you have more precise info on the exact tomcat version numbers? We just saw in Zim (DHIS 2.22) that the package manager automatically upgraded to 7.0.52 and they started seeing these problems. So maybe it is that version?
They will have to try and come up with a process of downgrading tomcat and holding that version via the package manager as a short term measure while they plan any dhis2 upgrade process.
So getting the exact tomcat versions where the URL checking was introduced will be helpful if you have them.
On 7 January 2017 at 12:56, Lars Helge Øverland lars@dhis2.org wrote:
Hi all,
the latest builds of tomcat (the servlet container mostly used with DHIS 2) has tightened up validation of characters in URLs, so that only characters defined as safe per RFC 1738 are allowed. Our apps had some cases of un-escaped use of the pipe character which was causing tomcat to occasionally return 400 bad request.
We have patched this now in 2.24, 2.25 and master.
Bottom line: If you plan to upgrade to very latest Tomcat 7, 8 or 8.5 builds on your server, make sure to upgrade to latest 2.24 or 2.25 of DHIS 2.
Thanks Jason. To make matters more complicated it looks like ubuntu maintains its own patch release numbering of tomcat. So for example it looks like the problem first raised in Zim after upgrading 7.0.52-1ubuntu0.7 to 7.0.52-1ubuntu0.8.
They can try to rewind that upgrade to see if good behaviour is restored.
Then I believe you can hold back further upgrades to certain packages with apt-mark hold . We’ll see.
How painful is it to patch dhis2 older versions? I was looking (without success) for relevant github commit.
is known to work in this situation for me. Lars suggested this version and it worked for us.
We had the exact same thing happen on another instance, which basically “broke” dhis2-tools, so for the time being, we are using this specific version of Tomcat as a local install to work around the problem until that instance can be upgraded.
Specifically, it was this commit (thanks to BAO for finding it)
which introduced this, which seems to be Tomcat 7.0.73, so something earlier than that should work as well. I am not sure which commit this was in Tomcat 8.
I can see this is going to cause quite a bit of chaos with large country installations where they are not able to be too agile with upgrading.
Do you have more precise info on the exact tomcat version numbers? We just saw in Zim (DHIS 2.22) that the package manager automatically upgraded to 7.0.52 and they started seeing these problems. So maybe it is that version?
They will have to try and come up with a process of downgrading tomcat and holding that version via the package manager as a short term measure while they plan any dhis2 upgrade process.
So getting the exact tomcat versions where the URL checking was introduced will be helpful if you have them.
On 7 January 2017 at 12:56, Lars Helge Øverland lars@dhis2.org wrote:
Hi all,
the latest builds of tomcat (the servlet container mostly used with DHIS 2) has tightened up validation of characters in URLs, so that only characters defined as safe per RFC 1738 are allowed. Our apps had some cases of un-escaped use of the pipe character which was causing tomcat to occasionally return 400 bad request.
We have patched this now in 2.24, 2.25 and master.
Bottom line: If you plan to upgrade to very latest Tomcat 7, 8 or 8.5 builds on your server, make sure to upgrade to latest 2.24 or 2.25 of DHIS 2.
Lars had advised me this would not be easy, as this fix would need to be made in several apps.
I did not have time to figure out exactly which Tomcat package would work, but your approach sounds reasonable to me. We took a temporary route and used one we knew would work until the upgrade to at least 2.24 is feasible.
is known to work in this situation for me. Lars suggested this version and it worked for us.
We had the exact same thing happen on another instance, which basically “broke” dhis2-tools, so for the time being, we are using this specific version of Tomcat as a local install to work around the problem until that instance can be upgraded.
Specifically, it was this commit (thanks to BAO for finding it)
which introduced this, which seems to be Tomcat 7.0.73, so something earlier than that should work as well. I am not sure which commit this was in Tomcat 8.
I can see this is going to cause quite a bit of chaos with large country installations where they are not able to be too agile with upgrading.
Do you have more precise info on the exact tomcat version numbers? We just saw in Zim (DHIS 2.22) that the package manager automatically upgraded to 7.0.52 and they started seeing these problems. So maybe it is that version?
They will have to try and come up with a process of downgrading tomcat and holding that version via the package manager as a short term measure while they plan any dhis2 upgrade process.
So getting the exact tomcat versions where the URL checking was introduced will be helpful if you have them.
On 7 January 2017 at 12:56, Lars Helge Øverland lars@dhis2.org wrote:
Hi all,
the latest builds of tomcat (the servlet container mostly used with DHIS 2) has tightened up validation of characters in URLs, so that only characters defined as safe per RFC 1738 are allowed. Our apps had some cases of un-escaped use of the pipe character which was causing tomcat to occasionally return 400 bad request.
We have patched this now in 2.24, 2.25 and master.
Bottom line: If you plan to upgrade to very latest Tomcat 7, 8 or 8.5 builds on your server, make sure to upgrade to latest 2.24 or 2.25 of DHIS 2.
It “should” work indeed. I haven’t tested out downgrading the tomcat related packages yet. It might not be so straightforward. Also of course it is a bit of a concern as all of the tomcat upgrades on a “normally” configured ubuntu system would be security upgrades. So we would be asking users to run with known vulnerabilities which I am a little uneasy about.
What we are saying effectively is that dhis2 v2.23 and earlier has a flaw which requires it to be run on a tomcat with known vulnerabilities. Effectively this translates to a vulnerability (in fact a bundle) in 2.23 for which the real remedy is to upgrade to 2.24. Downgrading tomcat is a distant second best workaround.
I still have to scratch my head a bit to figure out and test a neat/quick way to achieve this with dhis2-tools where it might be difficult to do a quick upgrade to 2.24.
Lars had advised me this would not be easy, as this fix would need to be made in several apps.
I did not have time to figure out exactly which Tomcat package would work, but your approach sounds reasonable to me. We took a temporary route and used one we knew would work until the upgrade to at least 2.24 is feasible.
Thanks Jason. To make matters more complicated it looks like ubuntu maintains its own patch release numbering of tomcat. So for example it looks like the problem first raised in Zim after upgrading 7.0.52-1ubuntu0.7 to 7.0.52-1ubuntu0.8.
They can try to rewind that upgrade to see if good behaviour is restored.
Then I believe you can hold back further upgrades to certain packages with apt-mark hold . We’ll see.
How painful is it to patch dhis2 older versions? I was looking (without success) for relevant github commit.
is known to work in this situation for me. Lars suggested this version and it worked for us.
We had the exact same thing happen on another instance, which basically “broke” dhis2-tools, so for the time being, we are using this specific version of Tomcat as a local install to work around the problem until that instance can be upgraded.
Specifically, it was this commit (thanks to BAO for finding it)
which introduced this, which seems to be Tomcat 7.0.73, so something earlier than that should work as well. I am not sure which commit this was in Tomcat 8.
I can see this is going to cause quite a bit of chaos with large country installations where they are not able to be too agile with upgrading.
Do you have more precise info on the exact tomcat version numbers? We just saw in Zim (DHIS 2.22) that the package manager automatically upgraded to 7.0.52 and they started seeing these problems. So maybe it is that version?
They will have to try and come up with a process of downgrading tomcat and holding that version via the package manager as a short term measure while they plan any dhis2 upgrade process.
So getting the exact tomcat versions where the URL checking was introduced will be helpful if you have them.
On 7 January 2017 at 12:56, Lars Helge Øverland lars@dhis2.org wrote:
Hi all,
the latest builds of tomcat (the servlet container mostly used with DHIS 2) has tightened up validation of characters in URLs, so that only characters defined as safe per RFC 1738 are allowed. Our apps had some cases of un-escaped use of the pipe character which was causing tomcat to occasionally return 400 bad request.
We have patched this now in 2.24, 2.25 and master.
Bottom line: If you plan to upgrade to very latest Tomcat 7, 8 or 8.5 builds on your server, make sure to upgrade to latest 2.24 or 2.25 of DHIS 2.