we have recently detected a security exploit on a couple of servers running dhis. The exploit seems to result in shell access with permissions of the user which is running tomcat.
Symptoms of the exploit are presence of:
a file /tmp/fake.cfg.
various files with numeric-only names in /tmp directory.
massive outgoing network traffic (> 200 Gb per day).
The files will be owned by the user running tomcat. The outgoing network traffic is likely to be part of denial-of-service attacks against other servers.
Cause of the exploit is likely to be one or more weaknesses in Struts 2, which is a web framework used in dhis. These weaknesses have been fixed in Struts version 2.3.15.1. We have upgraded dhis version 2.12, 2.13 and snapshot/trunk with the new version. You can download the new WAR files from dhis2.org/downloads as usual.
To remove the exploit you should do the following:
stop tomcat
upgrade your dhis version (to 2.12 or 2.13)
remove all of the above mentioned files from /tmp (all owned by tomcat user).
kill all processes owned by the tomcat user, or simply reboot the server.
delete all files and folders under /work/Catalina (not confirmed but to be on the safe side).
If you have been running tomcat as root (sudo) then a full operating system re-install is recommended. There is no way to completely verify what an exploit can do with full permissions. Running tomcat as root is strictly discouraged in any case.
Summary
In any case you should upgrade your dhis version, whether you see the symptoms or not.
If you see the symptoms but have been running dhis with regular, non-root privileges, you will be fine by following the removal steps.
If you see the symptoms and have been running dhis with root privileges, you should do a clean server installation.
Its great news for DHIS2 regular users and system administrators, that one of big security vulnerability has been found/detected and remedial action can be taken to resolve the problem. But i am not that much sure that most of the implementers would like to upgrade live application on their server only for this problem, who are using DHIS 2.12 build as an assumption as a very good stable release in series so far since they are using DHIS 2. Its good that application should be upgraded DHIS 2.12 to DHIS 2.13 on live servers, but at the same time scrum masters should also release some stable patches releases as well for DHIS 2.12 release for fixing above stated like problems, that will prevent unnecessary wastage of time and money in system application version up-gradation only for fixing miner problem. Because in normal and general software implementation practices, we use to release patches to fix these types of issues, at the same time implementers expectations are the same.
Regards
Brajesh Murari
···
Life Is A Collection of Poems.
On Wednesday, 25 December 2013 6:54 PM, Lars Helge Øverland larshelge@gmail.com wrote:
Hi,
we have recently detected a security exploit on a couple of servers running dhis. The exploit seems to result in shell access with permissions of the user which is running tomcat.
Symptoms of the exploit are presence of:
a file /tmp/fake.cfg.
various files with numeric-only names in /tmp directory.
massive outgoing network traffic (> 200 Gb per day).
The files will be owned by the user running tomcat. The outgoing network traffic is likely to be part of denial-of-service attacks against other servers.
Cause of the exploit is likely to be one or more weaknesses in Struts 2, which is a web framework used in dhis. These weaknesses have been fixed in Struts version 2.3.15.1. We have upgraded dhis version 2.12, 2.13 and snapshot/trunk with the new version. You can download the new WAR files from dhis2.org/downloads as usual.
To remove the exploit you should do the following:
stop tomcat
upgrade your dhis version (to 2.12 or 2.13)
remove all of the above mentioned files from /tmp (all owned by tomcat user).
kill all processes owned by the tomcat user, or simply reboot the server.
delete all files and folders under /work/Catalina (not confirmed but to be on the safe side).
If you have been running tomcat as root (sudo) then a full operating system re-install is recommended. There is no way to completely verify what an exploit can do with full permissions. Running tomcat as root is strictly discouraged in any case.
Summary
In any case you should upgrade your dhis version, whether you see the symptoms or not.
If you see the symptoms but have been running dhis with regular, non-root privileges, you will be fine by following the removal steps.
If you see the symptoms and have been running dhis with root privileges, you should do a clean server installation.
Lars’s mail could have provided a bit more explicit advice I think, but as you can see in Lars’s email, it is stated
“We have upgraded dhis version 2.12, 2.13 and snapshot/trunk with the new version.”
I think the clear message is that anyone using DHIS2 should upgrade to the latest versions 2.12 or 2.13. Older versions of DHIS2 will be subject to this exploit. It is also described in a bit more detail here.
The names do not have to be numerical only either. In order to be sure that you are not suffering from this, you can invoke
“ps -ef | grep tocmat” to see all the processes which are running with the tomcat user. If you are using a different username other than “tomcat6” or “tomcat7” you should replace the username with the actual name. Alternatively, you can do “ps -ef | grep tmp” to try and see if there is anything running which should not be running from the “/tmp” directory. You can the easily kill the process, but it will spawn again by itself. After the upgrade to the latest version however, it should not reappear.
If you need a patch for your own branch, as Lars points out, it has been committed to trunk here.
Its great news for DHIS2 regular users and system administrators, that one of big security vulnerability has been found/detected and remedial action can be taken to resolve the problem. But i am not that much sure that most of the implementers would like to upgrade live application on their server only for this problem, who are using DHIS 2.12 build as an assumption as a very good stable release in series so far since they are using DHIS 2. Its good that application should be upgraded DHIS 2.12 to DHIS 2.13 on live servers, but at the same time scrum masters should also release some stable patches releases as well for DHIS 2.12 release for fixing above stated like problems, that will prevent unnecessary wastage of time and money in system application version up-gradation only for fixing miner problem. Because in normal and general software implementation practices, we use to release patches to fix these types of issues, at the same time implementers expectations are the same.
Regards
Brajesh Murari
Life Is A Collection of Poems.
On Wednesday, 25 December 2013 6:54 PM, Lars Helge Øverland larshelge@gmail.com wrote:
Hi,
we have recently detected a security exploit on a couple of servers running dhis. The exploit seems to result in shell access with permissions of the user which is running tomcat.
Symptoms of the exploit are presence of:
a file /tmp/fake.cfg.
various files with numeric-only names in /tmp directory.
massive outgoing network traffic (> 200 Gb per day).
The files will be owned by the user running tomcat. The outgoing network traffic is likely to be part of denial-of-service attacks against other servers.
Cause of the exploit is likely to be one or more weaknesses in Struts 2, which is a web framework used in dhis. These weaknesses have been fixed in Struts version 2.3.15.1. We have upgraded dhis version 2.12, 2.13 and snapshot/trunk with the new version. You can download the new WAR files from dhis2.org/downloads as usual.
To remove the exploit you should do the following:
stop tomcat
upgrade your dhis version (to 2.12 or 2.13)
remove all of the above mentioned files from /tmp (all owned by tomcat user).
kill all processes owned by the tomcat user, or simply reboot the server.
delete all files and folders under /work/Catalina (not confirmed but to be on the safe side).
If you have been running tomcat as root (sudo) then a full operating system re-install is recommended. There is no way to completely verify what an exploit can do with full permissions. Running tomcat as root is strictly discouraged in any case.
Summary
In any case you should upgrade your dhis version, whether you see the symptoms or not.
If you see the symptoms but have been running dhis with regular, non-root privileges, you will be fine by following the removal steps.
If you see the symptoms and have been running dhis with root privileges, you should do a clean server installation.
Lars’s mail could have provided a bit more explicit advice I think, but as you can see in Lars’s email, it is stated
“We have upgraded dhis version 2.12, 2.13 and snapshot/trunk with the new version.”
I think the clear message is that anyone using DHIS2 should upgrade to the latest versions 2.12 or 2.13. Older versions of DHIS2 will be subject to this exploit. It is also described in a bit more detail here.
The names do not have to be numerical only either. In order to be sure that you are not suffering from this, you can invoke
“ps -ef | grep tocmat” to see all the processes which are running with the tomcat user. If you are using a different username other than “tomcat6” or “tomcat7” you should replace the username with the actual name. Alternatively, you can do “ps -ef | grep tmp” to try and see if there is anything running which should not be running from the “/tmp” directory. You can the easily kill the process, but it will spawn again by itself. After the upgrade to the latest version however, it should not reappear.
If you need a patch for your own branch, as Lars points out, it has been committed to trunk here.
Its great news for DHIS2 regular users and system administrators, that one of big security vulnerability has been found/detected and remedial action can be taken to resolve the problem. But i am not that much sure that most of the implementers would like to upgrade live application on their server only for this problem, who are using DHIS 2.12 build as an assumption as a very good stable release in series so far since they are using DHIS 2. Its good that application should be upgraded DHIS 2.12 to DHIS 2.13 on live servers, but at the same time scrum masters should also release some stable patches releases as well for DHIS 2.12 release for fixing above stated like problems, that will prevent unnecessary wastage of time and money in system application version up-gradation only for fixing miner problem. Because in normal and general software implementation practices, we use to release patches to fix these types of issues, at the same time implementers expectations are the same.
Regards
Brajesh Murari
Life Is A Collection of Poems.
On Wednesday, 25 December 2013 6:54 PM, Lars Helge Øverland larshelge@gmail.com wrote:
Hi,
we have recently detected a security exploit on a couple of servers running dhis. The exploit seems to result in shell access with permissions of the user which is running tomcat.
Symptoms of the exploit are presence of:
a file /tmp/fake.cfg.
various files with numeric-only names in /tmp directory.
massive outgoing network traffic (> 200 Gb per day).
The files will be owned by the user running tomcat. The outgoing network traffic is likely to be part of denial-of-service attacks against other servers.
Cause of the exploit is likely to be one or more weaknesses in Struts 2, which is a web framework used in dhis. These weaknesses have been fixed in Struts version 2.3.15.1. We have upgraded dhis version 2.12, 2.13 and snapshot/trunk with the new version. You can download the new WAR files from dhis2.org/downloads as usual.
To remove the exploit you should do the following:
stop tomcat
upgrade your dhis version (to 2.12 or 2.13)
remove all of the above mentioned files from /tmp (all owned by tomcat user).
kill all processes owned by the tomcat user, or simply reboot the server.
delete all files and folders under /work/Catalina (not confirmed but to be on the safe side).
If you have been running tomcat as root (sudo) then a full operating system re-install is recommended. There is no way to completely verify what an exploit can do with full permissions. Running tomcat as root is strictly discouraged in any case.
Summary
In any case you should upgrade your dhis version, whether you see the symptoms or not.
If you see the symptoms but have been running dhis with regular, non-root privileges, you will be fine by following the removal steps.
If you see the symptoms and have been running dhis with root privileges, you should do a clean server installation.