Bangladesh's main DHIS2 installation hacked and solved

Support - Assistance technique
1

Dear experts

Our main DHIS2 implementation (mishealth) for the health sector was hacked yesterday evening, around 4:30 PM local time. After login by any user it showing the attached message. We immediately stop the tomact7 service and check the database. We find the database is intact.

After investigation I find that the hacker inserted three files to do this.

First file “index.html” contain an alert “alert(“Admin, You Are Hacked by Malaysia Hacker!”)” and a body text

Hacked by BadCat

. Which was placed in the application folder /tomcat7/webapps/mishealth/.

Second files “index.html” contain another script which redirects to “pastebin.com/raw.php?i=LZEdbBz6” was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

Third file “guige.jsp” is contain a script was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

For our server, it seems that only first file is executing after login. I find few more suspicious files which I am investigating and will share with the experts in next few days.

I configured the server with only external open port is 8080. Other two ports (SSH and WEBMIN) are open for internal IP only. External access is possible only through VPN client. According to the firewall maintaining vendor, that hacker might access through 8080. How we prevent and secure that?

I configure the database in other server and that server is only accessible through one private IP block. The tomcat server, the backup servers and our administrator/development team are in that block.

Now please suggest how can we secure our servers more.

Regards

Muhammad Abdul Hannan Khan

hacked_screenshot.docx (153 KB)

···

Senior Technical Advisor - HIS

Priority Area Health

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH

House10/A, Road 90, Gulshan 2, Dhaka 1212, Bangladesh

T +880-2- 8816459, 8816412 ext 118

M+88 01819 239 241

M+88 01534 312 066

F +88 02 8813 875

E hannan.khan@giz.de

S hannan.khan.dhaka

B hannan-tech.blogspot.com

2

Hannan, which build of DHIS2 ? Which Java version? Ubuntu?

···

Sent from my mobile

On Feb 6, 2014 6:29 AM, “Hannan Khan” hannank@gmail.com wrote:

Dear experts

Our main DHIS2 implementation (mishealth) for the health sector was hacked yesterday evening, around 4:30 PM local time. After login by any user it showing the attached message. We immediately stop the tomact7 service and check the database. We find the database is intact.

After investigation I find that the hacker inserted three files to do this.

First file “index.html” contain an alert “alert(“Admin, You Are Hacked by Malaysia Hacker!”)” and a body text

Hacked by BadCat

. Which was placed in the application folder /tomcat7/webapps/mishealth/.

Second files “index.html” contain another script which redirects to “pastebin.com/raw.php?i=LZEdbBz6” was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

Third file “guige.jsp” is contain a script was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

For our server, it seems that only first file is executing after login. I find few more suspicious files which I am investigating and will share with the experts in next few days.

I configured the server with only external open port is 8080. Other two ports (SSH and WEBMIN) are open for internal IP only. External access is possible only through VPN client. According to the firewall maintaining vendor, that hacker might access through 8080. How we prevent and secure that?

I configure the database in other server and that server is only accessible through one private IP block. The tomcat server, the backup servers and our administrator/development team are in that block.

Now please suggest how can we secure our servers more.

Regards

Muhammad Abdul Hannan Khan

Senior Technical Advisor - HIS

Priority Area Health

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH

House10/A, Road 90, Gulshan 2, Dhaka 1212, Bangladesh

T +880-2- 8816459, 8816412 ext 118

M+88 01819 239 241

M+88 01534 312 066

F +88 02 8813 875

E hannan.khan@giz.de

S hannan.khan.dhaka

B hannan-tech.blogspot.com

3

Also make sure that your tomcat is up to date… there exists several vulnerabilities in older versions

(not sure how you installed it, but if you are using a linux distribution, its wise to install it through the package manager)

···

–
Morten

On Thu, Feb 6, 2014 at 1:00 PM, Knut Staring knutst@gmail.com wrote:

Hannan, which build of DHIS2 ? Which Java version? Ubuntu?

Sent from my mobile

On Feb 6, 2014 6:29 AM, “Hannan Khan” hannank@gmail.com wrote:

Dear experts

Our main DHIS2 implementation (mishealth) for the health sector was hacked yesterday evening, around 4:30 PM local time. After login by any user it showing the attached message. We immediately stop the tomact7 service and check the database. We find the database is intact.

After investigation I find that the hacker inserted three files to do this.

First file “index.html” contain an alert “alert(“Admin, You Are Hacked by Malaysia Hacker!”)” and a body text

Hacked by BadCat

. Which was placed in the application folder /tomcat7/webapps/mishealth/.

Second files “index.html” contain another script which redirects to “pastebin.com/raw.php?i=LZEdbBz6” was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

Third file “guige.jsp” is contain a script was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

For our server, it seems that only first file is executing after login. I find few more suspicious files which I am investigating and will share with the experts in next few days.

I configured the server with only external open port is 8080. Other two ports (SSH and WEBMIN) are open for internal IP only. External access is possible only through VPN client. According to the firewall maintaining vendor, that hacker might access through 8080. How we prevent and secure that?

I configure the database in other server and that server is only accessible through one private IP block. The tomcat server, the backup servers and our administrator/development team are in that block.

Now please suggest how can we secure our servers more.

Regards

Muhammad Abdul Hannan Khan

Senior Technical Advisor - HIS

Priority Area Health

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH

House10/A, Road 90, Gulshan 2, Dhaka 1212, Bangladesh

T +880-2- 8816459, 8816412 ext 118

M+88 01819 239 241

M+88 01534 312 066

F +88 02 8813 875

E hannan.khan@giz.de

S hannan.khan.dhaka

B hannan-tech.blogspot.com

4

For this implementation we are using DHIS2 version 2.12 build 11312. This version will be upgraded to version 13 this evening.

Java version 1.7.0_25 OpenJDK 64 bit server

Ubuntu 12.04.2.

···

On Thu, Feb 6, 2014 at 12:00 PM, Knut Staring knutst@gmail.com wrote:

Hannan, which build of DHIS2 ? Which Java version? Ubuntu?

Sent from my mobile

On Feb 6, 2014 6:29 AM, “Hannan Khan” hannank@gmail.com wrote:

Dear experts

Our main DHIS2 implementation (mishealth) for the health sector was hacked yesterday evening, around 4:30 PM local time. After login by any user it showing the attached message. We immediately stop the tomact7 service and check the database. We find the database is intact.

After investigation I find that the hacker inserted three files to do this.

First file “index.html” contain an alert “alert(“Admin, You Are Hacked by Malaysia Hacker!”)” and a body text

Hacked by BadCat

. Which was placed in the application folder /tomcat7/webapps/mishealth/.

Second files “index.html” contain another script which redirects to “pastebin.com/raw.php?i=LZEdbBz6” was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

Third file “guige.jsp” is contain a script was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

For our server, it seems that only first file is executing after login. I find few more suspicious files which I am investigating and will share with the experts in next few days.

I configured the server with only external open port is 8080. Other two ports (SSH and WEBMIN) are open for internal IP only. External access is possible only through VPN client. According to the firewall maintaining vendor, that hacker might access through 8080. How we prevent and secure that?

I configure the database in other server and that server is only accessible through one private IP block. The tomcat server, the backup servers and our administrator/development team are in that block.

Now please suggest how can we secure our servers more.

Regards

Muhammad Abdul Hannan Khan

Senior Technical Advisor - HIS

Priority Area Health

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH

House10/A, Road 90, Gulshan 2, Dhaka 1212, Bangladesh

T +880-2- 8816459, 8816412 ext 118

M+88 01819 239 241

M+88 01534 312 066

F +88 02 8813 875

E hannan.khan@giz.de

S hannan.khan.dhaka

B hannan-tech.blogspot.com

5

Yes Morten, I installed through the package manager.

The tomcat version is Apache Tomcat/7.0.26.

Regards

Hannan

···

On Thu, Feb 6, 2014 at 12:07 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Also make sure that your tomcat is up to date… there exists several vulnerabilities in older versions

(not sure how you installed it, but if you are using a linux distribution, its wise to install it through the package manager)

–
Morten

On Thu, Feb 6, 2014 at 1:00 PM, Knut Staring knutst@gmail.com wrote:

Hannan, which build of DHIS2 ? Which Java version? Ubuntu?

Sent from my mobile

On Feb 6, 2014 6:29 AM, “Hannan Khan” hannank@gmail.com wrote:

Dear experts

Our main DHIS2 implementation (mishealth) for the health sector was hacked yesterday evening, around 4:30 PM local time. After login by any user it showing the attached message. We immediately stop the tomact7 service and check the database. We find the database is intact.

After investigation I find that the hacker inserted three files to do this.

First file “index.html” contain an alert “alert(“Admin, You Are Hacked by Malaysia Hacker!”)” and a body text

Hacked by BadCat

. Which was placed in the application folder /tomcat7/webapps/mishealth/.

Second files “index.html” contain another script which redirects to “pastebin.com/raw.php?i=LZEdbBz6” was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

Third file “guige.jsp” is contain a script was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

For our server, it seems that only first file is executing after login. I find few more suspicious files which I am investigating and will share with the experts in next few days.

I configured the server with only external open port is 8080. Other two ports (SSH and WEBMIN) are open for internal IP only. External access is possible only through VPN client. According to the firewall maintaining vendor, that hacker might access through 8080. How we prevent and secure that?

I configure the database in other server and that server is only accessible through one private IP block. The tomcat server, the backup servers and our administrator/development team are in that block.

Now please suggest how can we secure our servers more.

Regards

Muhammad Abdul Hannan Khan

Senior Technical Advisor - HIS

Priority Area Health

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH

House10/A, Road 90, Gulshan 2, Dhaka 1212, Bangladesh

T +880-2- 8816459, 8816412 ext 118

M+88 01819 239 241

M+88 01534 312 066

F +88 02 8813 875

E hannan.khan@giz.de

S hannan.khan.dhaka

B hannan-tech.blogspot.com

6

Hi Hannan,
I had several servers (4 to be exact) which were compromised due to a vulnerability in Struts. Lars sent out an email a few weeks ago, that informed everyone they needed to upgrade immediately. I know of other server which have also been compromised. One was running Tomcat as root (an exceptionally bad idea). Because of the compromise, a full reinstallation of the server software would be required.

In your case, it does seem to be a bit more serious, and not consistent with the previous compromises I have seen. These compromises were limited to the machine sending out a huge amount of traffic, but otherwise, there did not “seem” to be any further issues.

A few tips, you may want to consider

  1. A complete reinstall of the system might be in order, given the extent of the attack.

  2. Be sure that the Tomcat process is not running as root, and that the user which can execute Tomcat cannot login to the system directly (i.e. has their shell set to /bin/false)

  3. Close port 8080 and remove the Tomcat manager. Instead, only have port 80/443 on the machine open. Additionally, do not run SSH on port 22, and be sure that you can only login to the server with a key, which is protected itself by a strong password.

  4. Consider attempting to look for vulnerabilities your self, with tools such as Nessus and Nmap

  5. Ensure that you are running a firewall on the server itself, i.e. do not trust your upstream providers firewall.

  6. Ensure that all Tomcat installs, Java,DHIS2 and the system software itself is fully up to date

  7. Consider running an IDS such as OSSEC on your machine to look for unauthorized intrusions.

  8. Use tools such as monit to monitor for spurious processes or suspicious file activity.

Hope this helps.

Best regards,

Jason

···

On Thu, Feb 6, 2014 at 8:36 AM, Hannan Khan hannank@gmail.com wrote:

Yes Morten, I installed through the package manager.

The tomcat version is Apache Tomcat/7.0.26.

Regards

Hannan

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Feb 6, 2014 at 12:07 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Also make sure that your tomcat is up to date… there exists several vulnerabilities in older versions

(not sure how you installed it, but if you are using a linux distribution, its wise to install it through the package manager)

–
Morten

On Thu, Feb 6, 2014 at 1:00 PM, Knut Staring knutst@gmail.com wrote:

Hannan, which build of DHIS2 ? Which Java version? Ubuntu?

Sent from my mobile

On Feb 6, 2014 6:29 AM, “Hannan Khan” hannank@gmail.com wrote:

Dear experts

Our main DHIS2 implementation (mishealth) for the health sector was hacked yesterday evening, around 4:30 PM local time. After login by any user it showing the attached message. We immediately stop the tomact7 service and check the database. We find the database is intact.

After investigation I find that the hacker inserted three files to do this.

First file “index.html” contain an alert “alert(“Admin, You Are Hacked by Malaysia Hacker!”)” and a body text

Hacked by BadCat

. Which was placed in the application folder /tomcat7/webapps/mishealth/.

Second files “index.html” contain another script which redirects to “pastebin.com/raw.php?i=LZEdbBz6” was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

Third file “guige.jsp” is contain a script was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

For our server, it seems that only first file is executing after login. I find few more suspicious files which I am investigating and will share with the experts in next few days.

I configured the server with only external open port is 8080. Other two ports (SSH and WEBMIN) are open for internal IP only. External access is possible only through VPN client. According to the firewall maintaining vendor, that hacker might access through 8080. How we prevent and secure that?

I configure the database in other server and that server is only accessible through one private IP block. The tomcat server, the backup servers and our administrator/development team are in that block.

Now please suggest how can we secure our servers more.

Regards

Muhammad Abdul Hannan Khan

Senior Technical Advisor - HIS

Priority Area Health

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH

House10/A, Road 90, Gulshan 2, Dhaka 1212, Bangladesh

T +880-2- 8816459, 8816412 ext 118

M+88 01819 239 241

M+88 01534 312 066

F +88 02 8813 875

E hannan.khan@giz.de

S hannan.khan.dhaka

B hannan-tech.blogspot.com

7

Hi Hannan,

I think this attack might also be related to the Struts exploit. We did see random jsp files being uploaded at one occasion.

The fix for the Struts exploit was done in 2.12 at revision 11341, so it means that you must upgrade your DHIS version (from 11312) in order to get protection.

regards,

Lars

···

On Thu, Feb 6, 2014 at 9:18 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Hannan,
I had several servers (4 to be exact) which were compromised due to a vulnerability in Struts. Lars sent out an email a few weeks ago, that informed everyone they needed to upgrade immediately. I know of other server which have also been compromised. One was running Tomcat as root (an exceptionally bad idea). Because of the compromise, a full reinstallation of the server software would be required.

In your case, it does seem to be a bit more serious, and not consistent with the previous compromises I have seen. These compromises were limited to the machine sending out a huge amount of traffic, but otherwise, there did not “seem” to be any further issues.

A few tips, you may want to consider

  1. A complete reinstall of the system might be in order, given the extent of the attack.
  1. Be sure that the Tomcat process is not running as root, and that the user which can execute Tomcat cannot login to the system directly (i.e. has their shell set to /bin/false)
  1. Close port 8080 and remove the Tomcat manager. Instead, only have port 80/443 on the machine open. Additionally, do not run SSH on port 22, and be sure that you can only login to the server with a key, which is protected itself by a strong password.
  1. Consider attempting to look for vulnerabilities your self, with tools such as Nessus and Nmap
  1. Ensure that you are running a firewall on the server itself, i.e. do not trust your upstream providers firewall.
  1. Ensure that all Tomcat installs, Java,DHIS2 and the system software itself is fully up to date
  1. Consider running an IDS such as OSSEC on your machine to look for unauthorized intrusions.
  1. Use tools such as monit to monitor for spurious processes or suspicious file activity.

Hope this helps.

Best regards,

Jason

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Feb 6, 2014 at 8:36 AM, Hannan Khan hannank@gmail.com wrote:

Yes Morten, I installed through the package manager.

The tomcat version is Apache Tomcat/7.0.26.

Regards

Hannan

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Feb 6, 2014 at 12:07 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Also make sure that your tomcat is up to date… there exists several vulnerabilities in older versions

(not sure how you installed it, but if you are using a linux distribution, its wise to install it through the package manager)

–
Morten

On Thu, Feb 6, 2014 at 1:00 PM, Knut Staring knutst@gmail.com wrote:

Hannan, which build of DHIS2 ? Which Java version? Ubuntu?

Sent from my mobile

On Feb 6, 2014 6:29 AM, “Hannan Khan” hannank@gmail.com wrote:

Dear experts

Our main DHIS2 implementation (mishealth) for the health sector was hacked yesterday evening, around 4:30 PM local time. After login by any user it showing the attached message. We immediately stop the tomact7 service and check the database. We find the database is intact.

After investigation I find that the hacker inserted three files to do this.

First file “index.html” contain an alert “alert(“Admin, You Are Hacked by Malaysia Hacker!”)” and a body text

Hacked by BadCat

. Which was placed in the application folder /tomcat7/webapps/mishealth/.

Second files “index.html” contain another script which redirects to “pastebin.com/raw.php?i=LZEdbBz6” was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

Third file “guige.jsp” is contain a script was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

For our server, it seems that only first file is executing after login. I find few more suspicious files which I am investigating and will share with the experts in next few days.

I configured the server with only external open port is 8080. Other two ports (SSH and WEBMIN) are open for internal IP only. External access is possible only through VPN client. According to the firewall maintaining vendor, that hacker might access through 8080. How we prevent and secure that?

I configure the database in other server and that server is only accessible through one private IP block. The tomcat server, the backup servers and our administrator/development team are in that block.

Now please suggest how can we secure our servers more.

Regards

Muhammad Abdul Hannan Khan

Senior Technical Advisor - HIS

Priority Area Health

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH

House10/A, Road 90, Gulshan 2, Dhaka 1212, Bangladesh

T +880-2- 8816459, 8816412 ext 118

M+88 01819 239 241

M+88 01534 312 066

F +88 02 8813 875

E hannan.khan@giz.de

S hannan.khan.dhaka

B hannan-tech.blogspot.com

8

Thanks Lars.

Today we update DHIS2 to version 13 build 12864.

Regards

Hannan

···

On Thu, Feb 6, 2014 at 5:50 PM, Lars Helge Øverland larshelge@gmail.com wrote:

Hi Hannan,

I think this attack might also be related to the Struts exploit. We did see random jsp files being uploaded at one occasion.

The fix for the Struts exploit was done in 2.12 at revision 11341, so it means that you must upgrade your DHIS version (from 11312) in order to get protection.

regards,

Lars

On Thu, Feb 6, 2014 at 9:18 AM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Hannan,
I had several servers (4 to be exact) which were compromised due to a vulnerability in Struts. Lars sent out an email a few weeks ago, that informed everyone they needed to upgrade immediately. I know of other server which have also been compromised. One was running Tomcat as root (an exceptionally bad idea). Because of the compromise, a full reinstallation of the server software would be required.

In your case, it does seem to be a bit more serious, and not consistent with the previous compromises I have seen. These compromises were limited to the machine sending out a huge amount of traffic, but otherwise, there did not “seem” to be any further issues.

A few tips, you may want to consider

  1. A complete reinstall of the system might be in order, given the extent of the attack.
  1. Be sure that the Tomcat process is not running as root, and that the user which can execute Tomcat cannot login to the system directly (i.e. has their shell set to /bin/false)
  1. Close port 8080 and remove the Tomcat manager. Instead, only have port 80/443 on the machine open. Additionally, do not run SSH on port 22, and be sure that you can only login to the server with a key, which is protected itself by a strong password.
  1. Consider attempting to look for vulnerabilities your self, with tools such as Nessus and Nmap
  1. Ensure that you are running a firewall on the server itself, i.e. do not trust your upstream providers firewall.
  1. Ensure that all Tomcat installs, Java,DHIS2 and the system software itself is fully up to date
  1. Consider running an IDS such as OSSEC on your machine to look for unauthorized intrusions.
  1. Use tools such as monit to monitor for spurious processes or suspicious file activity.

Hope this helps.

Best regards,

Jason

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Feb 6, 2014 at 8:36 AM, Hannan Khan hannank@gmail.com wrote:

Yes Morten, I installed through the package manager.

The tomcat version is Apache Tomcat/7.0.26.

Regards

Hannan

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Feb 6, 2014 at 12:07 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Also make sure that your tomcat is up to date… there exists several vulnerabilities in older versions

(not sure how you installed it, but if you are using a linux distribution, its wise to install it through the package manager)

–
Morten

On Thu, Feb 6, 2014 at 1:00 PM, Knut Staring knutst@gmail.com wrote:

Hannan, which build of DHIS2 ? Which Java version? Ubuntu?

Sent from my mobile

On Feb 6, 2014 6:29 AM, “Hannan Khan” hannank@gmail.com wrote:

Dear experts

Our main DHIS2 implementation (mishealth) for the health sector was hacked yesterday evening, around 4:30 PM local time. After login by any user it showing the attached message. We immediately stop the tomact7 service and check the database. We find the database is intact.

After investigation I find that the hacker inserted three files to do this.

First file “index.html” contain an alert “alert(“Admin, You Are Hacked by Malaysia Hacker!”)” and a body text

Hacked by BadCat

. Which was placed in the application folder /tomcat7/webapps/mishealth/.

Second files “index.html” contain another script which redirects to “pastebin.com/raw.php?i=LZEdbBz6” was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

Third file “guige.jsp” is contain a script was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

For our server, it seems that only first file is executing after login. I find few more suspicious files which I am investigating and will share with the experts in next few days.

I configured the server with only external open port is 8080. Other two ports (SSH and WEBMIN) are open for internal IP only. External access is possible only through VPN client. According to the firewall maintaining vendor, that hacker might access through 8080. How we prevent and secure that?

I configure the database in other server and that server is only accessible through one private IP block. The tomcat server, the backup servers and our administrator/development team are in that block.

Now please suggest how can we secure our servers more.

Regards

Muhammad Abdul Hannan Khan

Senior Technical Advisor - HIS

Priority Area Health

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH

House10/A, Road 90, Gulshan 2, Dhaka 1212, Bangladesh

T +880-2- 8816459, 8816412 ext 118

M+88 01819 239 241

M+88 01534 312 066

F +88 02 8813 875

E hannan.khan@giz.de

S hannan.khan.dhaka

B hannan-tech.blogspot.com

9

Thanks Jason for your comprehensive advice.

I tried to identify problem roots and I believe I find those files. And there is no problem so far.

From the beginning I am running tomcat service as user who cannot login to the system.

Point 2 and 3 I have to do. But earlier our another serer running on port 80 severely damaged by hacker attack (web server). I will be keep in-touch on this.

Any firewall you suggests? Also consider we have very narrow bandwidth; only 10 Mbps for 9 dhis2 systems with near about 12000 users average 300 concurrent user in top three systems;

Updates we are run weekly basis.

Point 6 and 7 I will do. How that will effect the system performance?

Regards

Hannan

···

On Thu, Feb 6, 2014 at 1:18 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Hannan,
I had several servers (4 to be exact) which were compromised due to a vulnerability in Struts. Lars sent out an email a few weeks ago, that informed everyone they needed to upgrade immediately. I know of other server which have also been compromised. One was running Tomcat as root (an exceptionally bad idea). Because of the compromise, a full reinstallation of the server software would be required.

In your case, it does seem to be a bit more serious, and not consistent with the previous compromises I have seen. These compromises were limited to the machine sending out a huge amount of traffic, but otherwise, there did not “seem” to be any further issues.

A few tips, you may want to consider

  1. A complete reinstall of the system might be in order, given the extent of the attack.
  1. Be sure that the Tomcat process is not running as root, and that the user which can execute Tomcat cannot login to the system directly (i.e. has their shell set to /bin/false)
  1. Close port 8080 and remove the Tomcat manager. Instead, only have port 80/443 on the machine open. Additionally, do not run SSH on port 22, and be sure that you can only login to the server with a key, which is protected itself by a strong password.
  1. Consider attempting to look for vulnerabilities your self, with tools such as Nessus and Nmap
  1. Ensure that you are running a firewall on the server itself, i.e. do not trust your upstream providers firewall.
  1. Ensure that all Tomcat installs, Java,DHIS2 and the system software itself is fully up to date
  1. Consider running an IDS such as OSSEC on your machine to look for unauthorized intrusions.
  1. Use tools such as monit to monitor for spurious processes or suspicious file activity.

Hope this helps.

Best regards,

Jason

On Thu, Feb 6, 2014 at 8:36 AM, Hannan Khan hannank@gmail.com wrote:

Yes Morten, I installed through the package manager.

The tomcat version is Apache Tomcat/7.0.26.

Regards

Hannan

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Feb 6, 2014 at 12:07 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Also make sure that your tomcat is up to date… there exists several vulnerabilities in older versions

(not sure how you installed it, but if you are using a linux distribution, its wise to install it through the package manager)

–
Morten

On Thu, Feb 6, 2014 at 1:00 PM, Knut Staring knutst@gmail.com wrote:

Hannan, which build of DHIS2 ? Which Java version? Ubuntu?

Sent from my mobile

On Feb 6, 2014 6:29 AM, “Hannan Khan” hannank@gmail.com wrote:

Dear experts

Our main DHIS2 implementation (mishealth) for the health sector was hacked yesterday evening, around 4:30 PM local time. After login by any user it showing the attached message. We immediately stop the tomact7 service and check the database. We find the database is intact.

After investigation I find that the hacker inserted three files to do this.

First file “index.html” contain an alert “alert(“Admin, You Are Hacked by Malaysia Hacker!”)” and a body text

Hacked by BadCat

. Which was placed in the application folder /tomcat7/webapps/mishealth/.

Second files “index.html” contain another script which redirects to “pastebin.com/raw.php?i=LZEdbBz6” was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

Third file “guige.jsp” is contain a script was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

For our server, it seems that only first file is executing after login. I find few more suspicious files which I am investigating and will share with the experts in next few days.

I configured the server with only external open port is 8080. Other two ports (SSH and WEBMIN) are open for internal IP only. External access is possible only through VPN client. According to the firewall maintaining vendor, that hacker might access through 8080. How we prevent and secure that?

I configure the database in other server and that server is only accessible through one private IP block. The tomcat server, the backup servers and our administrator/development team are in that block.

Now please suggest how can we secure our servers more.

Regards

Muhammad Abdul Hannan Khan

Senior Technical Advisor - HIS

Priority Area Health

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH

House10/A, Road 90, Gulshan 2, Dhaka 1212, Bangladesh

T +880-2- 8816459, 8816412 ext 118

M+88 01819 239 241

M+88 01534 312 066

F +88 02 8813 875

E hannan.khan@giz.de

S hannan.khan.dhaka

B hannan-tech.blogspot.com

10

Hi Hannan

I agree with Lars that what you describe (uploading of malicious files) sounds like an exploitation of the vulnerability in struts fixed around Christmas.

I also agree that you don’t want to run tomcat on port 80. The recommended configuration is to run a web proxy such as nginx or apache2 on port 80 and 443. Then tomcat can listen on port 8080 on localhost only.

···

On 6 February 2014 14:54, Hannan Khan hannank@gmail.com wrote:

Thanks Jason for your comprehensive advice.

I tried to identify problem roots and I believe I find those files. And there is no problem so far.

From the beginning I am running tomcat service as user who cannot login to the system.

Point 2 and 3 I have to do. But earlier our another serer running on port 80 severely damaged by hacker attack (web server). I will be keep in-touch on this.

Any firewall you suggests? Also consider we have very narrow bandwidth; only 10 Mbps for 9 dhis2 systems with near about 12000 users average 300 concurrent user in top three systems;

Updates we are run weekly basis.

Point 6 and 7 I will do. How that will effect the system performance?

Regards

Hannan

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Feb 6, 2014 at 1:18 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Hannan,
I had several servers (4 to be exact) which were compromised due to a vulnerability in Struts. Lars sent out an email a few weeks ago, that informed everyone they needed to upgrade immediately. I know of other server which have also been compromised. One was running Tomcat as root (an exceptionally bad idea). Because of the compromise, a full reinstallation of the server software would be required.

In your case, it does seem to be a bit more serious, and not consistent with the previous compromises I have seen. These compromises were limited to the machine sending out a huge amount of traffic, but otherwise, there did not “seem” to be any further issues.

A few tips, you may want to consider

  1. A complete reinstall of the system might be in order, given the extent of the attack.
  1. Be sure that the Tomcat process is not running as root, and that the user which can execute Tomcat cannot login to the system directly (i.e. has their shell set to /bin/false)
  1. Close port 8080 and remove the Tomcat manager. Instead, only have port 80/443 on the machine open. Additionally, do not run SSH on port 22, and be sure that you can only login to the server with a key, which is protected itself by a strong password.
  1. Consider attempting to look for vulnerabilities your self, with tools such as Nessus and Nmap
  1. Ensure that you are running a firewall on the server itself, i.e. do not trust your upstream providers firewall.
  1. Ensure that all Tomcat installs, Java,DHIS2 and the system software itself is fully up to date
  1. Consider running an IDS such as OSSEC on your machine to look for unauthorized intrusions.
  1. Use tools such as monit to monitor for spurious processes or suspicious file activity.

Hope this helps.

Best regards,

Jason

On Thu, Feb 6, 2014 at 8:36 AM, Hannan Khan hannank@gmail.com wrote:

Yes Morten, I installed through the package manager.

The tomcat version is Apache Tomcat/7.0.26.

Regards

Hannan

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Feb 6, 2014 at 12:07 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Also make sure that your tomcat is up to date… there exists several vulnerabilities in older versions

(not sure how you installed it, but if you are using a linux distribution, its wise to install it through the package manager)

–
Morten

On Thu, Feb 6, 2014 at 1:00 PM, Knut Staring knutst@gmail.com wrote:

Hannan, which build of DHIS2 ? Which Java version? Ubuntu?

Sent from my mobile

On Feb 6, 2014 6:29 AM, “Hannan Khan” hannank@gmail.com wrote:

Dear experts

Our main DHIS2 implementation (mishealth) for the health sector was hacked yesterday evening, around 4:30 PM local time. After login by any user it showing the attached message. We immediately stop the tomact7 service and check the database. We find the database is intact.

After investigation I find that the hacker inserted three files to do this.

First file “index.html” contain an alert “alert(“Admin, You Are Hacked by Malaysia Hacker!”)” and a body text

Hacked by BadCat

. Which was placed in the application folder /tomcat7/webapps/mishealth/.

Second files “index.html” contain another script which redirects to “pastebin.com/raw.php?i=LZEdbBz6” was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

Third file “guige.jsp” is contain a script was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

For our server, it seems that only first file is executing after login. I find few more suspicious files which I am investigating and will share with the experts in next few days.

I configured the server with only external open port is 8080. Other two ports (SSH and WEBMIN) are open for internal IP only. External access is possible only through VPN client. According to the firewall maintaining vendor, that hacker might access through 8080. How we prevent and secure that?

I configure the database in other server and that server is only accessible through one private IP block. The tomcat server, the backup servers and our administrator/development team are in that block.

Now please suggest how can we secure our servers more.

Regards

Muhammad Abdul Hannan Khan

Senior Technical Advisor - HIS

Priority Area Health

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH

House10/A, Road 90, Gulshan 2, Dhaka 1212, Bangladesh

T +880-2- 8816459, 8816412 ext 118

M+88 01819 239 241

M+88 01534 312 066

F +88 02 8813 875

E hannan.khan@giz.de

S hannan.khan.dhaka

B hannan-tech.blogspot.com

11

Thanks Bob.

My earlier attempt to configure nginx failed. I will try again this month.

Regards

Hannan

···

On Thu, Feb 6, 2014 at 10:09 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Hi Hannan

I agree with Lars that what you describe (uploading of malicious files) sounds like an exploitation of the vulnerability in struts fixed around Christmas.

I also agree that you don’t want to run tomcat on port 80. The recommended configuration is to run a web proxy such as nginx or apache2 on port 80 and 443. Then tomcat can listen on port 8080 on localhost only.

On 6 February 2014 14:54, Hannan Khan hannank@gmail.com wrote:

Thanks Jason for your comprehensive advice.

I tried to identify problem roots and I believe I find those files. And there is no problem so far.

From the beginning I am running tomcat service as user who cannot login to the system.

Point 2 and 3 I have to do. But earlier our another serer running on port 80 severely damaged by hacker attack (web server). I will be keep in-touch on this.

Any firewall you suggests? Also consider we have very narrow bandwidth; only 10 Mbps for 9 dhis2 systems with near about 12000 users average 300 concurrent user in top three systems;

Updates we are run weekly basis.

Point 6 and 7 I will do. How that will effect the system performance?

Regards

Hannan

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Feb 6, 2014 at 1:18 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Hannan,
I had several servers (4 to be exact) which were compromised due to a vulnerability in Struts. Lars sent out an email a few weeks ago, that informed everyone they needed to upgrade immediately. I know of other server which have also been compromised. One was running Tomcat as root (an exceptionally bad idea). Because of the compromise, a full reinstallation of the server software would be required.

In your case, it does seem to be a bit more serious, and not consistent with the previous compromises I have seen. These compromises were limited to the machine sending out a huge amount of traffic, but otherwise, there did not “seem” to be any further issues.

A few tips, you may want to consider

  1. A complete reinstall of the system might be in order, given the extent of the attack.
  1. Be sure that the Tomcat process is not running as root, and that the user which can execute Tomcat cannot login to the system directly (i.e. has their shell set to /bin/false)
  1. Close port 8080 and remove the Tomcat manager. Instead, only have port 80/443 on the machine open. Additionally, do not run SSH on port 22, and be sure that you can only login to the server with a key, which is protected itself by a strong password.
  1. Consider attempting to look for vulnerabilities your self, with tools such as Nessus and Nmap
  1. Ensure that you are running a firewall on the server itself, i.e. do not trust your upstream providers firewall.
  1. Ensure that all Tomcat installs, Java,DHIS2 and the system software itself is fully up to date
  1. Consider running an IDS such as OSSEC on your machine to look for unauthorized intrusions.
  1. Use tools such as monit to monitor for spurious processes or suspicious file activity.

Hope this helps.

Best regards,

Jason

On Thu, Feb 6, 2014 at 8:36 AM, Hannan Khan hannank@gmail.com wrote:

Yes Morten, I installed through the package manager.

The tomcat version is Apache Tomcat/7.0.26.

Regards

Hannan

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

On Thu, Feb 6, 2014 at 12:07 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

Also make sure that your tomcat is up to date… there exists several vulnerabilities in older versions

(not sure how you installed it, but if you are using a linux distribution, its wise to install it through the package manager)

–
Morten

On Thu, Feb 6, 2014 at 1:00 PM, Knut Staring knutst@gmail.com wrote:

Hannan, which build of DHIS2 ? Which Java version? Ubuntu?

Sent from my mobile

On Feb 6, 2014 6:29 AM, “Hannan Khan” hannank@gmail.com wrote:

Dear experts

Our main DHIS2 implementation (mishealth) for the health sector was hacked yesterday evening, around 4:30 PM local time. After login by any user it showing the attached message. We immediately stop the tomact7 service and check the database. We find the database is intact.

After investigation I find that the hacker inserted three files to do this.

First file “index.html” contain an alert “alert(“Admin, You Are Hacked by Malaysia Hacker!”)” and a body text

Hacked by BadCat

. Which was placed in the application folder /tomcat7/webapps/mishealth/.

Second files “index.html” contain another script which redirects to “pastebin.com/raw.php?i=LZEdbBz6” was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

Third file “guige.jsp” is contain a script was placed in the /tomcat7/webapps/mishealth/dhis-web-commons/security/.

For our server, it seems that only first file is executing after login. I find few more suspicious files which I am investigating and will share with the experts in next few days.

I configured the server with only external open port is 8080. Other two ports (SSH and WEBMIN) are open for internal IP only. External access is possible only through VPN client. According to the firewall maintaining vendor, that hacker might access through 8080. How we prevent and secure that?

I configure the database in other server and that server is only accessible through one private IP block. The tomcat server, the backup servers and our administrator/development team are in that block.

Now please suggest how can we secure our servers more.

Regards

Muhammad Abdul Hannan Khan

Senior Technical Advisor - HIS

Priority Area Health

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH

House10/A, Road 90, Gulshan 2, Dhaka 1212, Bangladesh

T +880-2- 8816459, 8816412 ext 118

M+88 01819 239 241

M+88 01534 312 066

F +88 02 8813 875

E hannan.khan@giz.de

S hannan.khan.dhaka

B hannan-tech.blogspot.com