Server processor use 100%

Dear Experts

I have an wired situation. one of our DHIS2 server running older war files (version 16), the OS was outdated and we have to upgrade the OS. After installing new OS Ubuntu 16.04 LTS all necessary component Java 8 and Tomcat 7 was installed by after running war file (version 16) after few minutes the tomcat7 is not operational as the processor use is 100%. there is only 1 user logged in and the application server using 2 processor and DB server is separate.

After trying several times I remove tomcat7 and install tomcat 8 with same war file, but situation is same. I called it wired as the db size is quite small, user is only few and the listing showing SSHD command by tomcat8 user is using 100% processor.

Any idea about the under line reason? need urgent help. Thank you all in advance.

Regards

image

···

Muhammad Abdul Hannan Khan

Team Leader

Support to the National HMIS

MIS, Director General of Health Service

Ministry of Health and Family Welfare

T +880-2- 58816459, 58816412 ext 118

F +88 02 58813 875

M+88 01819 239 241

M+88 01534 312 066

E hannank@gmail.com

S hannan.khan.dhaka

B hannan-tech.blogspot.com

L https://bd.linkedin.com/in/hannankhan

Hi Hannan

There is no circumstance that tomcat user should be running the sshd command. It could be this machine has been compromised. Unless you have some strange setup that you are logging in as tomcat user.

Please contact me directly if you want me to check.

Meanwhile you might want to have a look in /tmp directory and tomcat8 home directory to see if there are any strange files there:

ls -ls /tmp

You might find that there is a rogue sshd program that has been installed there. Note that if you are running a very old war file your risk of compromise is very high.

Bob

···

On 10 July 2017 at 05:09, Hannan Khan hannank@gmail.com wrote:

Dear Experts

I have an wired situation. one of our DHIS2 server running older war files (version 16), the OS was outdated and we have to upgrade the OS. After installing new OS Ubuntu 16.04 LTS all necessary component Java 8 and Tomcat 7 was installed by after running war file (version 16) after few minutes the tomcat7 is not operational as the processor use is 100%. there is only 1 user logged in and the application server using 2 processor and DB server is separate.

After trying several times I remove tomcat7 and install tomcat 8 with same war file, but situation is same. I called it wired as the db size is quite small, user is only few and the listing showing SSHD command by tomcat8 user is using 100% processor.

Any idea about the under line reason? need urgent help. Thank you all in advance.

Regards

Muhammad Abdul Hannan Khan

Team Leader

Support to the National HMIS

MIS, Director General of Health Service

Ministry of Health and Family Welfare

T +880-2- 58816459, 58816412 ext 118

F +88 02 58813 875

M+88 01819 239 241

M+88 01534 312 066

E hannank@gmail.com

S hannan.khan.dhaka

B hannan-tech.blogspot.com

L https://bd.linkedin.com/in/hannankhan

Sorry that should have been ‘ls -la /tmp’

···

On 10 July 2017 at 10:50, Bob Jolliffe bobjolliffe@gmail.com wrote:

Hi Hannan

There is no circumstance that tomcat user should be running the sshd command. It could be this machine has been compromised. Unless you have some strange setup that you are logging in as tomcat user.

Please contact me directly if you want me to check.

Meanwhile you might want to have a look in /tmp directory and tomcat8 home directory to see if there are any strange files there:

ls -ls /tmp

You might find that there is a rogue sshd program that has been installed there. Note that if you are running a very old war file your risk of compromise is very high.

Bob

On 10 July 2017 at 05:09, Hannan Khan hannank@gmail.com wrote:

Dear Experts

I have an wired situation. one of our DHIS2 server running older war files (version 16), the OS was outdated and we have to upgrade the OS. After installing new OS Ubuntu 16.04 LTS all necessary component Java 8 and Tomcat 7 was installed by after running war file (version 16) after few minutes the tomcat7 is not operational as the processor use is 100%. there is only 1 user logged in and the application server using 2 processor and DB server is separate.

After trying several times I remove tomcat7 and install tomcat 8 with same war file, but situation is same. I called it wired as the db size is quite small, user is only few and the listing showing SSHD command by tomcat8 user is using 100% processor.

Any idea about the under line reason? need urgent help. Thank you all in advance.

Regards

Muhammad Abdul Hannan Khan

Team Leader

Support to the National HMIS

MIS, Director General of Health Service

Ministry of Health and Family Welfare

T +880-2- 58816459, 58816412 ext 118

F +88 02 58813 875

M+88 01819 239 241

M+88 01534 312 066

E hannank@gmail.com

S hannan.khan.dhaka

B hannan-tech.blogspot.com

L https://bd.linkedin.com/in/hannankhan

Dear Bob

Sorry for replaying late. I quite busy to complete few incomplete tasks before I am going on holiday tomorrow for a week.

I have checked for few day with various options and my conclusion is that the security hole might be created by our old war file (version 16) with Stuart vulnerability which Lars warn all of us earlier. We upgraded all our servers and applications except this server. No suspicious files in the tmp folders.

It took control of Tomcat8 user and run SSHD and occupies 100% of 2 processors. When we kill the process and remove all war files and stop tomcat8 service it stared ATD command and it also occupy 100% of 2 processors. The data seems intact (through query and size). As our all DB servers have similar IP structure we immediately remove tomcat8 service, package and user. The VM server will also be decommissioned and will setup a new server with new cardinals. I will start upgrade work after I return.

Thank you for your valuable advice and kind concern.

Best regards

Hannan

···

On Mon, Jul 10, 2017 at 8:21 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Sorry that should have been ‘ls -la /tmp’

On 10 July 2017 at 10:50, Bob Jolliffe bobjolliffe@gmail.com wrote:

Hi Hannan

There is no circumstance that tomcat user should be running the sshd command. It could be this machine has been compromised. Unless you have some strange setup that you are logging in as tomcat user.

Please contact me directly if you want me to check.

Meanwhile you might want to have a look in /tmp directory and tomcat8 home directory to see if there are any strange files there:

ls -ls /tmp

You might find that there is a rogue sshd program that has been installed there. Note that if you are running a very old war file your risk of compromise is very high.

Bob

On 10 July 2017 at 05:09, Hannan Khan hannank@gmail.com wrote:

Dear Experts

I have an wired situation. one of our DHIS2 server running older war files (version 16), the OS was outdated and we have to upgrade the OS. After installing new OS Ubuntu 16.04 LTS all necessary component Java 8 and Tomcat 7 was installed by after running war file (version 16) after few minutes the tomcat7 is not operational as the processor use is 100%. there is only 1 user logged in and the application server using 2 processor and DB server is separate.

After trying several times I remove tomcat7 and install tomcat 8 with same war file, but situation is same. I called it wired as the db size is quite small, user is only few and the listing showing SSHD command by tomcat8 user is using 100% processor.

Any idea about the under line reason? need urgent help. Thank you all in advance.

Regards

Muhammad Abdul Hannan Khan

Team Leader

Support to the National HMIS

MIS, Director General of Health Service

Ministry of Health and Family Welfare

T +880-2- 58816459, 58816412 ext 118

F +88 02 58813 875

M+88 01819 239 241

M+88 01534 312 066

E hannank@gmail.com

S hannan.khan.dhaka

B hannan-tech.blogspot.com

L https://bd.linkedin.com/in/hannankhan

Muhammad Abdul Hannan Khan

Team Leader

Support to the National HMIS

MIS, Director General of Health Service

Ministry of Health and Family Welfare

T +880-2- 58816459, 58816412 ext 118

F +88 02 58813 875

M+88 01819 239 241

M+88 01534 312 066

E hannank@gmail.com

S hannan.khan.dhaka

B hannan-tech.blogspot.com

L https://bd.linkedin.com/in/hannankhan

Yes, Hannan that is similar to what I have seen a number of times this year. The attacker makes use of atd and/or crontab to execute malicious code. The good thing is that your tomcat was not running as root which would be potentially more damaging.

Obviously with access to the tomcat user then access to the database itself has been exposed. There is no indication that the database was the target of previous exploits so probably (hopefully) that is your case too. It is a really good illustration though of why, when you have multiple instances attaching to a database server, you should always use a separate database role/user for each. So when one database is exposed (through access to dhis.conf), at least they are not all exposed.

Enjoy your holiday. I am hoping to get off as well soon :slight_smile:

Regards

Bob

···

On 13 July 2017 at 16:01, Hannan Khan hannank@gmail.com wrote:

Dear Bob

Sorry for replaying late. I quite busy to complete few incomplete tasks before I am going on holiday tomorrow for a week.

I have checked for few day with various options and my conclusion is that the security hole might be created by our old war file (version 16) with Stuart vulnerability which Lars warn all of us earlier. We upgraded all our servers and applications except this server. No suspicious files in the tmp folders.

It took control of Tomcat8 user and run SSHD and occupies 100% of 2 processors. When we kill the process and remove all war files and stop tomcat8 service it stared ATD command and it also occupy 100% of 2 processors. The data seems intact (through query and size). As our all DB servers have similar IP structure we immediately remove tomcat8 service, package and user. The VM server will also be decommissioned and will setup a new server with new cardinals. I will start upgrade work after I return.

Thank you for your valuable advice and kind concern.

Best regards

Hannan

On Mon, Jul 10, 2017 at 8:21 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Sorry that should have been ‘ls -la /tmp’


Muhammad Abdul Hannan Khan

Team Leader

Support to the National HMIS

MIS, Director General of Health Service

Ministry of Health and Family Welfare

T +880-2- 58816459, 58816412 ext 118

F +88 02 58813 875

M+88 01819 239 241

M+88 01534 312 066

E hannank@gmail.com

S hannan.khan.dhaka

B hannan-tech.blogspot.com

L https://bd.linkedin.com/in/hannankhan

On 10 July 2017 at 10:50, Bob Jolliffe bobjolliffe@gmail.com wrote:

Hi Hannan

There is no circumstance that tomcat user should be running the sshd command. It could be this machine has been compromised. Unless you have some strange setup that you are logging in as tomcat user.

Please contact me directly if you want me to check.

Meanwhile you might want to have a look in /tmp directory and tomcat8 home directory to see if there are any strange files there:

ls -ls /tmp

You might find that there is a rogue sshd program that has been installed there. Note that if you are running a very old war file your risk of compromise is very high.

Bob

On 10 July 2017 at 05:09, Hannan Khan hannank@gmail.com wrote:

Dear Experts

I have an wired situation. one of our DHIS2 server running older war files (version 16), the OS was outdated and we have to upgrade the OS. After installing new OS Ubuntu 16.04 LTS all necessary component Java 8 and Tomcat 7 was installed by after running war file (version 16) after few minutes the tomcat7 is not operational as the processor use is 100%. there is only 1 user logged in and the application server using 2 processor and DB server is separate.

After trying several times I remove tomcat7 and install tomcat 8 with same war file, but situation is same. I called it wired as the db size is quite small, user is only few and the listing showing SSHD command by tomcat8 user is using 100% processor.

Any idea about the under line reason? need urgent help. Thank you all in advance.

Regards

Muhammad Abdul Hannan Khan

Team Leader

Support to the National HMIS

MIS, Director General of Health Service

Ministry of Health and Family Welfare

T +880-2- 58816459, 58816412 ext 118

F +88 02 58813 875

M+88 01819 239 241

M+88 01534 312 066

E hannank@gmail.com

S hannan.khan.dhaka

B hannan-tech.blogspot.com

L https://bd.linkedin.com/in/hannankhan

Thanks Bob.

···

On Thu, Jul 13, 2017 at 10:13 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Yes, Hannan that is similar to what I have seen a number of times this year. The attacker makes use of atd and/or crontab to execute malicious code. The good thing is that your tomcat was not running as root which would be potentially more damaging.

Obviously with access to the tomcat user then access to the database itself has been exposed. There is no indication that the database was the target of previous exploits so probably (hopefully) that is your case too. It is a really good illustration though of why, when you have multiple instances attaching to a database server, you should always use a separate database role/user for each. So when one database is exposed (through access to dhis.conf), at least they are not all exposed.

Enjoy your holiday. I am hoping to get off as well soon :slight_smile:

Regards

Bob

On 13 July 2017 at 16:01, Hannan Khan hannank@gmail.com wrote:

Dear Bob

Sorry for replaying late. I quite busy to complete few incomplete tasks before I am going on holiday tomorrow for a week.

I have checked for few day with various options and my conclusion is that the security hole might be created by our old war file (version 16) with Stuart vulnerability which Lars warn all of us earlier. We upgraded all our servers and applications except this server. No suspicious files in the tmp folders.

It took control of Tomcat8 user and run SSHD and occupies 100% of 2 processors. When we kill the process and remove all war files and stop tomcat8 service it stared ATD command and it also occupy 100% of 2 processors. The data seems intact (through query and size). As our all DB servers have similar IP structure we immediately remove tomcat8 service, package and user. The VM server will also be decommissioned and will setup a new server with new cardinals. I will start upgrade work after I return.

Thank you for your valuable advice and kind concern.

Best regards

Hannan

On Mon, Jul 10, 2017 at 8:21 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Sorry that should have been ‘ls -la /tmp’


Muhammad Abdul Hannan Khan

Team Leader

Support to the National HMIS

MIS, Director General of Health Service

Ministry of Health and Family Welfare

T +880-2- 58816459, 58816412 ext 118

F +88 02 58813 875

M+88 01819 239 241

M+88 01534 312 066

E hannank@gmail.com

S hannan.khan.dhaka

B hannan-tech.blogspot.com

L https://bd.linkedin.com/in/hannankhan

On 10 July 2017 at 10:50, Bob Jolliffe bobjolliffe@gmail.com wrote:

Hi Hannan

There is no circumstance that tomcat user should be running the sshd command. It could be this machine has been compromised. Unless you have some strange setup that you are logging in as tomcat user.

Please contact me directly if you want me to check.

Meanwhile you might want to have a look in /tmp directory and tomcat8 home directory to see if there are any strange files there:

ls -ls /tmp

You might find that there is a rogue sshd program that has been installed there. Note that if you are running a very old war file your risk of compromise is very high.

Bob

On 10 July 2017 at 05:09, Hannan Khan hannank@gmail.com wrote:

Dear Experts

I have an wired situation. one of our DHIS2 server running older war files (version 16), the OS was outdated and we have to upgrade the OS. After installing new OS Ubuntu 16.04 LTS all necessary component Java 8 and Tomcat 7 was installed by after running war file (version 16) after few minutes the tomcat7 is not operational as the processor use is 100%. there is only 1 user logged in and the application server using 2 processor and DB server is separate.

After trying several times I remove tomcat7 and install tomcat 8 with same war file, but situation is same. I called it wired as the db size is quite small, user is only few and the listing showing SSHD command by tomcat8 user is using 100% processor.

Any idea about the under line reason? need urgent help. Thank you all in advance.

Regards

Muhammad Abdul Hannan Khan

Team Leader

Support to the National HMIS

MIS, Director General of Health Service

Ministry of Health and Family Welfare

T +880-2- 58816459, 58816412 ext 118

F +88 02 58813 875

M+88 01819 239 241

M+88 01534 312 066

E hannank@gmail.com

S hannan.khan.dhaka

B hannan-tech.blogspot.com

L https://bd.linkedin.com/in/hannankhan

Muhammad Abdul Hannan Khan

Team Leader

Support to the National HMIS

MIS, Director General of Health Service

Ministry of Health and Family Welfare

T +880-2- 58816459, 58816412 ext 118

F +88 02 58813 875

M+88 01819 239 241

M+88 01534 312 066

E hannank@gmail.com

S hannan.khan.dhaka

B hannan-tech.blogspot.com

L https://bd.linkedin.com/in/hannankhan