Top command showing ~1000% cpu for java process

Support - Assistance technique
1

Hello all,

In some of our dhis2 instances we get unrealistically high 800+ or 1000% of cpu usage for a java process and it’s points /opt/java

Our systems are not intensively used during this time, is it because of any configurations, and strangely /opt/java is not the location we have java installed.

This is not affecting dhis2, but slowness is observed.Would like to know your thoughts on this

image
image1990×738 130 KB

2

Hi @jthomas ,

Do you have any other java application in your instance? Also, looks like this process is running from a very long time.
You can run “ps -ef | grep java” to get more info on which java processes are running. If you think it is a process that is not necessary anymore, you can terminate it.

Thanks,
Rithvik

1 Like
3

Hi @rithvik

there is no other java installations.

Actually it’s a bit more than killing that process, because the process id shows /opt/java which is not the real java we configured, and if we kill that process another process gets started automatically

So i’m thinking it could be a JRE related process? and may be some configurations can solve?

4

Hi Jins,

Thanks for reaching out to the community. Without raising any alarm bells, this may not necesarily be a java process but instead something else masquerading as Java.

First point is to address the levels of security on the server and to eliminate any potential breaches.

  1. Do you have a firewall enabled ?
  2. Is your server public facing ?
  3. Which version of DHIS2 are you running and have you patched it for whichever vulnerabilities have been discovered for that version ? Log4J and Spring4Shell come to mind.
  4. which applications are publicly accessible ? in other words, unix sockets listening on the internet accessible IP address.

Perhaps we could start there and try to eliminate those first. Apologies for leaning overly on the security stuff but the last time we experienced something high CPU usage that we couldn’t account for, our Security team discovered a breach on the server. We consequently had to rebuild it.

Looking forward to your response.

Kind regards,
Potlaki Moloi
HISP SA
Telegram: @potlaki

2 Likes
5

Hi Potlaki, Thanks alot for this conversation, really appreciate. and interesting prespective, I think it is similar to what you tell

  1. We have ufw enabled to allow only relevant ports, this is what you meant by firewall? Beyond this we don’t have any firewalls

image
image1158×560 24.9 KB

  1. It’s a public-facing, in the sense, anybody with ip address and password can login

  2. We are running 2.36.4, i think this latest of this family is 2.36.11, we haven’t applied any patch on log4j & spring4shell, probably we shall upgrade us to latest of 2.36

  3. ufw output at the beginning addresses this question.

One another reason why i also feel it’s a masquerading is because of their directory structure and /opt/

image
image926×1154 74.7 KB

image
image926×1348 80.6 KB

image
image926×1532 114 KB

1 Like
6

Hi @potlaki A humble reminder on this: Can i just like that identify this process and kill or remove for ever?

7

@jthomas,
Also, @mmarkevich suggested that “you can run ps uax | grep java to find the exact path of the 2nd executable in /opt and then check if it belongs to any known system packages”

1 Like