Our site suddenly start showing HTTP Status 404 – Not Found. When I checked the Catalina log, I saw that the docker container has suddenly stop working.
The server was working just fine minutes ago. The only thing I can say is that we have users trying to submit thousands of data now. Could that be responsible?
When I checked the log, I got the following errors
13-Jan-2023 16:00:00.919 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to start component [Connector[AJP/1.3-8009]]
org.apache.catalina.LifecycleException: Protocol handler start failed
I think it has to do with the database! Too many requests to the database -slow queries consuming the RAM. A restart of the database should restore the services to - terminates the slow queries brings back the system.
@1234 I suspect the high volume request because I just we have thousand s of data submission ongoing. Also I just created some visualizer and dashboard which added to the high volume query. I will see what can be done.
I query the database and it was working. Rebuilding the docker image and restarting the container did bring back the server.
I guess so. Because the docker container stopped working.
I removed the container and use the opportunity to upgrade to 2.39.0.1. the server is back now. I will try to investigate further what could be responsible. But the docker container stuff is really an easy going process for me.
what do you mean by connecting to local volume?
It was easy for me to run the project through a docker container. Tomcat installation was a bit challenging.
If you stop your docker container or accidently delete your docker container or your docker container gets corrupted you are at risk of losing all your data of that DHIS2 instance.
Therefore it’s always proffered to create a docker volume for your docker container, which prevents you from accidentally losing your data.
Yes I know installing via Docker can be easy, but once you get the hang of installing on Tomcat it’s not that difficult. If you need support for installing DHIS2 on Tomcat feel free to drop a message. More than happy to help.
Unless this has changed, which I doubt but pinging @phil / @bobj here, Docker is not advised to be run into production environments which it seems to be what you are doing. I thought it was just worth mentioning it here.
Is there a special reason for not using it in production environment? I am using a custom docker deployment locally and in production environment which has been working fine so far. But I would like to know why I should do otherwise since I already sold this to the agency.
One reason that comes to my mind is you can’t customize the DHIS2 instance at code level
Another reason is it’s easier to configure different tools like Tomcat, Nginx when they are installed separately and load balancing and other configurations becomes easier.
Also as I mentioned earlier if you don’t use docker volume you can risk loosing your data.
These are some common ideas that comes to my mind. But I’m sure others are more qualified than me to give DHIS2 specific reasons.
Currently I made some few changes to the DHIS2 Instance before I build the custom image. So I don’t think this apply if you are using a custom docker image.
I am using Nginx as part of my custom configuration for load balancing although with some limitation I can work around when the need arise. If I want to make changes to my Nginx configuration I can do that and simply rebuild the image and redeploy the container seperately.
As I said earlier I am using Docker Volume for the DHIS2 Instance and other container I deployed.
That’s the part I want to find out so that I will have good reason to suggest a change in how we deploy for production. Currently I can do any customization I want without disrupting the system.
Thanks for the tips. I still await if there are others who can shed more light on this.
To clarify, we cannot recommend the DHIS2 Docker images published to https://hub.docker.com/u/dhis2 for use in production. At this point we don’t have enough experience and therefore can’t vouch for their stability for “mission critical” production use. We are not saying that someone experienced with running Docker in production shouldn’t use them.
In other words, anyone deciding to use DHIS2 in Docker should be aware that they are doing so at their own risk.
Thanks for the clarification. I carefully setup a custom docker image and container with volume. So, far it has worked fine except the one time I suffer the server going off. I am yet to fully establish the cause.
The important thing is to understand why it is that your DHIS2 instance went down. This is likely to be some combination of your available system resources, your java, tomcat and DHIS2 configurations and your actual traffic patterns. It probably has little to do with whether you are running DHIS2 in a docker environment or not. So the docker part of the question is a bit of a red herring.
Finding the root of the problem might involve some combination of (i) monitoring, eg https://glowroot.org (its not really documented how to do this with docker but it can be done) (ii) analysis of nginx, tomcat and postgresql logs and (iii) trying to replicate the requests which caused the problem. Finding the root cause of these problems can be tricky.
My (absolutely wild) guess is that your tomcat/docker image either has insufficient RAM or you are not utilizing it effectively with JAVA_OPTS. That is what is typically behind the tomcat process dying completely. Tell us a little more about your environment, beyond the fact that you are using docker, and I can try to refine my theories
What are your JAVA_OPTS like for the dhis2 tomcat? ie how much heap space are you hoping to be able to allocate.
Is your postgresql also running inside a container, or unconstrained on the host. One thing I have seen is that postgresql will naturally try to be greedy and use whatever RAM it sees. Unlike java, which has some constraints enforced, the database will do what databases do. This can sometimes cause a problem with DHIS2 tomcat if java is unable to allocate memory you have told it that it has access to. So actually running postgresql in a container and limiting its memory is a good thing to do if you are running other services on the same machine. Or limit it with cgroups or just run it on a separate vm.
Note if my (wild) theory is correct and your tomcat has fallen over due to memory issues you will see some trace of this in tomcat and/or system logs.
Noted and I will suggest an increase. But in the meantime we are stuck with it.
Honestly I don’t know about this. I just use the configuration in dhis2 as is. But I noticed that this problem occur when there is special program and there is thousands of queries to the database.
The postgresql is running inside a container on the same host.
I’ll read more and explore these options.
I’ll try to see if I can explore the system log because I am not sure about the tomcat.
