We are using dhis2 version 2.7. Whenever we generating datamart are it taking long time and it never ends.The error message is (in the tomcat log file) ‘dhis2-org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren SEVERE: Exception invoking periodic operation: java.lang.OutOfMemoryError: Java heap space’.
We are using dhis2 version 2.7. Whenever we generating datamart are it taking long time and it never ends.The error message is (in the tomcat log file) ‘dhis2-org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren SEVERE: Exception invoking periodic operation: java.lang.OutOfMemoryError: Java heap space’.
Thanks for your advice. We are running dhis2 from 2010. In earlier version we dont have any problem creating datamart. After updating to version 2.7 we imported a big chuck of data from two data souce and after that we have problem creating datamart.
So far I know the configuration is ok, however I will check again and get back to you.
We are using dhis2 version 2.7. Whenever we generating datamart are it taking long time and it never ends.The error message is (in the tomcat log file) ‘dhis2-org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren SEVERE: Exception invoking periodic operation: java.lang.OutOfMemoryError: Java heap space’.
I am not really sure what your settings are, but you should at least
specify these options, of course adjusting them for the amount of
memory which can be allocated to DHIS2 .
Best regards,
Jason
···
On Mon, Sep 10, 2012 at 10:49 AM, Hannan Khan <hannank@gmail.com> wrote:
Dear Jason
Thanks for your advice. We are running dhis2 from 2010. In earlier version
we dont have any problem creating datamart. After updating to version 2.7 we
imported a big chuck of data from two data souce and after that we have
problem creating datamart.
So far I know the configuration is ok, however I will check again and get
back to you.
This is not a bug. You must increase the heap size. Please consult the
implementation manual for details.
Regards,
Jason
--
Sent from my mobile
On Sep 9, 2012 1:12 PM, "Hannan Khan" <hannank@gmail.com> wrote:
We are using dhis2 version 2.7. Whenever we generating datamart are it
taking long time and it never ends.The error message is (in the tomcat log
file)
'dhis2-org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
processChildren SEVERE: Exception invoking periodic operation:
java.lang.OutOfMemoryError: Java heap space'.
I put the Java heap space 1GB and Maximum 6GB (Xmx6144m and Xms1024m). But still getting ‘java.lang.OutOfMemoryError: Java heap space’ error. That why I asked that whether that is a bug or not.
I am trying to other options and contact you soon.
I put the Java heap space 1GB and Maximum 6GB (Xmx6144m and Xms1024m). But still getting ‘java.lang.OutOfMemoryError: Java heap space’ error. That why I asked that whether that is a bug or not.
I am trying to other options and contact you soon.
I put the Java heap space 1GB and Maximum 6GB (Xmx6144m and Xms1024m). But still getting ‘java.lang.OutOfMemoryError: Java heap space’ error. That why I asked that whether that is a bug or not.
I am trying to other options and contact you soon.
I put the Java heap space 1GB and Maximum 6GB (Xmx6144m and Xms1024m). But still getting ‘java.lang.OutOfMemoryError: Java heap space’ error. That why I asked that whether that is a bug or not.
I am trying to other options and contact you soon.
I put the Java heap space 1GB and Maximum 6GB (Xmx6144m and Xms1024m). But still getting ‘java.lang.OutOfMemoryError: Java heap space’ error. That why I asked that whether that is a bug or not.
I am trying to other options and contact you soon.
That’s a really big database… I think maybe Lars should answer if the memory of the server is large enough (I’m sure he can compare it with what he has been doing to tune the Kenya server).
···
–
Morten
On Tue, Sep 11, 2012 at 2:59 PM, Hannan Khan hannank@gmail.com wrote:
Hi Morten
Server have 8GB memory and database is 3159MB. We are using MySQL.
Regards
Hannan
On Tue, Sep 11, 2012 at 1:53 PM, Morten Olav Hansen mortenoh@gmail.com wrote:
Hi
How much memory does your server have? and what is the size of your database?
–
Morten
On Tue, Sep 11, 2012 at 2:51 PM, Hannan Khan hannank@gmail.com wrote:
I put the Java heap space 1GB and Maximum 6GB (Xmx6144m and Xms1024m). But still getting ‘java.lang.OutOfMemoryError: Java heap space’ error. That why I asked that whether that is a bug or not.
I am trying to other options and contact you soon.
I might found an alternative solution. Yesterday we run data mart for 9 month and it was completed successfully but when we run for 5 years, the errors arrived.
Today and trying with again and get back to you all with the result. But when we run datamart for 9 moth it takes 34 minutes, which is long is not it?
Regards
Hannan
···
On Tue, Sep 11, 2012 at 2:01 PM, Morten Olav Hansen mortenoh@gmail.com wrote:
That’s a really big database… I think maybe Lars should answer if the memory of the server is large enough (I’m sure he can compare it with what he has been doing to tune the Kenya server).
–
Morten
On Tue, Sep 11, 2012 at 2:59 PM, Hannan Khan hannank@gmail.com wrote:
Hi Morten
Server have 8GB memory and database is 3159MB. We are using MySQL.
Regards
Hannan
On Tue, Sep 11, 2012 at 1:53 PM, Morten Olav Hansen mortenoh@gmail.com wrote:
Hi
How much memory does your server have? and what is the size of your database?
–
Morten
On Tue, Sep 11, 2012 at 2:51 PM, Hannan Khan hannank@gmail.com wrote:
I put the Java heap space 1GB and Maximum 6GB (Xmx6144m and Xms1024m). But still getting ‘java.lang.OutOfMemoryError: Java heap space’ error. That why I asked that whether that is a bug or not.
I am trying to other options and contact you soon.
Perhaps this is a case where it would make sense to switch to
PostgreSQL and possibly also make use of more of the internal
crunching capabilities of the database?
Knut
···
On Tue, Sep 11, 2012 at 10:09 AM, Hannan Khan <hannank@gmail.com> wrote:
I might found an alternative solution. Yesterday we run data mart for 9
month and it was completed successfully but when we run for 5 years, the
errors arrived.
Today and trying with again and get back to you all with the result. But
when we run datamart for 9 moth it takes 34 minutes, which is long is not
it?
Regards
Hannan
On Tue, Sep 11, 2012 at 2:01 PM, Morten Olav Hansen <mortenoh@gmail.com> > wrote:
That's a really big database... I think maybe Lars should answer if the
memory of the server is large enough (I'm sure he can compare it with what
he has been doing to tune the Kenya server).
--
Morten
On Tue, Sep 11, 2012 at 2:59 PM, Hannan Khan <hannank@gmail.com> wrote:
Hi Morten
Server have 8GB memory and database is 3159MB. We are using MySQL.
Regards
Hannan
On Tue, Sep 11, 2012 at 1:53 PM, Morten Olav Hansen <mortenoh@gmail.com> >>> wrote:
Hi
How much memory does your server have? and what is the size of your
database?
--
Morten
On Tue, Sep 11, 2012 at 2:51 PM, Hannan Khan <hannank@gmail.com> wrote:
-XX:MaxPermSize=2048m and -XX:PermSize=512m
On Tue, Sep 11, 2012 at 11:27 AM, Jason Pickering >>>>> <jason.p.pickering@gmail.com> wrote:
The PergGen space is the important option. What are you JAVA_OPTS.?
--
Sent from my mobile
On Sep 11, 2012 7:25 AM, "Hannan Khan" <hannank@gmail.com> wrote:
Thanks Jason.
I put the Java heap space 1GB and Maximum 6GB (Xmx6144m and
Xms1024m). But still getting 'java.lang.OutOfMemoryError: Java heap space'
error. That why I asked that whether that is a bug or not.
I am trying to other options and contact you soon.
Regards
Hannan
On Mon, Sep 10, 2012 at 11:40 AM, Jason Pickering >>>>>>> <jason.p.pickering@gmail.com> wrote:
Hi Hannan,
You configuration may have worked previously, but some things were
changed in the datamart and you will need to allocate more memory to
it after 2.7.
I am not really sure what your settings are, but you should at least
specify these options, of course adjusting them for the amount of
memory which can be allocated to DHIS2 .
Best regards,
Jason
On Mon, Sep 10, 2012 at 10:49 AM, Hannan Khan <hannank@gmail.com> >>>>>>>> wrote:
> Dear Jason
>
> Thanks for your advice. We are running dhis2 from 2010. In earlier
> version
> we dont have any problem creating datamart. After updating to
> version 2.7 we
> imported a big chuck of data from two data souce and after that we
> have
> problem creating datamart.
>
> So far I know the configuration is ok, however I will check again
> and get
> back to you.
>
> Regards
>
> Hannan
> from Samsung Galaxy
>
> On Sep 9, 2012 5:39 PM, "Jason Pickering" >>>>>>>> > <jason.p.pickering@gmail.com> >>>>>>>> > wrote:
>>
>> This is not a bug. You must increase the heap size. Please
>> consult the
>> implementation manual for details.
>>
>> Regards,
>> Jason
>>
>> --
>> Sent from my mobile
>>
>> On Sep 9, 2012 1:12 PM, "Hannan Khan" <hannank@gmail.com> wrote:
>>>
>>> We are using dhis2 version 2.7. Whenever we generating datamart
>>> are it
>>> taking long time and it never ends.The error message is (in the
>>> tomcat log
>>> file)
>>>
>>> 'dhis2-org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
>>> processChildren SEVERE: Exception invoking periodic operation:
>>> java.lang.OutOfMemoryError: Java heap space'.
>>>
>>> I attached the log file.
>>>
>>> Regards
>>>
>>> Hannan Khan
>>>
>>> National Data Warehouse
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: DHIS 2 developers in Launchpad
>>> Post to : dhis2-devs@lists.launchpad.net
>>> Unsubscribe : DHIS 2 developers in Launchpad
>>> More help : ListHelp - Launchpad Help
>>>
>
Actually I never use PostgreSQL. So I asked few of friends and they suggests me that if the application are tuned and used in MySQL then stay with that.
As you are suggesting to use PostgreSQL can you suggest how I could I shift database from MySQL to PostgreSQL?
Regards
Hannan
···
On Tue, Sep 11, 2012 at 4:29 PM, Knut Staring knutst@gmail.com wrote:
Perhaps this is a case where it would make sense to switch to
PostgreSQL and possibly also make use of more of the internal
crunching capabilities of the database?
Knut
On Tue, Sep 11, 2012 at 10:09 AM, Hannan Khan hannank@gmail.com wrote:
I might found an alternative solution. Yesterday we run data mart for 9
month and it was completed successfully but when we run for 5 years, the
errors arrived.
Today and trying with again and get back to you all with the result. But
when we run datamart for 9 moth it takes 34 minutes, which is long is not
it?
Regards
Hannan
On Tue, Sep 11, 2012 at 2:01 PM, Morten Olav Hansen mortenoh@gmail.com > > > wrote:
That’s a really big database… I think maybe Lars should answer if the
memory of the server is large enough (I’m sure he can compare it with what
he has been doing to tune the Kenya server).
–
Morten
On Tue, Sep 11, 2012 at 2:59 PM, Hannan Khan hannank@gmail.com wrote:
Hi Morten
Server have 8GB memory and database is 3159MB. We are using MySQL.
Regards
Hannan
On Tue, Sep 11, 2012 at 1:53 PM, Morten Olav Hansen mortenoh@gmail.com > > >>> wrote:
Hi
How much memory does your server have? and what is the size of your
database?
–
Morten
On Tue, Sep 11, 2012 at 2:51 PM, Hannan Khan hannank@gmail.com wrote:
I think Lars is probably the definitive source here, but let me offer my views.
I think this has nothing to do with the database (MySQL or Postgresql) but rather how DHIS2 does the aggregation. A big chunk of data will get pulled into memory. If the data is too big and the memory is not enough, this error will result. I have seen it many times. We have had similar issues with our database, when aggregating historical data in one chunk. So, instead, I simply do it year by year, piece by piece. This is fairly easy to automate with a few CURL calls. We are using Postgres, and have had similar issues, which have been solved by
Increasing the PermSpace parameter
Aggregate less data at a time.
Switching to Postgres may have some other advantage, but I do not think it will solve this issue. Properly tuned, it should work fine.
Regards,
Jason
···
On Tue, Sep 11, 2012 at 4:06 PM, Hannan Khan hannank@gmail.com wrote:
Dear Knut
Actually I never use PostgreSQL. So I asked few of friends and they suggests me that if the application are tuned and used in MySQL then stay with that.
As you are suggesting to use PostgreSQL can you suggest how I could I shift database from MySQL to PostgreSQL?
Regards
Hannan
On Tue, Sep 11, 2012 at 4:29 PM, Knut Staring knutst@gmail.com wrote:
Perhaps this is a case where it would make sense to switch to
PostgreSQL and possibly also make use of more of the internal
crunching capabilities of the database?
Knut
On Tue, Sep 11, 2012 at 10:09 AM, Hannan Khan hannank@gmail.com wrote:
I might found an alternative solution. Yesterday we run data mart for 9
month and it was completed successfully but when we run for 5 years, the
errors arrived.
Today and trying with again and get back to you all with the result. But
when we run datamart for 9 moth it takes 34 minutes, which is long is not
it?
Regards
Hannan
On Tue, Sep 11, 2012 at 2:01 PM, Morten Olav Hansen mortenoh@gmail.com > > > > > wrote:
That’s a really big database… I think maybe Lars should answer if the
memory of the server is large enough (I’m sure he can compare it with what
he has been doing to tune the Kenya server).
–
Morten
On Tue, Sep 11, 2012 at 2:59 PM, Hannan Khan hannank@gmail.com wrote:
Hi Morten
Server have 8GB memory and database is 3159MB. We are using MySQL.
Regards
Hannan
On Tue, Sep 11, 2012 at 1:53 PM, Morten Olav Hansen mortenoh@gmail.com > > > > >>> wrote:
Hi
How much memory does your server have? and what is the size of your
database?
–
Morten
On Tue, Sep 11, 2012 at 2:51 PM, Hannan Khan hannank@gmail.com wrote:
Jason is right, no need to change from mysql. It is also OK to
aggregate a year at the time to reduce memory usage, hopefully you
don't need to do this too often. Can you please:
1. Paste the contents of the Help -> About page from Java opts and below.
2. Paste the number of data values you have in your database from Data
administration -> Data statistics.
It is also a point not to raise the value of Xmx to more than what is
actually available to the JVM, it might trick Java to put more objects
into memory than what is possible, leading to poorer performance/out
of mem errors. So if you have assigned a lot of memory to mysql or
other applications running on the server you might need to lower it a
bit.
I don't think you want to waste too much memory on permgen space when
what you really need as much heap as you can get. Just enough. I
find 256M actually suffices.
So you probably want something like JAVA_OPTS='-Xmx4096m -XX:MaxPermSize=256m'
This might be a bit tight, as Lars says, you may not have that 4096m
physically available if 3G has gone to mysql. I think I would try and
trim a bit from mysql rather than trim dhis2 much below this. 2G to
mysql should be adequate.
If its not, think about running dhis and tomcat on separate machines.
If you have gigabit ethernet between them, the performance cost of
transport is not too great and you can tune each one optimally to the
available resources.
Bob
···
On 11 September 2012 13:22, Lars Helge Øverland <larshelge@gmail.com> wrote:
Hi,
Jason is right, no need to change from mysql. It is also OK to
aggregate a year at the time to reduce memory usage, hopefully you
don't need to do this too often. Can you please:
1. Paste the contents of the Help -> About page from Java opts and below.
2. Paste the number of data values you have in your database from Data
administration -> Data statistics.
It is also a point not to raise the value of Xmx to more than what is
actually available to the JVM, it might trick Java to put more objects
into memory than what is possible, leading to poorer performance/out
of mem errors. So if you have assigned a lot of memory to mysql or
other applications running on the server you might need to lower it a
bit.
I actually put 6GB for Tomcat and and 2 GB for MySQL. My Tomcat and MySQL are in same server. My JAVA_OPTS is -Xms1024m -Xmx6144m -XX:MaxPermSize=2048m -XX:PermSize=512m.
We completed our testing and find that for our data volume running datamart for one year and below is ok. If we run for two year and above it have the same error.
Also note that there is message in the DHIS2 front end; it is run for ever and not completed the process. From the tomcat log I find that there is an error of Java heap size. So my suggestion is that to convey the error message through front so that the user know immediately that there is an error in this module, so he can continue with other works not waiting for the administrator to check the log.
For your information I configured Director General Health Service (DGHS) servers (another support to DHIS2nin the health sector) with Tomcat and MySQL in two different server and it is comfortably.The Server running Tomcat having two Gigabit network card; one with public IP for accessing DHIS2 and other card having local IP to accessing MySQL. The server having MySQL have also a Gigabit card having local IP.
I actually put 6GB for Tomcat and and 2 GB for MySQL. My Tomcat and MySQL
are in same server. My JAVA_OPTS is -Xms1024m -Xmx6144m
-XX:MaxPermSize=2048m -XX:PermSize=512m.
We completed our testing and find that for our data volume running datamart
for one year and below is ok. If we run for two year and above it have the
same error.
Bear in mind Lars advice about not exceeding the available physical
memory. It might well be that you would need more RAM to aggregate
this much data. Lars would have the best idea .. you didn't say how
many data values you are talking about?
Also note that there is message in the DHIS2 front end; it is run for ever
and not completed the process. From the tomcat log I find that there is an
error of Java heap size. So my suggestion is that to convey the error
message through front so that the user know immediately that there is an
error in this module, so he can continue with other works not waiting for
the administrator to check the log.
Do you mean there is no message?
Agree with the ideal to convey this somehow through the application.
Either by providing a log viewer, or perhaps even the messaging
service to alert critical events. Not sure of the best approach here,
but maybe others have ideas. There are quite a few possibilities.
Eg. a straightforward (sounding) hack that comes to mind is to have a
shell script which "tails" the log file continuously and pumps a
message through the web api to admin user for any uncaught exceptions.
Probably even logwatch could be drafted into this role, though
processing java multi-line log lines is a challenge.
There's also a communications design pattern related to logging
(reference too far back in my memory) about not making a bad situation
worse - if the system starts spewing exceptions because it is
overloaded in some way, overloading it even more by generating lots of
messages is not a good response. So one needs to be a bit smart ..
For your information I configured Director General Health Service (DGHS)
servers (another support to DHIS2nin the health sector) with Tomcat and
MySQL in two different server and it is comfortably.The Server running
Tomcat having two Gigabit network card; one with public IP for accessing
DHIS2 and other card having local IP to accessing MySQL. The server having
MySQL have also a Gigabit card having local IP.
I think there are many merits to this approach if you have the
hardware available.
Cheers
Bob
···
On 12 September 2012 07:45, Hannan Khan <hannank@gmail.com> wrote:
Regards
Hannan
On Tue, Sep 11, 2012 at 7:34 PM, Bob Jolliffe <bobjolliffe@gmail.com> wrote:
A few extra thoughts on java opts:
I don't think you want to waste too much memory on permgen space when
what you really need as much heap as you can get. Just enough. I
find 256M actually suffices.
So you probably want something like JAVA_OPTS='-Xmx4096m
-XX:MaxPermSize=256m'
This might be a bit tight, as Lars says, you may not have that 4096m
physically available if 3G has gone to mysql. I think I would try and
trim a bit from mysql rather than trim dhis2 much below this. 2G to
mysql should be adequate.
If its not, think about running dhis and tomcat on separate machines.
If you have gigabit ethernet between them, the performance cost of
transport is not too great and you can tune each one optimally to the
available resources.
Bob
On 11 September 2012 13:22, Lars Helge Øverland <larshelge@gmail.com> >> wrote:
> Hi,
>
> Jason is right, no need to change from mysql. It is also OK to
> aggregate a year at the time to reduce memory usage, hopefully you
> don't need to do this too often. Can you please:
>
> 1. Paste the contents of the Help -> About page from Java opts and
> below.
>
> 2. Paste the number of data values you have in your database from Data
> administration -> Data statistics.
>
> It is also a point not to raise the value of Xmx to more than what is
> actually available to the JVM, it might trick Java to put more objects
> into memory than what is possible, leading to poorer performance/out
> of mem errors. So if you have assigned a lot of memory to mysql or
> other applications running on the server you might need to lower it a
> bit.
>
> Lars
>
> _______________________________________________
> Mailing list: DHIS 2 developers in Launchpad
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : DHIS 2 developers in Launchpad
> More help : ListHelp - Launchpad Help