running 11 dhis2 webapps on one tomcat

bobj · 7 April 2009 07:47

Hi John

How are you and Saptar getting on with this challenge?

I did a little experiment this morning. The dhis.war is about 35M of
mostly compressed application and framework classes which is one of
the reasons why it is a herculean effort for tomcat to try and load it
multiple times into memory.

If you pull out all the WEB-INF/lib/*.jar files from the war and place
those into tomcat's lib directory you are left with a dhis.war which
is only 1.8M. (I did it on jetty, but same should apply to tomcat).
Everything continues to run as before and I suspect you will find that
you can now load gazillions of dhis2 instances without exhausting the
PermSize. Let me know how this works.

Of course the downside of de-packaging is that you will hit serious
complications if you are running different dhis2 versions which might
make use of different versions of libraries. It will be a case of
upgrade one - upgrade all.

Cheers
Bob

Johnlewis_Hisp · 7 April 2009 07:59

Bob, to run different application in the same tomcat we need to change the hibernate property file which is located in WEB-INF/lib/dhis-support-hibernate-2.0.1-SNAPSHOT.jar. IF we are moving out web-INF/lib folder then we are moving even the hibernate file also. Dont know we can solve this.

John

···

On Tue, Apr 7, 2009 at 1:17 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Hi John

How are you and Saptar getting on with this challenge?

I did a little experiment this morning. The dhis.war is about 35M of

mostly compressed application and framework classes which is one of

the reasons why it is a herculean effort for tomcat to try and load it

multiple times into memory.

If you pull out all the WEB-INF/lib/*.jar files from the war and place

those into tomcat’s lib directory you are left with a dhis.war which

is only 1.8M. (I did it on jetty, but same should apply to tomcat).

Everything continues to run as before and I suspect you will find that

you can now load gazillions of dhis2 instances without exhausting the

PermSize. Let me know how this works.

Of course the downside of de-packaging is that you will hit serious

complications if you are running different dhis2 versions which might

make use of different versions of libraries. It will be a case of

upgrade one - upgrade all.

Cheers

Bob

Knut_Staring · 7 April 2009 08:04

Bob, to run different application in the same tomcat we need to change the hibernate property file which is located in WEB-INF/lib/dhis-support-hibernate-2.0.1-SNAPSHOT.jar. IF we are moving out web-INF/lib folder then we are moving even the hibernate file also. Dont know we can solve this.

I think you should not move dhis-support-hibernate-2.0.1-SNAPSHOT.jar, just everything else.

Knut

···

On Tue, Apr 7, 2009 at 9:59 AM, John lewis johnlewis.hisp@gmail.com wrote:

John

On Tue, Apr 7, 2009 at 1:17 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Hi John

How are you and Saptar getting on with this challenge?

I did a little experiment this morning. The dhis.war is about 35M of

mostly compressed application and framework classes which is one of

the reasons why it is a herculean effort for tomcat to try and load it

multiple times into memory.

If you pull out all the WEB-INF/lib/*.jar files from the war and place

those into tomcat’s lib directory you are left with a dhis.war which

is only 1.8M. (I did it on jetty, but same should apply to tomcat).

Everything continues to run as before and I suspect you will find that

you can now load gazillions of dhis2 instances without exhausting the

PermSize. Let me know how this works.

Of course the downside of de-packaging is that you will hit serious

complications if you are running different dhis2 versions which might

make use of different versions of libraries. It will be a case of

upgrade one - upgrade all.

Cheers

Bob

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–
Cheers,
Knut Staring

bobj · 7 April 2009 08:24

Knut is correct. This should work. I'm really interested to hear.

BTW on tomcat I believe you should be putting the other jars in
shared/lib. Good luck.

Cheers
Bob

···

2009/4/7 Knut Staring <knutst@gmail.com>:

On Tue, Apr 7, 2009 at 9:59 AM, John lewis <johnlewis.hisp@gmail.com> wrote:

Bob, to run different application in the same tomcat we need to change the
hibernate property file which is located in
WEB-INF/lib/dhis-support-hibernate-2.0.1-SNAPSHOT.jar. IF we are moving out
web-INF/lib folder then we are moving even the hibernate file also. Dont
know we can solve this.

I think you should not move dhis-support-hibernate-2.0.1-SNAPSHOT.jar, just
everything else.

Knut

John

On Tue, Apr 7, 2009 at 1:17 PM, Bob Jolliffe <bobjolliffe@gmail.com> >> wrote:

Hi John

How are you and Saptar getting on with this challenge?

I did a little experiment this morning. The dhis.war is about 35M of
mostly compressed application and framework classes which is one of
the reasons why it is a herculean effort for tomcat to try and load it
multiple times into memory.
If you pull out all the WEB-INF/lib/*.jar files from the war and place
those into tomcat's lib directory you are left with a dhis.war which
is only 1.8M. (I did it on jetty, but same should apply to tomcat).
Everything continues to run as before and I suspect you will find that
you can now load gazillions of dhis2 instances without exhausting the
PermSize. Let me know how this works.

Of course the downside of de-packaging is that you will hit serious
complications if you are running different dhis2 versions which might
make use of different versions of libraries. It will be a case of
upgrade one - upgrade all.

Cheers
Bob

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

--
Cheers,
Knut Staring

bobj · 7 April 2009 09:01

Just another thought.

DHIS2 is now also configured to pick up a system property (dhis2.home)
in the absence of a DHIS2_HOME environment variable. Lars added this
feature so that I can deploy dhis2-lite without getting users to
fiddle with environment variables.

If you are deploying multiple instances of dhis2 it would be much
tidier to handcraft a context for each one in context.xml, in which
you set a dhis2.home context parameter appropriately. As well as
other useful goodies like the webapp path. Then you don't have to go
poking around at the hibernate.properties inside the jar files which
is pretty ugly. Mind you I don't think DHIS2 is currently configured
to look for context parameters but it is probably a simple
modification.

Another side effect of this approach might be that you actually only
need to deploy one war, and the individual contexts would each specify
that war together with a unique path. Something like (dreaming out
loud):

Or maybe even (taking a slightly different approach):

In this latter case the application should be able to pick up the values with:
try
{
InitialContext context = new InitialContext();
Object valueO = context.lookup("java:comp/env/dhis2.home");
}
catch(javax.naming.NamingException ne)
{ /* handle gracefully */ }

Just a thought.

Meanwhile proceed with what you are doing. I think it should work.

Cheers
Bob

···

2009/4/7 Bob Jolliffe <bobjolliffe@gmail.com>:

Knut is correct. This should work. I'm really interested to hear.

BTW on tomcat I believe you should be putting the other jars in
shared/lib. Good luck.

Cheers
Bob

2009/4/7 Knut Staring <knutst@gmail.com>:

On Tue, Apr 7, 2009 at 9:59 AM, John lewis <johnlewis.hisp@gmail.com> wrote:

Bob, to run different application in the same tomcat we need to change the
hibernate property file which is located in
WEB-INF/lib/dhis-support-hibernate-2.0.1-SNAPSHOT.jar. IF we are moving out
web-INF/lib folder then we are moving even the hibernate file also. Dont
know we can solve this.

I think you should not move dhis-support-hibernate-2.0.1-SNAPSHOT.jar, just
everything else.

Knut

John

On Tue, Apr 7, 2009 at 1:17 PM, Bob Jolliffe <bobjolliffe@gmail.com> >>> wrote:

Hi John

How are you and Saptar getting on with this challenge?

I did a little experiment this morning. The dhis.war is about 35M of
mostly compressed application and framework classes which is one of
the reasons why it is a herculean effort for tomcat to try and load it
multiple times into memory.
If you pull out all the WEB-INF/lib/*.jar files from the war and place
those into tomcat's lib directory you are left with a dhis.war which
is only 1.8M. (I did it on jetty, but same should apply to tomcat).
Everything continues to run as before and I suspect you will find that
you can now load gazillions of dhis2 instances without exhausting the
PermSize. Let me know how this works.

Of course the downside of de-packaging is that you will hit serious
complications if you are running different dhis2 versions which might
make use of different versions of libraries. It will be a case of
upgrade one - upgrade all.

Cheers
Bob

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

--
Cheers,
Knut Staring

Lars · 7 April 2009 12:35

These are really good ideas. Will try to think of a way to automate this separation.

···

On Tue, Apr 7, 2009 at 11:01 AM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Just another thought.

DHIS2 is now also configured to pick up a system property (dhis2.home)

in the absence of a DHIS2_HOME environment variable. Lars added this

feature so that I can deploy dhis2-lite without getting users to

fiddle with environment variables.

If you are deploying multiple instances of dhis2 it would be much

tidier to handcraft a context for each one in context.xml, in which

you set a dhis2.home context parameter appropriately. As well as

other useful goodies like the webapp path. Then you don’t have to go

poking around at the hibernate.properties inside the jar files which

is pretty ugly. Mind you I don’t think DHIS2 is currently configured

to look for context parameters but it is probably a simple

modification.

Another side effect of this approach might be that you actually only

need to deploy one war, and the individual contexts would each specify

that war together with a unique path. Something like (dreaming out

loud):

Or maybe even (taking a slightly different approach):

<Environment name=“dhis2.home” value=“/etc/dhis2/state1”

type=“java.lang.String” />

<Environment name=“dhis2.home” value=“/etc/dhis2/state2”

type=“java.lang.String” />

<Environment name=“dhis2.home” value=“/etc/dhis2/state3”

type=“java.lang.String” />

In this latter case the application should be able to pick up the values with:

try

{

InitialContext context = new InitialContext();

Object valueO = context.lookup(“java:comp/env/dhis2.home”);

}

catch(javax.naming.NamingException ne)

{ /* handle gracefully */ }

Just a thought.

Meanwhile proceed with what you are doing. I think it should work.

Cheers

Bob

2009/4/7 Bob Jolliffe bobjolliffe@gmail.com:

Knut is correct. This should work. I’m really interested to hear.

BTW on tomcat I believe you should be putting the other jars in

shared/lib. Good luck.

Cheers

Bob

bobj · 7 April 2009 13:53

I suspect this is a bit of a fringe case - I don't know too many folk
who might want to run 11 dhis instances on one server. But you never
know ...

Personally I like idea of being able to run identical wars for each
instance (rather than poking around the embedded hibernate
properties). In a really large setup - like they might end up having
in India - it might then be possible to imagine a pool of application
servers (glassfish or tomcat) and a pool of contexts. Contexts could
then be migrated fairly seemlessly and transparently between the
servers to optimize load balancing.

Of course this would be dependent on the web portal initialisation
picking up a context parameter and passing this on to hibernate in its
initialisation. I'll have a look at how this can be done. I know you
would do it quicker, but its a good exercise for me.

Tailoring DHIS2 for scalability from the lite to the massive ...

···

2009/4/7 Lars Helge Øverland <larshelge@gmail.com>:

On Tue, Apr 7, 2009 at 11:01 AM, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

Just another thought.

DHIS2 is now also configured to pick up a system property (dhis2.home)
in the absence of a DHIS2_HOME environment variable. Lars added this
feature so that I can deploy dhis2-lite without getting users to
fiddle with environment variables.

If you are deploying multiple instances of dhis2 it would be much
tidier to handcraft a context for each one in context.xml, in which
you set a dhis2.home context parameter appropriately. As well as
other useful goodies like the webapp path. Then you don't have to go
poking around at the hibernate.properties inside the jar files which
is pretty ugly. Mind you I don't think DHIS2 is currently configured
to look for context parameters but it is probably a simple
modification.

Another side effect of this approach might be that you actually only
need to deploy one war, and the individual contexts would each specify
that war together with a unique path. Something like (dreaming out
loud):

<Context path="/state1" docBase="dhis2.war">
<Parameter name="dhis2.home" value="/etc/dhis2/state1" />
</Context>
<Context path="/state2" docBase="dhis2.war">
<Parameter name="dhis2.home" value="/etc/dhis2/state2" />
</Context>
<Context path="/state3" docBase="dhis2.war">
<Parameter name="dhis2.home" value="/etc/dhis2/state3" />
</Context>

Or maybe even (taking a slightly different approach):

<Context path="/state1" docBase="dhis2.war">
<Environment name="dhis2.home" value="/etc/dhis2/state1"
type="java.lang.String" />
</Context>
<Context path="/state2" docBase="dhis2.war">
<Environment name="dhis2.home" value="/etc/dhis2/state2"
type="java.lang.String" />
</Context>
<Context path="/state3" docBase="dhis2.war">
<Environment name="dhis2.home" value="/etc/dhis2/state3"
type="java.lang.String" />
</Context>

In this latter case the application should be able to pick up the values
with:
try
{
InitialContext context = new InitialContext();
Object valueO = context.lookup("java:comp/env/dhis2.home");
}
catch(javax.naming.NamingException ne)
{ /* handle gracefully */ }

Just a thought.

Meanwhile proceed with what you are doing. I think it should work.

Cheers
Bob

2009/4/7 Bob Jolliffe <bobjolliffe@gmail.com>:
> Knut is correct. This should work. I'm really interested to hear.
>
> BTW on tomcat I believe you should be putting the other jars in
> shared/lib. Good luck.
>
> Cheers
> Bob
>

These are really good ideas. Will try to think of a way to automate this
separation.

Knut_Staring · 7 April 2009 14:06

I suspect this is a bit of a fringe case - I don't know too many folk
who might want to run 11 dhis instances on one server.

Not at all. Though 11 is a bit high, we have the same issue on our
"demo" server, and the need for such setups will just increase, I
think. On the other hand, running one instance on each of a bunch of
virtual machines on Amazon's EC2 cloud may be a better option from an
administrative point of view.

Personally I like idea of being able to run identical wars for each
instance (rather than poking around the embedded hibernate
properties). In a really large setup - like they might end up having
in India - it might then be possible to imagine a pool of application
servers (glassfish or tomcat) and a pool of contexts. Contexts could
then be migrated fairly seemlessly and transparently between the
servers to optimize load balancing.

Yes, I think this is what is needed (though the cloud is an alternative).

Of course this would be dependent on the web portal initialisation
picking up a context parameter and passing this on to hibernate in its
initialisation. I'll have a look at how this can be done. I know you
would do it quicker, but its a good exercise for me.

Tailoring DHIS2 for scalability from the lite to the massive ...

Indeed

Knut

···

On 4/7/09, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

2009/4/7 Lars Helge Øverland <larshelge@gmail.com>:

--
Cheers,
Knut Staring

Knut_Staring · 10 April 2009 10:22

For some reason, it seems it is not as straightforward as that with tomcat - if I move some .jar files from WEB-INF/lib Tomcat doesn’t seem to find them anymore.

Knut

···

On Tue, Apr 7, 2009 at 9:47 AM, Bob Jolliffe bobjolliffe@gmail.com wrote:

If you pull out all the WEB-INF/lib/*.jar files from the war and place

those into tomcat’s lib directory you are left with a dhis.war which

is only 1.8M. (I did it on jetty, but same should apply to tomcat).

bobj · 10 April 2009 11:02

On tomcat I put ALL jars in shared/lib. And I have this at the bottom
of my server.xml:

And yes, if I uncomment, I can start all 20 of these instances on my
2M machine without any particular stress on my available Perm space.
They are all sharing the same jar so the difference between 1 and 20
is pretty marginal. It does take a LONG time to load them all up
though. Caveats:
1. I'm not saying this is a good idea! Just a proof of concept. One
badly behaved app could bring down all the others.
2. The big issue now becomes heap usage. One of the areas we need to
think about optimizing some of the DHIS2 routines is in the use of
heap. I haven't looked in detail yet, but for example, the reason why
we have sometimes had memory issues with import is that we seem to be
effectively pulling all the data into the heap before flushing the
hibernate cache. This can be kind of a waste seeing that we are using
a SAX parser for the very reason that we might scale to very large
input streams. Probably we can look at ways of chunking these sort of
operations to increase scalability. In fact I'd be very surprised if
the Hibernate cache manager isn't smart enough to do this chunking
already. Either way this kind of issue is solvable.
3. Obviously, with the above described arrangement, my database
connections are all messed up. Still have to sort out the dhis2_home
context init_parameter.

Cheers
Bob

···

2009/4/10 Knut Staring <knutst@gmail.com>:

On Tue, Apr 7, 2009 at 9:47 AM, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

If you pull out all the WEB-INF/lib/*.jar files from the war and place
those into tomcat's lib directory you are left with a dhis.war which
is only 1.8M. (I did it on jetty, but same should apply to tomcat).

For some reason, it seems it is not as straightforward as that with tomcat -
if I move some .jar files from WEB-INF/lib Tomcat doesn't seem to find them
anymore.
Knut

sunbiz · 11 April 2009 18:10

From all the experiments on the state servers and our office server, we have learnt a few lessons, some of which may be useful to other implementers and developers:

We have started different tomcats on different ports and each tomcat uses 1GB max memory. The advantage of such a setup is that different state applications can be started/stopped/different memory allocations/threadpools, etc and one state application is not affecting the other state application… The disadvantage is that the port number in the URL can be discomforting to the user. So we created a domain and then each state has a subdomain mapped to the application. i.e. hp.nhsrc-hmis.org or bihar.nhsrc-hmis.org… We can play and move the state apps off servers/tomcats and no one realizes that we moved. Can be used to move apps when we know a load may be coming.

In our office I was able to deploy 14 state apps on 6GB max memory to one tomcat … but only when one app was deployed at a time from the manager application. or else the c3p0 bean would suggest a memory leak and the JDBC connection to MySQL would just get lost. This means whenever after 14 apps were deployed and tomcat was restarted, the apps would fail to deploy because all would deploy together.

To find what caused the error, I tried to do a profiler run and found out some points of optimization. Memory areas where hibernate sessions were active went through frequent paging… something of a bottleneck of sorts. The c3p0 bean was creating Datasource pools even when we were not using them and probably would only use later after the application was fully running and doing something useful. Being able to remove hibernate and its mates altogether can make the application very much lighter…

That brings me to an empirical question: Can JPA solve some of these memory issues?? Since it has learnt lessons from all the ORMs including hibernate and improvised a lot on EJB… I talked to abyot that we may want to try JPA in the CBS project and try to find an answer to that question.

BTW, thanks bob for the suggestion. I tried to put the libs out to tomcat, but they only work when all are put… I couldn’t make tomcat selective. That means we may have translations lost or all apps using the same language, since they are in the jars. Other than that, one loaded app can steal other apps resources… and when there is load on multiple apps running in this configuration, it still gives the JDBC connection lost exception… only difference is not during deployment, but when using the apps, which is more dangerous.

···

Regards,
Saptarshi PURKAYASTHA
Director R & D, HISP India
Health Information Systems Programme

My Tech Blog: http://sunnytalkstech.blogspot.com

You Live by CHOICE, Not by CHANCE

2009/4/10 Bob Jolliffe bobjolliffe@gmail.com

On tomcat I put ALL jars in shared/lib. And I have this at the bottom

of my server.xml:
    <Context path= "/state1" docBase="/home/bobj/dhis2/dhis.war"></Context>
  </Host>

</Engine>
And yes, if I uncomment, I can start all 20 of these instances on my

2M machine without any particular stress on my available Perm space.

They are all sharing the same jar so the difference between 1 and 20

is pretty marginal. It does take a LONG time to load them all up

though. Caveats:

I’m not saying this is a good idea! Just a proof of concept. One

badly behaved app could bring down all the others.

The big issue now becomes heap usage. One of the areas we need to

think about optimizing some of the DHIS2 routines is in the use of

heap. I haven’t looked in detail yet, but for example, the reason why

we have sometimes had memory issues with import is that we seem to be

effectively pulling all the data into the heap before flushing the

hibernate cache. This can be kind of a waste seeing that we are using

a SAX parser for the very reason that we might scale to very large

input streams. Probably we can look at ways of chunking these sort of

operations to increase scalability. In fact I’d be very surprised if

the Hibernate cache manager isn’t smart enough to do this chunking

already. Either way this kind of issue is solvable.

Obviously, with the above described arrangement, my database

connections are all messed up. Still have to sort out the dhis2_home

context init_parameter.

Cheers

Bob

2009/4/10 Knut Staring knutst@gmail.com:

On Tue, Apr 7, 2009 at 9:47 AM, Bob Jolliffe bobjolliffe@gmail.com wrote:

If you pull out all the WEB-INF/lib/*.jar files from the war and place

those into tomcat’s lib directory you are left with a dhis.war which

is only 1.8M. (I did it on jetty, but same should apply to tomcat).

For some reason, it seems it is not as straightforward as that with tomcat -

if I move some .jar files from WEB-INF/lib Tomcat doesn’t seem to find them

anymore.

Knut

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

sunbiz · 11 April 2009 18:14

Sorry for missing this out, I meant JPA+TopLink instead of hibernate in the previous email

···

Regards,
Saptarshi PURKAYASTHA
Director R & D, HISP India
Health Information Systems Programme

My Tech Blog: http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE

2009/4/11 Saptarshi Purkayastha sunbiz@gmail.com

From all the experiments on the state servers and our office server, we have learnt a few lessons, some of which may be useful to other implementers and developers:

We have started different tomcats on different ports and each tomcat uses 1GB max memory. The advantage of such a setup is that different state applications can be started/stopped/different memory allocations/threadpools, etc and one state application is not affecting the other state application… The disadvantage is that the port number in the URL can be discomforting to the user. So we created a domain and then each state has a subdomain mapped to the application. i.e. hp.nhsrc-hmis.org or bihar.nhsrc-hmis.org… We can play and move the state apps off servers/tomcats and no one realizes that we moved. Can be used to move apps when we know a load may be coming.

In our office I was able to deploy 14 state apps on 6GB max memory to one tomcat … but only when one app was deployed at a time from the manager application. or else the c3p0 bean would suggest a memory leak and the JDBC connection to MySQL would just get lost. This means whenever after 14 apps were deployed and tomcat was restarted, the apps would fail to deploy because all would deploy together.

To find what caused the error, I tried to do a profiler run and found out some points of optimization. Memory areas where hibernate sessions were active went through frequent paging… something of a bottleneck of sorts. The c3p0 bean was creating Datasource pools even when we were not using them and probably would only use later after the application was fully running and doing something useful. Being able to remove hibernate and its mates altogether can make the application very much lighter…

That brings me to an empirical question: Can JPA solve some of these memory issues?? Since it has learnt lessons from all the ORMs including hibernate and improvised a lot on EJB… I talked to abyot that we may want to try JPA in the CBS project and try to find an answer to that question.

BTW, thanks bob for the suggestion. I tried to put the libs out to tomcat, but they only work when all are put… I couldn’t make tomcat selective. That means we may have translations lost or all apps using the same language, since they are in the jars. Other than that, one loaded app can steal other apps resources… and when there is load on multiple apps running in this configuration, it still gives the JDBC connection lost exception… only difference is not during deployment, but when using the apps, which is more dangerous.

Regards,
Saptarshi PURKAYASTHA
Director R & D, HISP India
Health Information Systems Programme

My Tech Blog: http://sunnytalkstech.blogspot.com

You Live by CHOICE, Not by CHANCE

2009/4/10 Bob Jolliffe bobjolliffe@gmail.com
On tomcat I put ALL jars in shared/lib. And I have this at the bottom

of my server.xml:
    <Context path= "/state1" docBase="/home/bobj/dhis2/dhis.war"></Context>
  </Host>

</Engine>
And yes, if I uncomment, I can start all 20 of these instances on my

2M machine without any particular stress on my available Perm space.

They are all sharing the same jar so the difference between 1 and 20

is pretty marginal. It does take a LONG time to load them all up

though. Caveats:

I’m not saying this is a good idea! Just a proof of concept. One

badly behaved app could bring down all the others.

The big issue now becomes heap usage. One of the areas we need to

think about optimizing some of the DHIS2 routines is in the use of

heap. I haven’t looked in detail yet, but for example, the reason why

we have sometimes had memory issues with import is that we seem to be

effectively pulling all the data into the heap before flushing the

hibernate cache. This can be kind of a waste seeing that we are using

a SAX parser for the very reason that we might scale to very large

input streams. Probably we can look at ways of chunking these sort of

operations to increase scalability. In fact I’d be very surprised if

the Hibernate cache manager isn’t smart enough to do this chunking

already. Either way this kind of issue is solvable.

Obviously, with the above described arrangement, my database

connections are all messed up. Still have to sort out the dhis2_home

context init_parameter.

Cheers

Bob

2009/4/10 Knut Staring knutst@gmail.com:

On Tue, Apr 7, 2009 at 9:47 AM, Bob Jolliffe bobjolliffe@gmail.com wrote:

If you pull out all the WEB-INF/lib/*.jar files from the war and place

those into tomcat’s lib directory you are left with a dhis.war which

is only 1.8M. (I did it on jetty, but same should apply to tomcat).

For some reason, it seems it is not as straightforward as that with tomcat -

if I move some .jar files from WEB-INF/lib Tomcat doesn’t seem to find them

anymore.

Knut

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

bobj · 12 April 2009 10:50

Hi Saptarshi

From all the experiments on the state servers and our office server, we have
learnt a few lessons, some of which may be useful to other implementers and
developers:

We have started different tomcats on different ports and each tomcat uses
1GB max memory. The advantage of such a setup is that different state
applications can be started/stopped/different memory
allocations/threadpools, etc and one state application is not affecting the
other state application... The disadvantage is that the port number in the
URL can be discomforting to the user. So we created a domain and then each
state has a subdomain mapped to the application. i.e. hp.nhsrc-hmis.org or
bihar.nhsrc-hmis.org... We can play and move the state apps off
servers/tomcats and no one realizes that we moved. Can be used to move apps
when we know a load may be coming.

In our office I was able to deploy 14 state apps on 6GB max memory to one
tomcat ... but only when one app was deployed at a time from the manager
application. or else the c3p0 bean would suggest a memory leak and the JDBC
connection to MySQL would just get lost. This means whenever after 14 apps
were deployed and tomcat was restarted, the apps would fail to deploy
because all would deploy together.

To find what caused the error, I tried to do a profiler run and found out
some points of optimization. Memory areas where hibernate sessions were
active went through frequent paging... something of a bottleneck of sorts.
The c3p0 bean was creating Datasource pools even when we were not using them
and probably would only use later after the application was fully running
and doing something useful. Being able to remove hibernate and its mates
altogether can make the application very much lighter...

That brings me to an empirical question: Can JPA solve some of these memory
issues?? Since it has learnt lessons from all the ORMs including hibernate
and improvised a lot on EJB... I talked to abyot that we may want to try JPA
in the CBS project and try to find an answer to that question.

I think finding where the optimizations are with hibernate and fixing
them is a more realistic short term strategy than migrating away from
hibernate. That's a pretty major move. Having said that I see we do
have some ibatis code in dhis as well. Probably there is a historical
story here.

Besides optimizing the existing hibernate code, I think the correct
way to go is to start getting more rigorous about defining and using
functionality through the services/API/model layer. There are a
couple of places where we may be bypassing altogether, and still
others where we have code in the dhis-web modules which needs to be
pushed down into the model. Then we should certainly look at
optimizing the model - at some stage it might even include swapping
out the hibernate layer, but ideally only when we know this can be
done transparent to the layers above. I've been looking quite a bit
at the hibernate related stuff and there is quite a bit of invested
work in here (I think Lars would know best). I doubt if folk would
easily walk away from that.

BTW, thanks bob for the suggestion. I tried to put the libs out to tomcat,
but they only work when all are put... I couldn't make tomcat selective.
That means we may have translations lost or all apps using the same
language, since they are in the jars. Other than that, one loaded app can
steal other apps resources... and when there is load on multiple apps
running in this configuration, it still gives the JDBC connection lost
exception... only difference is not during deployment, but when using the
apps, which is more dangerous.

In the short term I agree you cannot do it this way. There are too
many issues to resolve. Longer term I think it would be good to
resolve all of the above. Your working arrangement with the multiple
tomcat instances has some serious benefits regarding isolation, but is
less than optimal regarding resource allocation. I guess there is a
balance to be struck - you can get even better isolation with
virtualisation but the costs keep rising ...

Are you running https?

Cheers
Bob

···

2009/4/11 Saptarshi Purkayastha <sunbiz@gmail.com>:

---
Regards,
Saptarshi PURKAYASTHA
Director R & D, HISP India
Health Information Systems Programme

My Tech Blog: http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE

2009/4/10 Bob Jolliffe <bobjolliffe@gmail.com>
On tomcat I put ALL jars in shared/lib. And I have this at the bottom
of my server.xml:
   &lt;Context path= &quot;/state1&quot;
docBase="/home/bobj/dhis2/dhis.war"></Context>



















</Host>
</Engine>
</Service>
</Server>

And yes, if I uncomment, I can start all 20 of these instances on my
2M machine without any particular stress on my available Perm space.
They are all sharing the same jar so the difference between 1 and 20
is pretty marginal. It does take a LONG time to load them all up
though. Caveats:
1. I'm not saying this is a good idea! Just a proof of concept. One
badly behaved app could bring down all the others.
2. The big issue now becomes heap usage. One of the areas we need to
think about optimizing some of the DHIS2 routines is in the use of
heap. I haven't looked in detail yet, but for example, the reason why
we have sometimes had memory issues with import is that we seem to be
effectively pulling all the data into the heap before flushing the
hibernate cache. This can be kind of a waste seeing that we are using
a SAX parser for the very reason that we might scale to very large
input streams. Probably we can look at ways of chunking these sort of
operations to increase scalability. In fact I'd be very surprised if
the Hibernate cache manager isn't smart enough to do this chunking
already. Either way this kind of issue is solvable.
3. Obviously, with the above described arrangement, my database
connections are all messed up. Still have to sort out the dhis2_home
context init_parameter.

Cheers
Bob

2009/4/10 Knut Staring <knutst@gmail.com>:
> On Tue, Apr 7, 2009 at 9:47 AM, Bob Jolliffe <bobjolliffe@gmail.com> >> > wrote:
>>
>> If you pull out all the WEB-INF/lib/*.jar files from the war and place
>> those into tomcat's lib directory you are left with a dhis.war which
>> is only 1.8M. (I did it on jetty, but same should apply to tomcat).
>
> For some reason, it seems it is not as straightforward as that with
> tomcat -
> if I move some .jar files from WEB-INF/lib Tomcat doesn't seem to find
> them
> anymore.
> Knut

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

Lars · 14 April 2009 13:05

A few comments:

I agree that we could define and implement the various use-cases more properly in the service layer, and see if more code could be moved from the web layer (I actually had a go at this last week for the reporting module). The way I see it this applies the most to the customized data entry part and the Indian local modules.
The Ibatis code is related to connecting to a MS Access database for the DHIS 1.4 file import.
We should definitely see if optimizing Hibernate could be done before reimplementing the persistence layer. The pooling can be tuned (even switched off), have a look here:

http://www.hibernate.org/214.html

Otherwise, we should maybe make it easier to switch persistence layer implementations by collecting all beans dealing with persistence in a separate maven project. Changing the persistence layer would then be a matter of modifying a dependency definition in a POM.

bobj · 14 April 2009 13:36

A few comments:

- I agree that we could define and implement the various use-cases more
properly in the service layer, and see if more code could be moved from the
web layer (I actually had a go at this last week for the reporting module).
The way I see it this applies the most to the customized data entry part and
the Indian local modules.

Agreed. There seems to be a bunch of code in dhis-web-commons
relating to ordering of data elements which might also be pushed down.
Lets open a new blueprint item to start collecting these cases. I
haven't looked much at the Indian local modules yet though I know we
will have to start picking at the mysql specific code soon if we are
to be able to deploy on h2.

- The Ibatis code is related to connecting to a MS Access database for the
DHIS 1.4 file import.

I thought this was the case. BTW if people are looking for
MSAccess-like functionality, I've been fiddling with my h2 databases
using OpenOffice Base and its pretty good. Check
Tutorial for simple
setup.

- We should definitely see if optimizing Hibernate could be done before
reimplementing the persistence layer. The pooling can be tuned (even
switched off), have a look here:

Page Redirection

Otherwise, we should maybe make it easier to switch persistence layer
implementations by collecting all beans dealing with persistence in a
separate maven project. Changing the persistence layer would then be a
matter of modifying a dependency definition in a POM.

This sounds like the right approach to me,

Bob

···

2009/4/14 Lars Helge Øverland <larshelge@gmail.com>:

Knut_Staring · 17 April 2009 07:33

Trying to folllow this, but having trouble connecting - what should be the URL for the DHIS lite database?

jdbc:h2:~/dhis2

And why does the wizard insist on creating a .odb file as well, when I just want to connect to an existing base?

Knut

···

2009/4/14 Bob Jolliffe bobjolliffe@gmail.com

BTW if people are looking for

MSAccess-like functionality, I’ve been fiddling with my h2 databases

using OpenOffice Base and its pretty good. Check

http://www.h2database.com/html/tutorial.html#open_office for simple

setup.

bobj · 17 April 2009 08:55

BTW if people are looking for
MSAccess-like functionality, I've been fiddling with my h2 databases
using OpenOffice Base and its pretty good. Check
Tutorial for simple
setup.

Trying to folllow this, but having trouble connecting - what should be the
URL for the DHIS lite database?

jdbc:h2:~/dhis2

Yes thats the right form (if thats where your database is - mine is
jdbc:h2:~/dhis2/db).
Driver = org.h2.Driver

And why does the wizard insist on creating a .odb file as well, when I just
want to connect to an existing base?

The .odb file contains all the other goodies - forms, reports,
queries. Kind of like access.

I find it quite useful - and pretty fast with h2. Of course, given
our current setup with embedded mode, you can't connect at the same
time as the dhis2 app is running. This may be a good thing. If not
you could revert to Lars' suggestion of 'mixed mode'.

Cheers
Bob

···

2009/4/17 Knut Staring <knutst@gmail.com>:

2009/4/14 Bob Jolliffe <bobjolliffe@gmail.com>

pmpdelcroix · 3 February 2023 10:51

I am not sure my answer will on the subject but since DHIS 2.38 context variable can be used

I have it running with 2 DHIS2, I just use different context file

pmpdelcroix · 3 February 2023 10:52

I just saw the post date …