scaling to many orgunits ...

Support - Assistance technique
1

Just sharing some thoughts before I lose them …

Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.

Our orgunit tree (the one-true-tree) is maintained using parentid pointers. This is simple enough to maintain and updates and insertions are efficient.

Of course updates and insertions are relatively rare. What we need to do much more frequently is selecting various subtrees. All the facilities in a district, all the districts in a province etc.

Also the depth of our trees are relatively shallow. Most places seem to have around 5 levels. There is some trend that this starts to increase, but not exponentially. We might conceive one day of 8 or even 10 levels but not 100s or 1000s of levels. Its this shallowness which makes the _orgunitstructure table viable as the number of columns in that table will always be within a practical limit…

Selecting for tree structures (traversal) which are built using ‘parentid’ is not very efficient. Postgres offers ‘WITH RECURSIVE’ which is pretty cool if you can get your head around it, but not supported in mysql (and not necessarily fast either).

So when selecting subtrees the _orgunitstructure table is our best friend. I use it for the mydatamart aggregatedXXvalue queries and it looks like Lars uses it in the scheduled datamart job as well as well. And it is the obvious table to use when implementing filtering as I am now looking at re mydatamart export.

The problem is (with this and the other resource tables) how to maintain integrity. Currently we generate this table on demand (from the user) from the parentid pointers. If users forget to do it then all sorts of things fail. There has been some discussion on list of generating this, at least nightly which would be an improvement. There would be some benefit in maintaining it dynamically, ie “triggered” during those relatively rare updates and insertions of new orgunits. One possible consequence of this approach would be that the parentid on the orgunit becomes effectively redundant. It could be argued that its only current use is to generate the _orgunitstructure table.

A consequence of the parentid being redundant is that hierarchy is maintained in a separate table to the orgunits themselves. And there is no reason why there should be only one _orgunitstructure table. There could be any number, limited only by the number of hierarchies we needed to maintain. There’s a thought …

Of course the other use for the parentid is when you are exporting a bundle of orgunits in a way which reflects the (or a) hierarchy. In which case it is really kind to clients to serialize this tree as a breadth-first traversal ie. they come out in order which is quick and easy to rebuild on the client. That is what I am looking at doing now to try and help out our mydatamart metadata export. And I will of course use the _orgunitstructure table to make this trivial on the server side.

But some principles derived from the above discussion:

  1. try to avoid using parentid directly in code … always link to the orgunitstructuretable. This might ease its eventual demise

  2. all places where we export collections of orgunits should be in a proper traversal - breadth first or depth firt shouldn’t matter really but I’m going to do the latter.

Bob

2

Hello Bob,

Wishing you and everyone on dhis2-dev list, a joyous Christmas season.

I am getting build failure in main trunk build with info as given below,

[INFO] -------------------------------------------------------------

[ERROR] COMPILATION ERROR :

[INFO] -------------------------------------------------------------

[ERROR] \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22] cannot find symbol

symbol : method compare(int,int)

location: class java.lang.Integer

[INFO] 1 error

Just sharing some thoughts before I lose them …

Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.

One day later after your this email, Morten Olav Hansen did a miner change

in one custom comparator ie. OrganisationUnitByLevelComparator.java .

in lp:dhis2 branch with revision number 9242. Before 9242, this comparator

was looks like this given below

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return o1.getLevel() - o2.getLevel();

}

}

Now, after 9242, it has been like

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return Integer.compare( o1.getOrganisationUnitLevel(), o2.getOrganisationUnitLevel() );

}

}

After doing this change, committer left a log as “Updated OrgUnitByLevelComparator to not rely on pre-populated level field, Added ‘level Sorted’ parameter to OrgUnitController, set to true if you want orgUnits sored by level”. Could you let us know why this pretty change results with DHIS2 build failure !!! I am not sure but is there any link in between pretty cool commit 9242 and your this email !!! Pls elaborate and provide us your expert view on this issue, i am expecting, we should do pretty cool hand around this issue and you let us know how we should comes out from this

“Build Failure” !!! I wish this issue should be closed before 23rd December.

Our orgunit tree (the one-true-tree) is maintained using parentid pointers. This is simple enough to maintain and updates and insertions are efficient.

Of course updates and insertions are relatively rare. What we need to do much more frequently is selecting various subtrees. All the facilities in a district, all the districts in a province etc.

Also the depth of our trees are relatively shallow. Most places seem to have around 5 levels. There is some trend that this starts to increase, but not exponentially. We might conceive one day of 8 or even 10 levels but not 100s or 1000s of levels. Its this shallowness which makes the _orgunitstructure table viable as the number of columns in that table will always be within a practical limit…

Selecting for tree structures (traversal) which are built using ‘parentid’ is not very efficient. Postgres offers ‘WITH RECURSIVE’ which is pretty cool if you can get your head around it, but not supported in mysql (and not necessarily fast either).

So when selecting subtrees the _orgunitstructure table is our best friend. I use it for the mydatamart aggregatedXXvalue queries and it looks like Lars uses it in the scheduled datamart job as well as well. And it is the obvious table to use when implementing filtering as I am now looking at re mydatamart export.

The problem is (with this and the other resource tables) how to maintain integrity. Currently we generate this table on demand (from the user) from the parentid pointers. If users forget to do it then all sorts of things fail. There has been some discussion on list of generating this, at least nightly which would be an improvement. There would be some benefit in maintaining it dynamically, ie “triggered” during those relatively rare updates and insertions of new orgunits. One possible consequence of this approach would be that the parentid on the orgunit becomes effectively redundant. It could be argued that its only current use is to generate the _orgunitstructure table.

A consequence of the parentid being redundant is that hierarchy is maintained in a separate table to the orgunits themselves. And there is no reason why there should be only one _orgunitstructure table. There could be any number, limited only by the number of hierarchies we needed to maintain. There’s a thought …

Of course the other use for the parentid is when you are exporting a bundle of orgunits in a way which reflects the (or a) hierarchy. In which case it is really kind to clients to serialize this tree as a breadth-first traversal ie. they come out in order which is quick and easy to rebuild on the client. That is what I am looking at doing now to try and help out our mydatamart metadata export. And I will of course use the _orgunitstructure table to make this trivial on the server side.

But some principles derived from the above discussion:

  1. try to avoid using parentid directly in code … always link to the orgunitstructuretable. This might ease its eventual demise
  1. all places where we export collections of orgunits should be in a proper traversal - breadth first or depth firt shouldn’t matter really but I’m going to do the latter.

Bob

Merry Christmas

:slight_smile:

Regards,

Brajesh

···

On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

–
Regards,
Brajesh

3

Brajesh I am not sure I understand the tone of your email nor the impressive cc list. But anyway …

If you have a build failure then please describe that simply on the list and perhaps people can help you. It is worth noting that the continuous integration server http://apps.dhis2.org/ci/ is building fine so at least the current trunk is fine.

In general if there was a compilation failure on trunk (and I am not sure I can verify because I don’t think I built 9242 but you can look back through the emails and see) an alert is sent to the devs list and it is resolved very quickly. Never more than a few hours. So I think you are safe regarding the issue being resolved before December 24.

Please verify that you have updated to latest trunk and see if the problem persists. If so then give a detailed report to the list. But please do so in a separate thread.

Regards

Bob

···

On 14 December 2012 21:31, Brajesh Murari brajesh2murari@gmail.com wrote:

Hello Bob,

Wishing you and everyone on dhis2-dev list, a joyous Christmas season.

I am getting build failure in main trunk build with info as given below,

[INFO] -------------------------------------------------------------

[ERROR] COMPILATION ERROR :

[INFO] -------------------------------------------------------------

[ERROR] \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22] cannot find symbol

symbol : method compare(int,int)

location: class java.lang.Integer

[INFO] 1 error

On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Just sharing some thoughts before I lose them …

Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.

One day later after your this email, Morten Olav Hansen did a miner change

in one custom comparator ie. OrganisationUnitByLevelComparator.java .

in lp:dhis2 branch with revision number 9242. Before 9242, this comparator

was looks like this given below

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return o1.getLevel() - o2.getLevel();

}

}

Now, after 9242, it has been like

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return Integer.compare( o1.getOrganisationUnitLevel(), o2.getOrganisationUnitLevel() );

}

}

After doing this change, committer left a log as “Updated OrgUnitByLevelComparator to not rely on pre-populated level field, Added ‘level Sorted’ parameter to OrgUnitController, set to true if you want orgUnits sored by level”. Could you let us know why this pretty change results with DHIS2 build failure !!! I am not sure but is there any link in between pretty cool commit 9242 and your this email !!! Pls elaborate and provide us your expert view on this issue, i am expecting, we should do pretty cool hand around this issue and you let us know how we should comes out from this

“Build Failure” !!! I wish this issue should be closed before 23rd December.

Our orgunit tree (the one-true-tree) is maintained using parentid pointers. This is simple enough to maintain and updates and insertions are efficient.

Of course updates and insertions are relatively rare. What we need to do much more frequently is selecting various subtrees. All the facilities in a district, all the districts in a province etc.

Also the depth of our trees are relatively shallow. Most places seem to have around 5 levels. There is some trend that this starts to increase, but not exponentially. We might conceive one day of 8 or even 10 levels but not 100s or 1000s of levels. Its this shallowness which makes the _orgunitstructure table viable as the number of columns in that table will always be within a practical limit…

Selecting for tree structures (traversal) which are built using ‘parentid’ is not very efficient. Postgres offers ‘WITH RECURSIVE’ which is pretty cool if you can get your head around it, but not supported in mysql (and not necessarily fast either).

So when selecting subtrees the _orgunitstructure table is our best friend. I use it for the mydatamart aggregatedXXvalue queries and it looks like Lars uses it in the scheduled datamart job as well as well. And it is the obvious table to use when implementing filtering as I am now looking at re mydatamart export.

The problem is (with this and the other resource tables) how to maintain integrity. Currently we generate this table on demand (from the user) from the parentid pointers. If users forget to do it then all sorts of things fail. There has been some discussion on list of generating this, at least nightly which would be an improvement. There would be some benefit in maintaining it dynamically, ie “triggered” during those relatively rare updates and insertions of new orgunits. One possible consequence of this approach would be that the parentid on the orgunit becomes effectively redundant. It could be argued that its only current use is to generate the _orgunitstructure table.

A consequence of the parentid being redundant is that hierarchy is maintained in a separate table to the orgunits themselves. And there is no reason why there should be only one _orgunitstructure table. There could be any number, limited only by the number of hierarchies we needed to maintain. There’s a thought …

Of course the other use for the parentid is when you are exporting a bundle of orgunits in a way which reflects the (or a) hierarchy. In which case it is really kind to clients to serialize this tree as a breadth-first traversal ie. they come out in order which is quick and easy to rebuild on the client. That is what I am looking at doing now to try and help out our mydatamart metadata export. And I will of course use the _orgunitstructure table to make this trivial on the server side.

But some principles derived from the above discussion:

  1. try to avoid using parentid directly in code … always link to the orgunitstructuretable. This might ease its eventual demise
  1. all places where we export collections of orgunits should be in a proper traversal - breadth first or depth firt shouldn’t matter really but I’m going to do the latter.

Bob

Merry Christmas

:slight_smile:

Regards,

Brajesh

–
Regards,
Brajesh

4

There should not be any issues with that code, and as bob said our ci server seems to compile it fine (also, no other dev has complained) . Update, clean, install and if it fails update java.

···

On 14 December 2012 21:31, Brajesh Murari brajesh2murari@gmail.com wrote:

Hello Bob,

Wishing you and everyone on dhis2-dev list, a joyous Christmas season.

I am getting build failure in main trunk build with info as given below,

[INFO] -------------------------------------------------------------

[ERROR] COMPILATION ERROR :

[INFO] -------------------------------------------------------------

[ERROR] \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22] cannot find symbol

symbol : method compare(int,int)

location: class java.lang.Integer

[INFO] 1 error

On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Just sharing some thoughts before I lose them …

Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.

One day later after your this email, Morten Olav Hansen did a miner change

in one custom comparator ie. OrganisationUnitByLevelComparator.java .

in lp:dhis2 branch with revision number 9242. Before 9242, this comparator

was looks like this given below

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return o1.getLevel() - o2.getLevel();

}

}

Now, after 9242, it has been like

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return Integer.compare( o1.getOrganisationUnitLevel(), o2.getOrganisationUnitLevel() );

}

}

After doing this change, committer left a log as “Updated OrgUnitByLevelComparator to not rely on pre-populated level field, Added ‘level Sorted’ parameter to OrgUnitController, set to true if you want orgUnits sored by level”. Could you let us know why this pretty change results with DHIS2 build failure !!! I am not sure but is there any link in between pretty cool commit 9242 and your this email !!! Pls elaborate and provide us your expert view on this issue, i am expecting, we should do pretty cool hand around this issue and you let us know how we should comes out from this

“Build Failure” !!! I wish this issue should be closed before 23rd December.

Our orgunit tree (the one-true-tree) is maintained using parentid pointers. This is simple enough to maintain and updates and insertions are efficient.

Of course updates and insertions are relatively rare. What we need to do much more frequently is selecting various subtrees. All the facilities in a district, all the districts in a province etc.

Also the depth of our trees are relatively shallow. Most places seem to have around 5 levels. There is some trend that this starts to increase, but not exponentially. We might conceive one day of 8 or even 10 levels but not 100s or 1000s of levels. Its this shallowness which makes the _orgunitstructure table viable as the number of columns in that table will always be within a practical limit…

Selecting for tree structures (traversal) which are built using ‘parentid’ is not very efficient. Postgres offers ‘WITH RECURSIVE’ which is pretty cool if you can get your head around it, but not supported in mysql (and not necessarily fast either).

So when selecting subtrees the _orgunitstructure table is our best friend. I use it for the mydatamart aggregatedXXvalue queries and it looks like Lars uses it in the scheduled datamart job as well as well. And it is the obvious table to use when implementing filtering as I am now looking at re mydatamart export.

The problem is (with this and the other resource tables) how to maintain integrity. Currently we generate this table on demand (from the user) from the parentid pointers. If users forget to do it then all sorts of things fail. There has been some discussion on list of generating this, at least nightly which would be an improvement. There would be some benefit in maintaining it dynamically, ie “triggered” during those relatively rare updates and insertions of new orgunits. One possible consequence of this approach would be that the parentid on the orgunit becomes effectively redundant. It could be argued that its only current use is to generate the _orgunitstructure table.

A consequence of the parentid being redundant is that hierarchy is maintained in a separate table to the orgunits themselves. And there is no reason why there should be only one _orgunitstructure table. There could be any number, limited only by the number of hierarchies we needed to maintain. There’s a thought …

Of course the other use for the parentid is when you are exporting a bundle of orgunits in a way which reflects the (or a) hierarchy. In which case it is really kind to clients to serialize this tree as a breadth-first traversal ie. they come out in order which is quick and easy to rebuild on the client. That is what I am looking at doing now to try and help out our mydatamart metadata export. And I will of course use the _orgunitstructure table to make this trivial on the server side.

But some principles derived from the above discussion:

  1. try to avoid using parentid directly in code … always link to the orgunitstructuretable. This might ease its eventual demise
  1. all places where we export collections of orgunits should be in a proper traversal - breadth first or depth firt shouldn’t matter really but I’m going to do the latter.

Bob

Merry Christmas

:slight_smile:

Regards,

Brajesh

–
Regards,
Brajesh

5

If you have an old version of Java, the commit I just did should probably fix it. But I would suggest that you upgrade.

···

–
Morten

On Sat, Dec 15, 2012 at 8:12 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

There should not be any issues with that code, and as bob said our ci server seems to compile it fine (also, no other dev has complained) . Update, clean, install and if it fails update java.

On Dec 15, 2012 2:50 AM, “Bob Jolliffe” bobjolliffe@gmail.com wrote:

Brajesh I am not sure I understand the tone of your email nor the impressive cc list. But anyway …

If you have a build failure then please describe that simply on the list and perhaps people can help you. It is worth noting that the continuous integration server http://apps.dhis2.org/ci/ is building fine so at least the current trunk is fine.

In general if there was a compilation failure on trunk (and I am not sure I can verify because I don’t think I built 9242 but you can look back through the emails and see) an alert is sent to the devs list and it is resolved very quickly. Never more than a few hours. So I think you are safe regarding the issue being resolved before December 24.

Please verify that you have updated to latest trunk and see if the problem persists. If so then give a detailed report to the list. But please do so in a separate thread.

Regards

Bob

On 14 December 2012 21:31, Brajesh Murari brajesh2murari@gmail.com wrote:

Hello Bob,

Wishing you and everyone on dhis2-dev list, a joyous Christmas season.

I am getting build failure in main trunk build with info as given below,

[INFO] -------------------------------------------------------------

[ERROR] COMPILATION ERROR :

[INFO] -------------------------------------------------------------

[ERROR] \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22] cannot find symbol

symbol : method compare(int,int)

location: class java.lang.Integer

[INFO] 1 error

On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Just sharing some thoughts before I lose them …

Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.

One day later after your this email, Morten Olav Hansen did a miner change

in one custom comparator ie. OrganisationUnitByLevelComparator.java .

in lp:dhis2 branch with revision number 9242. Before 9242, this comparator

was looks like this given below

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return o1.getLevel() - o2.getLevel();

}

}

Now, after 9242, it has been like

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return Integer.compare( o1.getOrganisationUnitLevel(), o2.getOrganisationUnitLevel() );

}

}

After doing this change, committer left a log as “Updated OrgUnitByLevelComparator to not rely on pre-populated level field, Added ‘level Sorted’ parameter to OrgUnitController, set to true if you want orgUnits sored by level”. Could you let us know why this pretty change results with DHIS2 build failure !!! I am not sure but is there any link in between pretty cool commit 9242 and your this email !!! Pls elaborate and provide us your expert view on this issue, i am expecting, we should do pretty cool hand around this issue and you let us know how we should comes out from this

“Build Failure” !!! I wish this issue should be closed before 23rd December.

Our orgunit tree (the one-true-tree) is maintained using parentid pointers. This is simple enough to maintain and updates and insertions are efficient.

Of course updates and insertions are relatively rare. What we need to do much more frequently is selecting various subtrees. All the facilities in a district, all the districts in a province etc.

Also the depth of our trees are relatively shallow. Most places seem to have around 5 levels. There is some trend that this starts to increase, but not exponentially. We might conceive one day of 8 or even 10 levels but not 100s or 1000s of levels. Its this shallowness which makes the _orgunitstructure table viable as the number of columns in that table will always be within a practical limit…

Selecting for tree structures (traversal) which are built using ‘parentid’ is not very efficient. Postgres offers ‘WITH RECURSIVE’ which is pretty cool if you can get your head around it, but not supported in mysql (and not necessarily fast either).

So when selecting subtrees the _orgunitstructure table is our best friend. I use it for the mydatamart aggregatedXXvalue queries and it looks like Lars uses it in the scheduled datamart job as well as well. And it is the obvious table to use when implementing filtering as I am now looking at re mydatamart export.

The problem is (with this and the other resource tables) how to maintain integrity. Currently we generate this table on demand (from the user) from the parentid pointers. If users forget to do it then all sorts of things fail. There has been some discussion on list of generating this, at least nightly which would be an improvement. There would be some benefit in maintaining it dynamically, ie “triggered” during those relatively rare updates and insertions of new orgunits. One possible consequence of this approach would be that the parentid on the orgunit becomes effectively redundant. It could be argued that its only current use is to generate the _orgunitstructure table.

A consequence of the parentid being redundant is that hierarchy is maintained in a separate table to the orgunits themselves. And there is no reason why there should be only one _orgunitstructure table. There could be any number, limited only by the number of hierarchies we needed to maintain. There’s a thought …

Of course the other use for the parentid is when you are exporting a bundle of orgunits in a way which reflects the (or a) hierarchy. In which case it is really kind to clients to serialize this tree as a breadth-first traversal ie. they come out in order which is quick and easy to rebuild on the client. That is what I am looking at doing now to try and help out our mydatamart metadata export. And I will of course use the _orgunitstructure table to make this trivial on the server side.

But some principles derived from the above discussion:

  1. try to avoid using parentid directly in code … always link to the orgunitstructuretable. This might ease its eventual demise
  1. all places where we export collections of orgunits should be in a proper traversal - breadth first or depth firt shouldn’t matter really but I’m going to do the latter.

Bob

Merry Christmas

:slight_smile:

Regards,

Brajesh

–
Regards,
Brajesh

6

Hi Morten,

I am using java version with info below

brajesh.murari@mohlapbrajesh /cygdrive/d/src/dhis2/dhis-2

$ java -version

java version “1.6.0_26”

Java™ SE Runtime Environment (build 1.6.0_26-b03)

Java HotSpot™ Client VM (build 20.1-b02, mixed mode)

After your fix, its now working, with same above java version. If you do any

java version up-gradation in scm-dhis2, don’t forget to update its info here as well on

http://www.dhis2.org/development

I will upgrade my java version on 24th December in morning at 7:30 AM IST after

taking fresh shower with cold/child water. I thought, may be there would be any real time idea implementation with your commit 9242 without blueprint or any bug fixing after Bob’s universal declaration ie. “Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.”

···

–
Morten

On Sat, Dec 15, 2012 at 8:12 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

There should not be any issues with that code, and as bob said our ci server seems to compile it fine (also, no other dev has complained) . Update, clean, install and if it fails update java.

On Dec 15, 2012 2:50 AM, “Bob Jolliffe” bobjolliffe@gmail.com wrote:

Brajesh I am not sure I understand the tone of your email nor the impressive cc list. But anyway …

If you have a build failure then please describe that simply on the list and perhaps people can help you. It is worth noting that the continuous integration server http://apps.dhis2.org/ci/ is building fine so at least the current trunk is fine.

In general if there was a compilation failure on trunk (and I am not sure I can verify because I don’t think I built 9242 but you can look back through the emails and see) an alert is sent to the devs list and it is resolved very quickly. Never more than a few hours. So I think you are safe regarding the issue being resolved before December 24.

Please verify that you have updated to latest trunk and see if the problem persists. If so then give a detailed report to the list. But please do so in a separate thread.

Regards

Bob

On 14 December 2012 21:31, Brajesh Murari brajesh2murari@gmail.com wrote:

Hello Bob,

Wishing you and everyone on dhis2-dev list, a joyous Christmas season.

I am getting build failure in main trunk build with info as given below,

[INFO] -------------------------------------------------------------

[ERROR] COMPILATION ERROR :

[INFO] -------------------------------------------------------------

[ERROR] \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22] cannot find symbol

symbol : method compare(int,int)

location: class java.lang.Integer

[INFO] 1 error

On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Just sharing some thoughts before I lose them …

Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.

One day later after your this email, Morten Olav Hansen did a miner change

in one custom comparator ie. OrganisationUnitByLevelComparator.java .

in lp:dhis2 branch with revision number 9242. Before 9242, this comparator

was looks like this given below

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return o1.getLevel() - o2.getLevel();

}

}

Now, after 9242, it has been like

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return Integer.compare( o1.getOrganisationUnitLevel(), o2.getOrganisationUnitLevel() );

}

}

After doing this change, committer left a log as “Updated OrgUnitByLevelComparator to not rely on pre-populated level field, Added ‘level Sorted’ parameter to OrgUnitController, set to true if you want orgUnits sored by level”. Could you let us know why this pretty change results with DHIS2 build failure !!! I am not sure but is there any link in between pretty cool commit 9242 and your this email !!! Pls elaborate and provide us your expert view on this issue, i am expecting, we should do pretty cool hand around this issue and you let us know how we should comes out from this

“Build Failure” !!! I wish this issue should be closed before 23rd December.

Our orgunit tree (the one-true-tree) is maintained using parentid pointers. This is simple enough to maintain and updates and insertions are efficient.

Of course updates and insertions are relatively rare. What we need to do much more frequently is selecting various subtrees. All the facilities in a district, all the districts in a province etc.

Also the depth of our trees are relatively shallow. Most places seem to have around 5 levels. There is some trend that this starts to increase, but not exponentially. We might conceive one day of 8 or even 10 levels but not 100s or 1000s of levels. Its this shallowness which makes the _orgunitstructure table viable as the number of columns in that table will always be within a practical limit…

Selecting for tree structures (traversal) which are built using ‘parentid’ is not very efficient. Postgres offers ‘WITH RECURSIVE’ which is pretty cool if you can get your head around it, but not supported in mysql (and not necessarily fast either).

So when selecting subtrees the _orgunitstructure table is our best friend. I use it for the mydatamart aggregatedXXvalue queries and it looks like Lars uses it in the scheduled datamart job as well as well. And it is the obvious table to use when implementing filtering as I am now looking at re mydatamart export.

The problem is (with this and the other resource tables) how to maintain integrity. Currently we generate this table on demand (from the user) from the parentid pointers. If users forget to do it then all sorts of things fail. There has been some discussion on list of generating this, at least nightly which would be an improvement. There would be some benefit in maintaining it dynamically, ie “triggered” during those relatively rare updates and insertions of new orgunits. One possible consequence of this approach would be that the parentid on the orgunit becomes effectively redundant. It could be argued that its only current use is to generate the _orgunitstructure table.

A consequence of the parentid being redundant is that hierarchy is maintained in a separate table to the orgunits themselves. And there is no reason why there should be only one _orgunitstructure table. There could be any number, limited only by the number of hierarchies we needed to maintain. There’s a thought …

Of course the other use for the parentid is when you are exporting a bundle of orgunits in a way which reflects the (or a) hierarchy. In which case it is really kind to clients to serialize this tree as a breadth-first traversal ie. they come out in order which is quick and easy to rebuild on the client. That is what I am looking at doing now to try and help out our mydatamart metadata export. And I will of course use the _orgunitstructure table to make this trivial on the server side.

But some principles derived from the above discussion:

  1. try to avoid using parentid directly in code … always link to the orgunitstructuretable. This might ease its eventual demise
  1. all places where we export collections of orgunits should be in a proper traversal - breadth first or depth firt shouldn’t matter really but I’m going to do the latter.

Bob

Merry Christmas

:slight_smile:

Regards,

Brajesh

–

7

Ok

···

–
Morten

On Sat, Dec 15, 2012 at 8:12 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

There should not be any issues with that code, and as bob said our ci server seems to compile it fine (also, no other dev has complained) . Update, clean, install and if it fails update java.

On Dec 15, 2012 2:50 AM, “Bob Jolliffe” bobjolliffe@gmail.com wrote:

Brajesh I am not sure I understand the tone of your email nor the impressive cc list. But anyway …

If you have a build failure then please describe that simply on the list and perhaps people can help you. It is worth noting that the continuous integration server http://apps.dhis2.org/ci/ is building fine so at least the current trunk is fine.

In general if there was a compilation failure on trunk (and I am not sure I can verify because I don’t think I built 9242 but you can look back through the emails and see) an alert is sent to the devs list and it is resolved very quickly. Never more than a few hours. So I think you are safe regarding the issue being resolved before December 24.

Please verify that you have updated to latest trunk and see if the problem persists. If so then give a detailed report to the list. But please do so in a separate thread.

Regards

Bob

On 14 December 2012 21:31, Brajesh Murari brajesh2murari@gmail.com wrote:

Hello Bob,

Wishing you and everyone on dhis2-dev list, a joyous Christmas season.

I am getting build failure in main trunk build with info as given below,

[INFO] -------------------------------------------------------------

[ERROR] COMPILATION ERROR :

[INFO] -------------------------------------------------------------

[ERROR] \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22] cannot find symbol

symbol : method compare(int,int)

location: class java.lang.Integer

[INFO] 1 error

On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Just sharing some thoughts before I lose them …

Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.

One day later after your this email, Morten Olav Hansen did a miner change

in one custom comparator ie. OrganisationUnitByLevelComparator.java .

in lp:dhis2 branch with revision number 9242. Before 9242, this comparator

was looks like this given below

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return o1.getLevel() - o2.getLevel();

}

}

Now, after 9242, it has been like

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return Integer.compare( o1.getOrganisationUnitLevel(), o2.getOrganisationUnitLevel() );

}

}

After doing this change, committer left a log as “Updated OrgUnitByLevelComparator to not rely on pre-populated level field, Added ‘level Sorted’ parameter to OrgUnitController, set to true if you want orgUnits sored by level”. Could you let us know why this pretty change results with DHIS2 build failure !!! I am not sure but is there any link in between pretty cool commit 9242 and your this email !!! Pls elaborate and provide us your expert view on this issue, i am expecting, we should do pretty cool hand around this issue and you let us know how we should comes out from this

“Build Failure” !!! I wish this issue should be closed before 23rd December.

Our orgunit tree (the one-true-tree) is maintained using parentid pointers. This is simple enough to maintain and updates and insertions are efficient.

Of course updates and insertions are relatively rare. What we need to do much more frequently is selecting various subtrees. All the facilities in a district, all the districts in a province etc.

Also the depth of our trees are relatively shallow. Most places seem to have around 5 levels. There is some trend that this starts to increase, but not exponentially. We might conceive one day of 8 or even 10 levels but not 100s or 1000s of levels. Its this shallowness which makes the _orgunitstructure table viable as the number of columns in that table will always be within a practical limit…

Selecting for tree structures (traversal) which are built using ‘parentid’ is not very efficient. Postgres offers ‘WITH RECURSIVE’ which is pretty cool if you can get your head around it, but not supported in mysql (and not necessarily fast either).

So when selecting subtrees the _orgunitstructure table is our best friend. I use it for the mydatamart aggregatedXXvalue queries and it looks like Lars uses it in the scheduled datamart job as well as well. And it is the obvious table to use when implementing filtering as I am now looking at re mydatamart export.

The problem is (with this and the other resource tables) how to maintain integrity. Currently we generate this table on demand (from the user) from the parentid pointers. If users forget to do it then all sorts of things fail. There has been some discussion on list of generating this, at least nightly which would be an improvement. There would be some benefit in maintaining it dynamically, ie “triggered” during those relatively rare updates and insertions of new orgunits. One possible consequence of this approach would be that the parentid on the orgunit becomes effectively redundant. It could be argued that its only current use is to generate the _orgunitstructure table.

A consequence of the parentid being redundant is that hierarchy is maintained in a separate table to the orgunits themselves. And there is no reason why there should be only one _orgunitstructure table. There could be any number, limited only by the number of hierarchies we needed to maintain. There’s a thought …

Of course the other use for the parentid is when you are exporting a bundle of orgunits in a way which reflects the (or a) hierarchy. In which case it is really kind to clients to serialize this tree as a breadth-first traversal ie. they come out in order which is quick and easy to rebuild on the client. That is what I am looking at doing now to try and help out our mydatamart metadata export. And I will of course use the _orgunitstructure table to make this trivial on the server side.

But some principles derived from the above discussion:

  1. try to avoid using parentid directly in code … always link to the orgunitstructuretable. This might ease its eventual demise
  1. all places where we export collections of orgunits should be in a proper traversal - breadth first or depth firt shouldn’t matter really but I’m going to do the latter.

Bob

Merry Christmas

:slight_smile:

Regards,

Brajesh

–

8

Hi Morten,

Pardon to put you in little bit more work around, although my build process is working fine in my local system …I have three quires which i want to share with you and others on list before I lose them…

  1. I just want to know which one is the best !!!

public class OrganisationUnitByLevelComparator

implements Comparator<OrganisationUnit>

{

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 )

{

    return **new Integer**( o1.getOrganisationUnitLevel() ).compareTo( o2.getOrganisationUnitLevel() );

}

}

Note: In each comparison, creating a new wrapper Integer object, ie. if this could be called approx 40,000 times to compare two different Organisation Units might be more in some case ( they might be name of households of a certain West African country ) while loading a Tree than creating 40,000 new wrapper Integer objects just for doing comparison, and that might be treated as bottle neck in such large scale operation as an assumption.

**or **

Light weighted code like given below

public class OrganisationUnitByLevelComparator

implements Comparator<OrganisationUnit>

{

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 )

{

    return o1.getOrgaonisationUnitLevel() - o2.getOrganisationUnitLevel();

}

}

  1. Would you like to review your commit in revision 9329 !!!

  2. Are you in a favor to configure Crucible with DHIS2 root trunk like other tools eg. CI (Jinkins/Atlassian Bamboo etc…) so that other can put their code review comments on dhis2 committed code by great commiter like you !!!

http://www.atlassian.com/software/crucible/overview

If yes in second and third query, than i am strongly recommend, just do it in DHIS2 before December 24th this year.

Regards,

~Brajesh~

···

–
Morten

On Sat, Dec 15, 2012 at 8:12 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

There should not be any issues with that code, and as bob said our ci server seems to compile it fine (also, no other dev has complained) . Update, clean, install and if it fails update java.

On Dec 15, 2012 2:50 AM, “Bob Jolliffe” bobjolliffe@gmail.com wrote:

Brajesh I am not sure I understand the tone of your email nor the impressive cc list. But anyway …

If you have a build failure then please describe that simply on the list and perhaps people can help you. It is worth noting that the continuous integration server http://apps.dhis2.org/ci/ is building fine so at least the current trunk is fine.

In general if there was a compilation failure on trunk (and I am not sure I can verify because I don’t think I built 9242 but you can look back through the emails and see) an alert is sent to the devs list and it is resolved very quickly. Never more than a few hours. So I think you are safe regarding the issue being resolved before December 24.

Please verify that you have updated to latest trunk and see if the problem persists. If so then give a detailed report to the list. But please do so in a separate thread.

Regards

Bob

On 14 December 2012 21:31, Brajesh Murari brajesh2murari@gmail.com wrote:

Hello Bob,

Wishing you and everyone on dhis2-dev list, a joyous Christmas season.

I am getting build failure in main trunk build with info as given below,

[INFO] -------------------------------------------------------------

[ERROR] COMPILATION ERROR :

[INFO] -------------------------------------------------------------

[ERROR] \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22] cannot find symbol

symbol : method compare(int,int)

location: class java.lang.Integer

[INFO] 1 error

On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Just sharing some thoughts before I lose them …

Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.

One day later after your this email, Morten Olav Hansen did a miner change

in one custom comparator ie. OrganisationUnitByLevelComparator.java .

in lp:dhis2 branch with revision number 9242. Before 9242, this comparator

was looks like this given below

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return o1.getLevel() - o2.getLevel();

}

}

Now, after 9242, it has been like

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return Integer.compare( o1.getOrganisationUnitLevel(), o2.getOrganisationUnitLevel() );

}

}

After doing this change, committer left a log as “Updated OrgUnitByLevelComparator to not rely on pre-populated level field, Added ‘level Sorted’ parameter to OrgUnitController, set to true if you want orgUnits sored by level”. Could you let us know why this pretty change results with DHIS2 build failure !!! I am not sure but is there any link in between pretty cool commit 9242 and your this email !!! Pls elaborate and provide us your expert view on this issue, i am expecting, we should do pretty cool hand around this issue and you let us know how we should comes out from this

“Build Failure” !!! I wish this issue should be closed before 23rd December.

Our orgunit tree (the one-true-tree) is maintained using parentid pointers. This is simple enough to maintain and updates and insertions are efficient.

Of course updates and insertions are relatively rare. What we need to do much more frequently is selecting various subtrees. All the facilities in a district, all the districts in a province etc.

Also the depth of our trees are relatively shallow. Most places seem to have around 5 levels. There is some trend that this starts to increase, but not exponentially. We might conceive one day of 8 or even 10 levels but not 100s or 1000s of levels. Its this shallowness which makes the _orgunitstructure table viable as the number of columns in that table will always be within a practical limit…

Selecting for tree structures (traversal) which are built using ‘parentid’ is not very efficient. Postgres offers ‘WITH RECURSIVE’ which is pretty cool if you can get your head around it, but not supported in mysql (and not necessarily fast either).

So when selecting subtrees the _orgunitstructure table is our best friend. I use it for the mydatamart aggregatedXXvalue queries and it looks like Lars uses it in the scheduled datamart job as well as well. And it is the obvious table to use when implementing filtering as I am now looking at re mydatamart export.

The problem is (with this and the other resource tables) how to maintain integrity. Currently we generate this table on demand (from the user) from the parentid pointers. If users forget to do it then all sorts of things fail. There has been some discussion on list of generating this, at least nightly which would be an improvement. There would be some benefit in maintaining it dynamically, ie “triggered” during those relatively rare updates and insertions of new orgunits. One possible consequence of this approach would be that the parentid on the orgunit becomes effectively redundant. It could be argued that its only current use is to generate the _orgunitstructure table.

A consequence of the parentid being redundant is that hierarchy is maintained in a separate table to the orgunits themselves. And there is no reason why there should be only one _orgunitstructure table. There could be any number, limited only by the number of hierarchies we needed to maintain. There’s a thought …

Of course the other use for the parentid is when you are exporting a bundle of orgunits in a way which reflects the (or a) hierarchy. In which case it is really kind to clients to serialize this tree as a breadth-first traversal ie. they come out in order which is quick and easy to rebuild on the client. That is what I am looking at doing now to try and help out our mydatamart metadata export. And I will of course use the _orgunitstructure table to make this trivial on the server side.

But some principles derived from the above discussion:

  1. try to avoid using parentid directly in code … always link to the orgunitstructuretable. This might ease its eventual demise
  1. all places where we export collections of orgunits should be in a proper traversal - breadth first or depth firt shouldn’t matter really but I’m going to do the latter.

Bob

Merry Christmas

:slight_smile:

Regards,

Brajesh

–

9

This was just a quick-fix done at the airport. I will look at it later.

···

–
Morten

On Mon, Dec 17, 2012 at 7:13 PM, Brajesh Murari brajesh2murari@gmail.com wrote:

Hi Morten,

Pardon to put you in little bit more work around, although my build process is working fine in my local system …I have three quires which i want to share with you and others on list before I lose them…

  1. I just want to know which one is the best !!!

public class OrganisationUnitByLevelComparator

implements Comparator<OrganisationUnit>

{

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 )

{

    return **new Integer**( o1.getOrganisationUnitLevel() ).compareTo( o2.getOrganisationUnitLevel() );

}

}

Note: In each comparison, creating a new wrapper Integer object, ie. if this could be called approx 40,000 times to compare two different Organisation Units might be more in some case ( they might be name of households of a certain West African country ) while loading a Tree than creating 40,000 new wrapper Integer objects just for doing comparison, and that might be treated as bottle neck in such large scale operation as an assumption.

**or **

Light weighted code like given below

public class OrganisationUnitByLevelComparator

implements Comparator<OrganisationUnit>

{

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 )

{

    return o1.getOrgaonisationUnitLevel() - o2.getOrganisationUnitLevel();

}

}

  1. Would you like to review your commit in revision 9329 !!!
  1. Are you in a favor to configure Crucible with DHIS2 root trunk like other tools eg. CI (Jinkins/Atlassian Bamboo etc…) so that other can put their code review comments on dhis2 committed code by great commiter like you !!!

http://www.atlassian.com/software/crucible/overview

If yes in second and third query, than i am strongly recommend, just do it in DHIS2 before December 24th this year.

Regards,

~Brajesh~

On Sun, Dec 16, 2012 at 1:33 AM, Morten Olav Hansen mortenoh@gmail.com wrote:

Ok

On Dec 15, 2012 10:12 PM, “Brajesh Murari” brajesh2murari@gmail.com wrote:

Hi Morten,

I am using java version with info below

brajesh.murari@mohlapbrajesh /cygdrive/d/src/dhis2/dhis-2

$ java -version

java version “1.6.0_26”

Java™ SE Runtime Environment (build 1.6.0_26-b03)

Java HotSpot™ Client VM (build 20.1-b02, mixed mode)

After your fix, its now working, with same above java version. If you do any

java version up-gradation in scm-dhis2, don’t forget to update its info here as well on

http://www.dhis2.org/development

I will upgrade my java version on 24th December in morning at 7:30 AM IST after

taking fresh shower with cold/child water. I thought, may be there would be any real time idea implementation with your commit 9242 without blueprint or any bug fixing after Bob’s universal declaration ie. “Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.”

On Sat, Dec 15, 2012 at 10:58 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

If you have an old version of Java, the commit I just did should probably fix it. But I would suggest that you upgrade.
*>>>*Developers usually don’t take fresh checkout nor they like to do some crazy white box testing all the time, that’s why perhaps they may not complain usually, but there are some developers who don’t even relay on CI nightly build status update, that’s why they complain about build failure…which was perhaps true.

:slight_smile:

Any way, build procedure is working fine now in my system.

thanks for informing us.

:slight_smile:

–
Regards,
Brajesh

–
Regards,
Brajesh

–
Morten

On Sat, Dec 15, 2012 at 8:12 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

There should not be any issues with that code, and as bob said our ci server seems to compile it fine (also, no other dev has complained) . Update, clean, install and if it fails update java.

On Dec 15, 2012 2:50 AM, “Bob Jolliffe” bobjolliffe@gmail.com wrote:

Brajesh I am not sure I understand the tone of your email nor the impressive cc list. But anyway …

If you have a build failure then please describe that simply on the list and perhaps people can help you. It is worth noting that the continuous integration server http://apps.dhis2.org/ci/ is building fine so at least the current trunk is fine.

In general if there was a compilation failure on trunk (and I am not sure I can verify because I don’t think I built 9242 but you can look back through the emails and see) an alert is sent to the devs list and it is resolved very quickly. Never more than a few hours. So I think you are safe regarding the issue being resolved before December 24.

Please verify that you have updated to latest trunk and see if the problem persists. If so then give a detailed report to the list. But please do so in a separate thread.

Regards

Bob

On 14 December 2012 21:31, Brajesh Murari brajesh2murari@gmail.com wrote:

Hello Bob,

Wishing you and everyone on dhis2-dev list, a joyous Christmas season.

I am getting build failure in main trunk build with info as given below,

[INFO] -------------------------------------------------------------

[ERROR] COMPILATION ERROR :

[INFO] -------------------------------------------------------------

[ERROR] \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22] cannot find symbol

symbol : method compare(int,int)

location: class java.lang.Integer

[INFO] 1 error

On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Just sharing some thoughts before I lose them …

Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.

One day later after your this email, Morten Olav Hansen did a miner change

in one custom comparator ie. OrganisationUnitByLevelComparator.java .

in lp:dhis2 branch with revision number 9242. Before 9242, this comparator

was looks like this given below

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return o1.getLevel() - o2.getLevel();

}

}

Now, after 9242, it has been like

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return Integer.compare( o1.getOrganisationUnitLevel(), o2.getOrganisationUnitLevel() );

}

}

After doing this change, committer left a log as “Updated OrgUnitByLevelComparator to not rely on pre-populated level field, Added ‘level Sorted’ parameter to OrgUnitController, set to true if you want orgUnits sored by level”. Could you let us know why this pretty change results with DHIS2 build failure !!! I am not sure but is there any link in between pretty cool commit 9242 and your this email !!! Pls elaborate and provide us your expert view on this issue, i am expecting, we should do pretty cool hand around this issue and you let us know how we should comes out from this

“Build Failure” !!! I wish this issue should be closed before 23rd December.

Our orgunit tree (the one-true-tree) is maintained using parentid pointers. This is simple enough to maintain and updates and insertions are efficient.

Of course updates and insertions are relatively rare. What we need to do much more frequently is selecting various subtrees. All the facilities in a district, all the districts in a province etc.

Also the depth of our trees are relatively shallow. Most places seem to have around 5 levels. There is some trend that this starts to increase, but not exponentially. We might conceive one day of 8 or even 10 levels but not 100s or 1000s of levels. Its this shallowness which makes the _orgunitstructure table viable as the number of columns in that table will always be within a practical limit…

Selecting for tree structures (traversal) which are built using ‘parentid’ is not very efficient. Postgres offers ‘WITH RECURSIVE’ which is pretty cool if you can get your head around it, but not supported in mysql (and not necessarily fast either).

So when selecting subtrees the _orgunitstructure table is our best friend. I use it for the mydatamart aggregatedXXvalue queries and it looks like Lars uses it in the scheduled datamart job as well as well. And it is the obvious table to use when implementing filtering as I am now looking at re mydatamart export.

The problem is (with this and the other resource tables) how to maintain integrity. Currently we generate this table on demand (from the user) from the parentid pointers. If users forget to do it then all sorts of things fail. There has been some discussion on list of generating this, at least nightly which would be an improvement. There would be some benefit in maintaining it dynamically, ie “triggered” during those relatively rare updates and insertions of new orgunits. One possible consequence of this approach would be that the parentid on the orgunit becomes effectively redundant. It could be argued that its only current use is to generate the _orgunitstructure table.

A consequence of the parentid being redundant is that hierarchy is maintained in a separate table to the orgunits themselves. And there is no reason why there should be only one _orgunitstructure table. There could be any number, limited only by the number of hierarchies we needed to maintain. There’s a thought …

Of course the other use for the parentid is when you are exporting a bundle of orgunits in a way which reflects the (or a) hierarchy. In which case it is really kind to clients to serialize this tree as a breadth-first traversal ie. they come out in order which is quick and easy to rebuild on the client. That is what I am looking at doing now to try and help out our mydatamart metadata export. And I will of course use the _orgunitstructure table to make this trivial on the server side.

But some principles derived from the above discussion:

  1. try to avoid using parentid directly in code … always link to the orgunitstructuretable. This might ease its eventual demise
  1. all places where we export collections of orgunits should be in a proper traversal - breadth first or depth firt shouldn’t matter really but I’m going to do the latter.

Bob

Merry Christmas

:slight_smile:

Regards,

Brajesh

–

10

Hi Morten,

Pardon ! I am just coming from office after doing about 9 hours of hard work

like other DHIS2 implementer s in more than 30 countries. I am not sure you

are talking about which Quick-Fix !!! The Quick-fix with revision number 9242

or 9323 !!! We are not getting exactly ! Could you let us know which one exactly !

If you means you did pretty cool commit and quick-fix with revision 9323 at

airport than where you did pretty cool commit with revision 9242 !!!

With kind Regards,

Brajesh

···

–
Morten

On Mon, Dec 17, 2012 at 7:13 PM, Brajesh Murari brajesh2murari@gmail.com wrote:

Hi Morten,

Pardon to put you in little bit more work around, although my build process is working fine in my local system …I have three quires which i want to share with you and others on list before I lose them…

  1. I just want to know which one is the best !!!

public class OrganisationUnitByLevelComparator

implements Comparator<OrganisationUnit>

{

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 )

{

    return **new Integer**( o1.getOrganisationUnitLevel() ).compareTo( o2.getOrganisationUnitLevel() );

}

}

Note: In each comparison, creating a new wrapper Integer object, ie. if this could be called approx 40,000 times to compare two different Organisation Units might be more in some case ( they might be name of households of a certain West African country ) while loading a Tree than creating 40,000 new wrapper Integer objects just for doing comparison, and that might be treated as bottle neck in such large scale operation as an assumption.

**or **

Light weighted code like given below

public class OrganisationUnitByLevelComparator

implements Comparator<OrganisationUnit>

{

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 )

{

    return o1.getOrgaonisationUnitLevel() - o2.getOrganisationUnitLevel();

}

}

  1. Would you like to review your commit in revision 9329 !!!
  1. Are you in a favor to configure Crucible with DHIS2 root trunk like other tools eg. CI (Jinkins/Atlassian Bamboo etc…) so that other can put their code review comments on dhis2 committed code by great commiter like you !!!

http://www.atlassian.com/software/crucible/overview

If yes in second and third query, than i am strongly recommend, just do it in DHIS2 before December 24th this year.

Regards,

~Brajesh~

On Sun, Dec 16, 2012 at 1:33 AM, Morten Olav Hansen mortenoh@gmail.com wrote:

Ok

On Dec 15, 2012 10:12 PM, “Brajesh Murari” brajesh2murari@gmail.com wrote:

Hi Morten,

I am using java version with info below

brajesh.murari@mohlapbrajesh /cygdrive/d/src/dhis2/dhis-2

$ java -version

java version “1.6.0_26”

Java™ SE Runtime Environment (build 1.6.0_26-b03)

Java HotSpot™ Client VM (build 20.1-b02, mixed mode)

After your fix, its now working, with same above java version. If you do any

java version up-gradation in scm-dhis2, don’t forget to update its info here as well on

http://www.dhis2.org/development

I will upgrade my java version on 24th December in morning at 7:30 AM IST after

taking fresh shower with cold/child water. I thought, may be there would be any real time idea implementation with your commit 9242 without blueprint or any bug fixing after Bob’s universal declaration ie. “Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.”

On Sat, Dec 15, 2012 at 10:58 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

If you have an old version of Java, the commit I just did should probably fix it. But I would suggest that you upgrade.
*>>>*Developers usually don’t take fresh checkout nor they like to do some crazy white box testing all the time, that’s why perhaps they may not complain usually, but there are some developers who don’t even relay on CI nightly build status update, that’s why they complain about build failure…which was perhaps true.

:slight_smile:

Any way, build procedure is working fine now in my system.

thanks for informing us.

:slight_smile:

–
Regards,
Brajesh

–
Regards,
Brajesh

–
Morten

On Sat, Dec 15, 2012 at 8:12 PM, Morten Olav Hansen mortenoh@gmail.com wrote:

There should not be any issues with that code, and as bob said our ci server seems to compile it fine (also, no other dev has complained) . Update, clean, install and if it fails update java.

On Dec 15, 2012 2:50 AM, “Bob Jolliffe” bobjolliffe@gmail.com wrote:

Brajesh I am not sure I understand the tone of your email nor the impressive cc list. But anyway …

If you have a build failure then please describe that simply on the list and perhaps people can help you. It is worth noting that the continuous integration server http://apps.dhis2.org/ci/ is building fine so at least the current trunk is fine.

In general if there was a compilation failure on trunk (and I am not sure I can verify because I don’t think I built 9242 but you can look back through the emails and see) an alert is sent to the devs list and it is resolved very quickly. Never more than a few hours. So I think you are safe regarding the issue being resolved before December 24.

Please verify that you have updated to latest trunk and see if the problem persists. If so then give a detailed report to the list. But please do so in a separate thread.

Regards

Bob

On 14 December 2012 21:31, Brajesh Murari brajesh2murari@gmail.com wrote:

Hello Bob,

Wishing you and everyone on dhis2-dev list, a joyous Christmas season.

I am getting build failure in main trunk build with info as given below,

[INFO] -------------------------------------------------------------

[ERROR] COMPILATION ERROR :

[INFO] -------------------------------------------------------------

[ERROR] \src\dhis2\dhis-2\dhis-api\src\main\java\org\hisp\dhis\organisationunit\comparator\OrganisationUnitByLevelComparator.java:[43,22] cannot find symbol

symbol : method compare(int,int)

location: class java.lang.Integer

[INFO] 1 error

On Sat, Dec 8, 2012 at 9:45 PM, Bob Jolliffe bobjolliffe@gmail.com wrote:

Just sharing some thoughts before I lose them …

Been looking at the case of a certain West African country with 40000 orgunits. Which is actually not a huge number of objects and we could very likely be looking at orgunit tables bigger than this as villages and even households start finding their way into the tree.

One day later after your this email, Morten Olav Hansen did a miner change

in one custom comparator ie. OrganisationUnitByLevelComparator.java .

in lp:dhis2 branch with revision number 9242. Before 9242, this comparator

was looks like this given below

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return o1.getLevel() - o2.getLevel();

}

}

Now, after 9242, it has been like

@Override

public int compare( OrganisationUnit o1, OrganisationUnit o2 ){

return Integer.compare( o1.getOrganisationUnitLevel(), o2.getOrganisationUnitLevel() );

}

}

After doing this change, committer left a log as “Updated OrgUnitByLevelComparator to not rely on pre-populated level field, Added ‘level Sorted’ parameter to OrgUnitController, set to true if you want orgUnits sored by level”. Could you let us know why this pretty change results with DHIS2 build failure !!! I am not sure but is there any link in between pretty cool commit 9242 and your this email !!! Pls elaborate and provide us your expert view on this issue, i am expecting, we should do pretty cool hand around this issue and you let us know how we should comes out from this

“Build Failure” !!! I wish this issue should be closed before 23rd December.

Our orgunit tree (the one-true-tree) is maintained using parentid pointers. This is simple enough to maintain and updates and insertions are efficient.

Of course updates and insertions are relatively rare. What we need to do much more frequently is selecting various subtrees. All the facilities in a district, all the districts in a province etc.

Also the depth of our trees are relatively shallow. Most places seem to have around 5 levels. There is some trend that this starts to increase, but not exponentially. We might conceive one day of 8 or even 10 levels but not 100s or 1000s of levels. Its this shallowness which makes the _orgunitstructure table viable as the number of columns in that table will always be within a practical limit…

Selecting for tree structures (traversal) which are built using ‘parentid’ is not very efficient. Postgres offers ‘WITH RECURSIVE’ which is pretty cool if you can get your head around it, but not supported in mysql (and not necessarily fast either).

So when selecting subtrees the _orgunitstructure table is our best friend. I use it for the mydatamart aggregatedXXvalue queries and it looks like Lars uses it in the scheduled datamart job as well as well. And it is the obvious table to use when implementing filtering as I am now looking at re mydatamart export.

The problem is (with this and the other resource tables) how to maintain integrity. Currently we generate this table on demand (from the user) from the parentid pointers. If users forget to do it then all sorts of things fail. There has been some discussion on list of generating this, at least nightly which would be an improvement. There would be some benefit in maintaining it dynamically, ie “triggered” during those relatively rare updates and insertions of new orgunits. One possible consequence of this approach would be that the parentid on the orgunit becomes effectively redundant. It could be argued that its only current use is to generate the _orgunitstructure table.

A consequence of the parentid being redundant is that hierarchy is maintained in a separate table to the orgunits themselves. And there is no reason why there should be only one _orgunitstructure table. There could be any number, limited only by the number of hierarchies we needed to maintain. There’s a thought …

Of course the other use for the parentid is when you are exporting a bundle of orgunits in a way which reflects the (or a) hierarchy. In which case it is really kind to clients to serialize this tree as a breadth-first traversal ie. they come out in order which is quick and easy to rebuild on the client. That is what I am looking at doing now to try and help out our mydatamart metadata export. And I will of course use the _orgunitstructure table to make this trivial on the server side.

But some principles derived from the above discussion:

  1. try to avoid using parentid directly in code … always link to the orgunitstructuretable. This might ease its eventual demise
  1. all places where we export collections of orgunits should be in a proper traversal - breadth first or depth firt shouldn’t matter really but I’m going to do the latter.

Bob

Merry Christmas

:slight_smile:

Regards,

Brajesh

–