I started the mydatamart project nearly 2 years ago and its reached a certain point in its life where some people are using it and its limitations are becoming clear. The Liberia workshop highlighted a particular concern related to the painfully slow metadata import, becoming effectively unusable for large numbers of orgunits. I am planning to spend this week (only) on addressing some longstanding issues.
In the short term there are a few options we have discussed.
to restrict the piece of the orgunit tree which is downloaded as metadata on the server side to user’s subtree plus its peers and further restricted to a certain level. So first question regarding the level: if we want to leave the client exactly the way it is at the moment, can we enforce a maximum level on the server side (eg 3 deep, or 4 deep)? I wonder how many existing mydatamart users will find this a breaking change. Or should it be parameterized, with defaults so existing mydatamart clients continue to work as before.
the current method of getting the metadata into the sqlite database is via a (quite complicated) xslt transform which converts the dxf1 metadata into sql … generating the resource tables as it goes. This could be made quicker by (i) implementing a straight C/C++ parser and (ii) pushing the orgunit metadata directly into the database and generating the orgunitstructure after the fact. Unfortunately I don’t want to expend extra effort on dxf1 so anything new like this would be much better done on the newer dhis2 api.
Again I am considering doing this as a simple standalone tool to start with while we plot the larger migration. So users can carry on using mydatamart as before, but run a little prog on the side to replace the painful priming of the database metadata.
The main mydatamart program is written in Tcl which is kind of cool (though a bit of a mess) but I am also painfully aware that there is probably me and 3 others in the world proficient in using Tcl and that hasn’t changed markedly in our project over the past 2 years. So unless I am to be sole maintainer of this program for the rest of my life, we have a few choices:
(i) reimplement mydatamart in java, maybe as a swing app. Java doesn’t have native access to sqlite but can probably reach it through the odbc bridge.
(ii) reimplement mydatamart in QT. A bit of effort to get the opensource version working nicely on windows with MS Visual C++ compiler, but then a nice slick environment for developers who might find QT more accessible than Tcl (and it has strong Norwegian antecedents ).
(iii) refactor the horrible tcl code into nicer tcl code using something like snit or incrTcl and encourage new learners
Can we get some ideas of developer sentiment on the above? Personally I don’t really like java desktop apps but it maybe the most pragmatic. Qt would probably be the nicest but, well its not java and might be harder to get support from other programmers in our java oriented community. But otherwise it is really good, polished and well documented and quite easy to get into. (iii) is tempting but might just be me digging an even deeper hole for myself
Please give opinions. The choice of 3 will have some impact on how I approach 2.
Glad the mydatamart is grabbing the headlines now J . The major issue has been as you said is the metadata download ( we are currently going around this by sharing “ready made” datamart files for users to download new data – which is a more straight forward process)
I don’t think the a maximum level should be enforced on the server side, rather “parameterized” with default user profiles.
No comments……..
QT will be a good balance between Java and Tcl – Java not having native access to sqlite may bring problems down the line….
I started the mydatamart project nearly 2 years ago and its reached a certain point in its life where some people are using it and its limitations are becoming clear. The Liberia workshop highlighted a particular concern related to the painfully slow metadata import, becoming effectively unusable for large numbers of orgunits. I am planning to spend this week (only) on addressing some longstanding issues.
In the short term there are a few options we have discussed.
to restrict the piece of the orgunit tree which is downloaded as metadata on the server side to user’s subtree plus its peers and further restricted to a certain level. So first question regarding the level: if we want to leave the client exactly the way it is at the moment, can we enforce a maximum level on the server side (eg 3 deep, or 4 deep)? I wonder how many existing mydatamart users will find this a breaking change. Or should it be parameterized, with defaults so existing mydatamart clients continue to work as before.
the current method of getting the metadata into the sqlite database is via a (quite complicated) xslt transform which converts the dxf1 metadata into sql … generating the resource tables as it goes. This could be made quicker by (i) implementing a straight C/C++ parser and (ii) pushing the orgunit metadata directly into the database and generating the orgunitstructure after the fact. Unfortunately I don’t want to expend extra effort on dxf1 so anything new like this would be much better done on the newer dhis2 api.
Again I am considering doing this as a simple standalone tool to start with while we plot the larger migration. So users can carry on using mydatamart as before, but run a little prog on the side to replace the painful priming of the database metadata.
The main mydatamart program is written in Tcl which is kind of cool (though a bit of a mess) but I am also painfully aware that there is probably me and 3 others in the world proficient in using Tcl and that hasn’t changed markedly in our project over the past 2 years. So unless I am to be sole maintainer of this program for the rest of my life, we have a few choices:
(i) reimplement mydatamart in java, maybe as a swing app. Java doesn’t have native access to sqlite but can probably reach it through the odbc bridge.
(ii) reimplement mydatamart in QT. A bit of effort to get the opensource version working nicely on windows with MS Visual C++ compiler, but then a nice slick environment for developers who might find QT more accessible than Tcl (and it has strong Norwegian antecedents ).
(iii) refactor the horrible tcl code into nicer tcl code using something like snit or incrTcl and encourage new learners
Can we get some ideas of developer sentiment on the above? Personally I don’t really like java desktop apps but it maybe the most pragmatic. Qt would probably be the nicest but, well its not java and might be harder to get support from other programmers in our java oriented community. But otherwise it is really good, polished and well documented and quite easy to get into. (iii) is tempting but might just be me digging an even deeper hole for myself
Please give opinions. The choice of 3 will have some impact on how I approach 2.
Using .NET might not be stupid if our target is windows only. It should make sure the download is small, and most people have some version of the .NET framework installed already (but one would have to target a version that exists on Windows XP I assume). I think Lars is an expert
Glad the mydatamart is grabbing the headlines now J . The major issue has been as you said is the metadata download ( we are currently going around this by sharing “ready made” datamart files for users to download new data – which is a more straight forward process)
I don’t think the a maximum level should be enforced on the server side, rather “parameterized” with default user profiles.
No comments……..
QT will be a good balance between Java and Tcl – Java not having native access to sqlite may bring problems down the line….
Sent: Monday, December 10, 2012 10:54 AM To: dhis2-devs Subject: [Dhis2-devs] [mydatamart] technology choices
Hi devs
I started the mydatamart project nearly 2 years ago and its reached a certain point in its life where some people are using it and its limitations are becoming clear. The Liberia workshop highlighted a particular concern related to the painfully slow metadata import, becoming effectively unusable for large numbers of orgunits. I am planning to spend this week (only) on addressing some longstanding issues.
In the short term there are a few options we have discussed.
to restrict the piece of the orgunit tree which is downloaded as metadata on the server side to user’s subtree plus its peers and further restricted to a certain level. So first question regarding the level: if we want to leave the client exactly the way it is at the moment, can we enforce a maximum level on the server side (eg 3 deep, or 4 deep)? I wonder how many existing mydatamart users will find this a breaking change. Or should it be parameterized, with defaults so existing mydatamart clients continue to work as before.
the current method of getting the metadata into the sqlite database is via a (quite complicated) xslt transform which converts the dxf1 metadata into sql … generating the resource tables as it goes. This could be made quicker by (i) implementing a straight C/C++ parser and (ii) pushing the orgunit metadata directly into the database and generating the orgunitstructure after the fact. Unfortunately I don’t want to expend extra effort on dxf1 so anything new like this would be much better done on the newer dhis2 api.
Again I am considering doing this as a simple standalone tool to start with while we plot the larger migration. So users can carry on using mydatamart as before, but run a little prog on the side to replace the painful priming of the database metadata.
The main mydatamart program is written in Tcl which is kind of cool (though a bit of a mess) but I am also painfully aware that there is probably me and 3 others in the world proficient in using Tcl and that hasn’t changed markedly in our project over the past 2 years. So unless I am to be sole maintainer of this program for the rest of my life, we have a few choices:
(i) reimplement mydatamart in java, maybe as a swing app. Java doesn’t have native access to sqlite but can probably reach it through the odbc bridge.
(ii) reimplement mydatamart in QT. A bit of effort to get the opensource version working nicely on windows with MS Visual C++ compiler, but then a nice slick environment for developers who might find QT more accessible than Tcl (and it has strong Norwegian antecedents ).
(iii) refactor the horrible tcl code into nicer tcl code using something like snit or incrTcl and encourage new learners
Can we get some ideas of developer sentiment on the above? Personally I don’t really like java desktop apps but it maybe the most pragmatic. Qt would probably be the nicest but, well its not java and might be harder to get support from other programmers in our java oriented community. But otherwise it is really good, polished and well documented and quite easy to get into. (iii) is tempting but might just be me digging an even deeper hole for myself
Please give opinions. The choice of 3 will have some impact on how I approach 2.
).