Currently the authentication used by mydatamart is a bit of a
pre-web-api hack. A login session cookie is obtained from
login.action and that is used to access the aggregated data csv data
from an action in reporting module.
I'd like to shift this action into the web-api both because (i) using
basic auth will be so much easier and (ii) that is probably where it
should be anyway. What do you think?
The urls are not beautifully restful like some of the rest, but I am
open to suggestion. Currently they look something like eg:
I'd like to shift this action into the web-api both because (i) using
basic auth will be so much easier and (ii) that is probably where it
should be anyway. What do you think?
I think it sounds like a good idea! (and another use-case for the web-api!)
The urls are not beautifully restful like some of the rest, but I am
open to suggestion. Currently they look something like eg:
with post parameters:
"dataSourceLevel=4&dataSourceRoot=539&startDate=20110201&endDate=20120528"
(note the use of internal ids for orgunits - currently we do not store
uids in the local datamart)
Would it be possible to use the dataValues part of the web-api? if we
extend it? or do you require very special meta-data? (will you be
using /api/metaData for that part?)
I'd like to shift this action into the web-api both because (i) using
basic auth will be so much easier and (ii) that is probably where it
should be anyway. What do you think?
I think it sounds like a good idea! (and another use-case for the web-api!)
The urls are not beautifully restful like some of the rest, but I am
open to suggestion. Currently they look something like eg:
with post parameters:
"dataSourceLevel=4&dataSourceRoot=539&startDate=20110201&endDate=20120528"
(note the use of internal ids for orgunits - currently we do not store
uids in the local datamart)
Would it be possible to use the dataValues part of the web-api? if we
extend it? or do you require very special meta-data? (will you be
using /api/metaData for that part?)
No. All that is being streamed is a gzipped csv file of aggregated
datavalues or aggregated indicator values from the datamart. I should
check do we have any dataMart theme in our api .. I don't think we do
(yet).
So I guess if we were to think generally, we should think of different
useful api methods for accessing (slices of) our datamart. Currently
I am happy to think quite primitively to just move the current actions
across. But with a placeholder for more refined ideas we can
introduce later. So I am thinking of something like a
DataMartValueController for want of a better name. And getting the
actual data in exactly the same way I currently do - using a fairly
finely tuned jdbc resultset iterator as these datamart dumps can get
quite large.
Bob
···
On 22 May 2012 17:06, Morten Olav Hansen <mortenoh@gmail.com> wrote:
No. All that is being streamed is a gzipped csv file of aggregated
datavalues or aggregated indicator values from the datamart. I should
check do we have any dataMart theme in our api .. I don't think we do
(yet).
Actually, we do have AggregatedValueController.. which I guess is what
you want. I don't think anyone is using it at moment, but I'm not
sure.. so you can probably hack and slash that controller into what
you want..
Lars, are we using this for anything? (is this part of the grid
renderer etc?) I remember making it.. but never used it..
So I guess if we were to think generally, we should think of different
useful api methods for accessing (slices of) our datamart. Currently
I am happy to think quite primitively to just move the current actions
across. But with a placeholder for more refined ideas we can
introduce later. So I am thinking of something like a
DataMartValueController for want of a better name. And getting the
actual data in exactly the same way I currently do - using a fairly
finely tuned jdbc resultset iterator as these datamart dumps can get
quite large.
Yeah, if you were to go with the approach used in the controller.. it
would probably be too slow (and not gzipped! atm..). Being able to get
slices of aggregated data as CVS sounds useful.. maybe json/xml also?
On 22 May 2012 17:20, Bob Jolliffe <bobjolliffe@gmail.com> wrote:
On 22 May 2012 17:06, Morten Olav Hansen <mortenoh@gmail.com> wrote:
I'd like to shift this action into the web-api both because (i) using
basic auth will be so much easier and (ii) that is probably where it
should be anyway. What do you think?
I think it sounds like a good idea! (and another use-case for the web-api!)
The urls are not beautifully restful like some of the rest, but I am
open to suggestion. Currently they look something like eg:
with post parameters:
"dataSourceLevel=4&dataSourceRoot=539&startDate=20110201&endDate=20120528"
(note the use of internal ids for orgunits - currently we do not store
uids in the local datamart)
Would it be possible to use the dataValues part of the web-api? if we
extend it? or do you require very special meta-data? (will you be
using /api/metaData for that part?)
No. All that is being streamed is a gzipped csv file of aggregated
datavalues or aggregated indicator values from the datamart. I should
check do we have any dataMart theme in our api .. I don't think we do
(yet).
So I guess if we were to think generally, we should think of different
useful api methods for accessing (slices of) our datamart. Currently
I am happy to think quite primitively to just move the current actions
across. But with a placeholder for more refined ideas we can
introduce later. So I am thinking of something like a
DataMartValueController for want of a better name. And getting the
actual data in exactly the same way I currently do - using a fairly
finely tuned jdbc resultset iterator as these datamart dumps can get
quite large.
No. All that is being streamed is a gzipped csv file of aggregated
datavalues or aggregated indicator values from the datamart. I should
check do we have any dataMart theme in our api .. I don't think we do
(yet).
Actually, we do have AggregatedValueController.. which I guess is what
you want. I don't think anyone is using it at moment, but I'm not
sure.. so you can probably hack and slash that controller into what
you want..
Lars, are we using this for anything? (is this part of the grid
renderer etc?) I remember making it.. but never used it..
That is a good question. Please shout up if this is currently in use.
Either way I can leave the single method there if people have already
built stuff with it. If not, then I might take liberty to modify it
...
So I guess if we were to think generally, we should think of different
useful api methods for accessing (slices of) our datamart. Currently
I am happy to think quite primitively to just move the current actions
across. But with a placeholder for more refined ideas we can
introduce later. So I am thinking of something like a
DataMartValueController for want of a better name. And getting the
actual data in exactly the same way I currently do - using a fairly
finely tuned jdbc resultset iterator as these datamart dumps can get
quite large.
Yeah, if you were to go with the approach used in the controller.. it
would probably be too slow (and not gzipped! atm..). Being able to get
slices of aggregated data as CVS sounds useful.. maybe json/xml also?
Yes well taking slices of the datamart is essential - currently that
is what mydatamart is doing - carving off chunks of data for a
particular orgunit subtree for a particular set of periods.
Strange though it might seem coming from me, I've not been tempted to
do either an xml or json rendition. For this sort of data, compressed
csv seems most appropriate. Starting to think like one of those
R-struck statisticians
···
On 22 May 2012 17:26, Morten Olav Hansen <mortenoh@gmail.com> wrote:
Lars, are we using this for anything? (is this part of the grid
renderer etc?) I remember making it.. but never used it..
That is a good question. Please shout up if this is currently in use.
Either way I can leave the single method there if people have already
built stuff with it. If not, then I might take liberty to modify it
I don't think we modify it.. but Lars would know. I was thinking about
our grid rendering, but that is done by the ReportTableController I
think.
Strange though it might seem coming from me, I've not been tempted to
do either an xml or json rendition. For this sort of data, compressed
csv seems most appropriate. Starting to think like one of those
R-struck statisticians
I think at least JSON would be nice (so you can create a web based
data browser, etc).
Lars, are we using this for anything? (is this part of the grid
renderer etc?) I remember making it.. but never used it..
That is a good question. Please shout up if this is currently in use.
Either way I can leave the single method there if people have already
built stuff with it. If not, then I might take liberty to modify it
I don't think we modify it.. but Lars would know. I was thinking about
our grid rendering, but that is done by the ReportTableController I
think.
Strange though it might seem coming from me, I've not been tempted to
do either an xml or json rendition. For this sort of data, compressed
csv seems most appropriate. Starting to think like one of those
R-struck statisticians
I think at least JSON would be nice (so you can create a web based
data browser, etc).
Not sure if web based aggregate data browser is high on the list of
priorities. I guess if people want to browse data they would
typically do this through a reporttable. Thats what I do anyway. My
priority is to simply have a convenient place to get hold of the
gzipped csv streams for mydatamart using basic authentication.
The current AggregatedValueController is neat, but there are two
things which need to be different:
1. it doesn't make sense to pass the list of orgunits where that list
can be very large. With the mydatamart request we post two parameters
- the "root" orgunit and the "level" and let dhis figure out which
orgunits are at that level in this part of the tree. That was the
requirement to be met. Though having said that, having the ability
to pass a list might also be useful - for example if the list of
orgunits you are concerned with spans different parts of the tree.
This can be the case with NGOs etc. But even there it might make
better sense to use a parameter (like orgunitgroup) to filter rather
sending an explicit list.
2. the jackson object based approach is nice, but won't scale well to
many concurrent requests for very large chunks of data, which is what
we are seeing in practice. Which is why I am streaming off a jdbc
iterator which uses a finite and controllable amount of memory.
So maybe I should leave what is there alone and simply move the
current actions across from reporting/exp to web-api and map it to a
different url than aggregateValue. Though this might be confusing.
Again, is anybody using aggregateValue in the web-api?
Regards
Bob
···
On 24 May 2012 09:13, Morten Olav Hansen <mortenoh@gmail.com> wrote:
