defining datavalueset reports for dhis2 web api

Sharing some thoughts about using the web-api for facility reporting ....

Setting up a client to produce datavalueset reports using the web api
gets a bit complicated when you are using categorycombos. The problem
is that you can retrieve the metadata for a dataset, but that just
gives you the list of dataelements to report - but finding out which
categoryoptioncombos are required for each dataelement involves quite
a bit more querying of the api.

Creating an sdmx style data structure definition is difficult because
of the "raggedness" of our datasets (they are not neat datacubes with
uniform dimensionality). So another way to approach this is to
acquire report templates for each datavalueset - ie. retrieve the
template from dhis and the client is then only required to configure
itself to provide the values for each row in the template.

The web api doesn't provide these directly, but they are easy enough
to generate off the metadata. At a minimum you require the datasets,
the dataelements and the categoryoptioncombos. If the client is a
facility based system (like openmrs) then its really not necessary to
get the list of 1000's of orgunits.

Using the attached xsl and the url below is my first stab at this:

curl -v -X GET -u admin:district
"DHIS2 App Hub;

xsltproc dxf2template.xslt - |xmllint --format -

The resulting output (also attached) is I think the minimum
information required to fully configure a datavalueset producer based
on a template, using dxf2, sdmx, csv, xforms or what have you. I am
assuming that a facility already 'knows' its facility identifier or
code.

Bob

dxf2template.xslt (2.03 KB)

reportTemplates.xml (166 KB)

Hi Bob,

This is very interesting.
Much closer to xforms where the form is sent without the dataValues and can be filled with values and sent back.

Some comments:

1.) I wonder though why you are sending all the dataElements in the xml?? Is that required or we should be able to generate only those required on a dataSet or section of a form?

2.) There should be a dataType field for the elements or are u suggesting “chatty” conversation to get the types??

Other than that I wonder if the client has to deal with the transform. Or was this just an example to generate it and will be done on the server-side.

I would suggest that the client should be able to set Accept headers and get the template in formats that you mentioned in your email…

···

Regards,
Saptarshi PURKAYASTHA

My Tech Blog: http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE

On 16 July 2012 12:57, Bob Jolliffe bobjolliffe@gmail.com wrote:

Sharing some thoughts about using the web-api for facility reporting …

Setting up a client to produce datavalueset reports using the web api

gets a bit complicated when you are using categorycombos. The problem

is that you can retrieve the metadata for a dataset, but that just

gives you the list of dataelements to report - but finding out which

categoryoptioncombos are required for each dataelement involves quite

a bit more querying of the api.

Creating an sdmx style data structure definition is difficult because

of the “raggedness” of our datasets (they are not neat datacubes with

uniform dimensionality). So another way to approach this is to

acquire report templates for each datavalueset - ie. retrieve the

template from dhis and the client is then only required to configure

itself to provide the values for each row in the template.

The web api doesn’t provide these directly, but they are easy enough

to generate off the metadata. At a minimum you require the datasets,

the dataelements and the categoryoptioncombos. If the client is a

facility based system (like openmrs) then its really not necessary to

get the list of 1000’s of orgunits.

Using the attached xsl and the url below is my first stab at this:

curl -v -X GET -u admin:district

http://apps.dhis2.org/demo/api/metaData.xml?assumeTrue=false&categoryOptionCombos=true&dataElements=true&dataSets=true

xsltproc dxf2template.xslt - |xmllint --format -

The resulting output (also attached) is I think the minimum

information required to fully configure a datavalueset producer based

on a template, using dxf2, sdmx, csv, xforms or what have you. I am

assuming that a facility already ‘knows’ its facility identifier or

code.

Bob


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Bob,

This is very interesting.
Much closer to xforms where the form is sent without the dataValues and can
be filled with values and sent back.

Some comments:
1.) I wonder though why you are sending all the dataElements in the xml?? Is
that required or we should be able to generate only those required on a
dataSet or section of a form?

Yes I thought about this. If you want all the datavaluest templates
(like I have done here) then it it makes sense to get the full list of
dataelements .. some datelements might even get reused across forms.

If you just want the specification for a single datavalueset template
then you really do just need the list of dataelements for that
datavalueset. Notice that you do already have that list of required
dataelements in the dataset xml - unfortunately this list format (it
is generic for all identifiable objects) doesn't contain the
categorycombo so a bit of duplication is required.

2.) There should be a dataType field for the elements or are u suggesting
"chatty" conversation to get the types??

There should. I was assuming "numeric" but I guess that is not fair.

Other than that I wonder if the client has to deal with the transform. Or
was this just an example to generate it and will be done on the server-side.
I would suggest that the client should be able to set Accept headers and get
the template in formats that you mentioned in your email...

I agree that the client's life could be made easier if such a
transform was done on the server side. I just put this out as a
feeler to see if people who are building dxf2 clients might find it
useful. Of course its not a big problem to do on the client side and
has the advantage of not needing to build consensus around what a
"standard" representation should be. But such a representation
available from the server could make building clients even easier.

Where would you map it? If you want to get all the reports like I
have done then something like "application/reportTemplate+xml" on
metadata url might do, The idea being that a client could read in the
list which templates it wanted to configure/save.

Slightly more chatty but maybe also sensible (but I think maybe more
complicated to implement), would be to use
"application/reportTemplate+xml" on the dataset url (with the
restricted list of dataelements).

I am looking at this in the context of an openmrs client module. I'd
be happy to do the transform on the client side for now to iron out
any ruffles.

Bob

···

On 16 July 2012 13:46, Saptarshi Purkayastha <sunbiz@gmail.com> wrote:

---
Regards,
Saptarshi PURKAYASTHA

My Tech Blog: http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE

On 16 July 2012 12:57, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

Sharing some thoughts about using the web-api for facility reporting ....

Setting up a client to produce datavalueset reports using the web api
gets a bit complicated when you are using categorycombos. The problem
is that you can retrieve the metadata for a dataset, but that just
gives you the list of dataelements to report - but finding out which
categoryoptioncombos are required for each dataelement involves quite
a bit more querying of the api.

Creating an sdmx style data structure definition is difficult because
of the "raggedness" of our datasets (they are not neat datacubes with
uniform dimensionality). So another way to approach this is to
acquire report templates for each datavalueset - ie. retrieve the
template from dhis and the client is then only required to configure
itself to provide the values for each row in the template.

The web api doesn't provide these directly, but they are easy enough
to generate off the metadata. At a minimum you require the datasets,
the dataelements and the categoryoptioncombos. If the client is a
facility based system (like openmrs) then its really not necessary to
get the list of 1000's of orgunits.

Using the attached xsl and the url below is my first stab at this:

curl -v -X GET -u admin:district

"DHIS2 App Hub;
> xsltproc dxf2template.xslt - |xmllint --format -

The resulting output (also attached) is I think the minimum
information required to fully configure a datavalueset producer based
on a template, using dxf2, sdmx, csv, xforms or what have you. I am
assuming that a facility already 'knows' its facility identifier or
code.

Bob

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

Including type ...

dxf2template.xslt (2.04 KB)

···

On 16 July 2012 14:21, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

On 16 July 2012 13:46, Saptarshi Purkayastha <sunbiz@gmail.com> wrote:

Hi Bob,

This is very interesting.
Much closer to xforms where the form is sent without the dataValues and can
be filled with values and sent back.

Some comments:
1.) I wonder though why you are sending all the dataElements in the xml?? Is
that required or we should be able to generate only those required on a
dataSet or section of a form?

Yes I thought about this. If you want all the datavaluest templates
(like I have done here) then it it makes sense to get the full list of
dataelements .. some datelements might even get reused across forms.

If you just want the specification for a single datavalueset template
then you really do just need the list of dataelements for that
datavalueset. Notice that you do already have that list of required
dataelements in the dataset xml - unfortunately this list format (it
is generic for all identifiable objects) doesn't contain the
categorycombo so a bit of duplication is required.

2.) There should be a dataType field for the elements or are u suggesting
"chatty" conversation to get the types??

There should. I was assuming "numeric" but I guess that is not fair.

Other than that I wonder if the client has to deal with the transform. Or
was this just an example to generate it and will be done on the server-side.
I would suggest that the client should be able to set Accept headers and get
the template in formats that you mentioned in your email...

I agree that the client's life could be made easier if such a
transform was done on the server side. I just put this out as a
feeler to see if people who are building dxf2 clients might find it
useful. Of course its not a big problem to do on the client side and
has the advantage of not needing to build consensus around what a
"standard" representation should be. But such a representation
available from the server could make building clients even easier.

Where would you map it? If you want to get all the reports like I
have done then something like "application/reportTemplate+xml" on
metadata url might do, The idea being that a client could read in the
list which templates it wanted to configure/save.

Slightly more chatty but maybe also sensible (but I think maybe more
complicated to implement), would be to use
"application/reportTemplate+xml" on the dataset url (with the
restricted list of dataelements).

I am looking at this in the context of an openmrs client module. I'd
be happy to do the transform on the client side for now to iron out
any ruffles.

Bob

---
Regards,
Saptarshi PURKAYASTHA

My Tech Blog: http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE

On 16 July 2012 12:57, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

Sharing some thoughts about using the web-api for facility reporting ....

Setting up a client to produce datavalueset reports using the web api
gets a bit complicated when you are using categorycombos. The problem
is that you can retrieve the metadata for a dataset, but that just
gives you the list of dataelements to report - but finding out which
categoryoptioncombos are required for each dataelement involves quite
a bit more querying of the api.

Creating an sdmx style data structure definition is difficult because
of the "raggedness" of our datasets (they are not neat datacubes with
uniform dimensionality). So another way to approach this is to
acquire report templates for each datavalueset - ie. retrieve the
template from dhis and the client is then only required to configure
itself to provide the values for each row in the template.

The web api doesn't provide these directly, but they are easy enough
to generate off the metadata. At a minimum you require the datasets,
the dataelements and the categoryoptioncombos. If the client is a
facility based system (like openmrs) then its really not necessary to
get the list of 1000's of orgunits.

Using the attached xsl and the url below is my first stab at this:

curl -v -X GET -u admin:district

"DHIS2 App Hub;
> xsltproc dxf2template.xslt - |xmllint --format -

The resulting output (also attached) is I think the minimum
information required to fully configure a datavalueset producer based
on a template, using dxf2, sdmx, csv, xforms or what have you. I am
assuming that a facility already 'knows' its facility identifier or
code.

Bob

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

If you just want the specification for a single datavalueset template

then you really do just need the list of dataelements for that

datavalueset. Notice that you do already have that list of required

dataelements in the dataset xml - unfortunately this list format (it

is generic for all identifiable objects) doesn’t contain the

categorycombo so a bit of duplication is required.

Yes, I was thinking if this duplication can be avoided. So only the required disAggregations are received…

There should. I was assuming “numeric” but I guess that is not fair.

Although the type might be dataType like you did in the next xslt, it’d still not be able to do the xforms/javarosa-style xforms because its datatype and not viewtype. Like I could represent a multi-select through checkbox or list etc. So, that case of having view-representation is still missing.

I agree that the client’s life could be made easier if such a

transform was done on the server side. I just put this out as a

feeler to see if people who are building dxf2 clients might find it

useful. Of course its not a big problem to do on the client side and

has the advantage of not needing to build consensus around what a

“standard” representation should be. But such a representation

available from the server could make building clients even easier.

xslt is fairly memory intensive because its in-memory. The newest XSLT 3.0 standard brings streaming, but that’s not what you are using. This means that clients need a lot of memory on large transforms and I’m thinking of mobile when I was referring to be able to get this transform done on the server-side.

Where would you map it? If you want to get all the reports like I

have done then something like “application/reportTemplate+xml” on

metadata url might do, The idea being that a client could read in the

list which templates it wanted to configure/save.

Slightly more chatty but maybe also sensible (but I think maybe more

complicated to implement), would be to use

“application/reportTemplate+xml” on the dataset url (with the

restricted list of dataelements).

a small nitpick that mime-types are generally not camelCase.

Yes, but we could have the client configure/save the XSLT on the server and on next calls get that response only. Sadly, javarosa-xforms uses text/xml and nothing more specific :frowning: . Some mobile client could nicely send an XSLT for JSON and get that representation from the server for reportTempate??

···

On 16 July 2012 15:21, Bob Jolliffe bobjolliffe@gmail.com wrote:


Regards,
Saptarshi PURKAYASTHA

My Tech Blog: http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE

Bob


Regards,

Saptarshi PURKAYASTHA

My Tech Blog: http://sunnytalkstech.blogspot.com

You Live by CHOICE, Not by CHANCE

On 16 July 2012 12:57, Bob Jolliffe bobjolliffe@gmail.com wrote:

Sharing some thoughts about using the web-api for facility reporting …

Setting up a client to produce datavalueset reports using the web api

gets a bit complicated when you are using categorycombos. The problem

is that you can retrieve the metadata for a dataset, but that just

gives you the list of dataelements to report - but finding out which

categoryoptioncombos are required for each dataelement involves quite

a bit more querying of the api.

Creating an sdmx style data structure definition is difficult because

of the “raggedness” of our datasets (they are not neat datacubes with

uniform dimensionality). So another way to approach this is to

acquire report templates for each datavalueset - ie. retrieve the

template from dhis and the client is then only required to configure

itself to provide the values for each row in the template.

The web api doesn’t provide these directly, but they are easy enough

to generate off the metadata. At a minimum you require the datasets,

the dataelements and the categoryoptioncombos. If the client is a

facility based system (like openmrs) then its really not necessary to

get the list of 1000’s of orgunits.

Using the attached xsl and the url below is my first stab at this:

curl -v -X GET -u admin:district

http://apps.dhis2.org/demo/api/metaData.xml?assumeTrue=false&categoryOptionCombos=true&dataElements=true&dataSets=true

xsltproc dxf2template.xslt - |xmllint --format -

The resulting output (also attached) is I think the minimum

information required to fully configure a datavalueset producer based

on a template, using dxf2, sdmx, csv, xforms or what have you. I am

assuming that a facility already ‘knows’ its facility identifier or

code.

Bob


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

If you just want the specification for a single datavalueset template
then you really do just need the list of dataelements for that
datavalueset. Notice that you do already have that list of required
dataelements in the dataset xml - unfortunately this list format (it
is generic for all identifiable objects) doesn't contain the
categorycombo so a bit of duplication is required.

Yes, I was thinking if this duplication can be avoided. So only the required
disAggregations are received...

what you gain on the swings you are going to lose on the roundabouts.
As I say it depends really on whether you want to download one form or
all the forms. I can see from your mobile use case, it is one form ..

There should. I was assuming "numeric" but I guess that is not fair.

Although the type might be dataType like you did in the next xslt, it'd
still not be able to do the xforms/javarosa-style xforms because its
datatype and not viewtype. Like I could represent a multi-select through
checkbox or list etc. So, that case of having view-representation is still
missing.

Yes. But I guess that is missing in our datamodel. This would be
defined in a Form object ...

I agree that the client's life could be made easier if such a
transform was done on the server side. I just put this out as a
feeler to see if people who are building dxf2 clients might find it
useful. Of course its not a big problem to do on the client side and
has the advantage of not needing to build consensus around what a
"standard" representation should be. But such a representation
available from the server could make building clients even easier.

xslt is fairly memory intensive because its in-memory. The newest XSLT 3.0
standard brings streaming, but that's not what you are using.

not sure anybody is yet :slight_smile:

This means
that clients need a lot of memory on large transforms and I'm thinking of
mobile when I was referring to be able to get this transform done on the
server-side.

Where would you map it? If you want to get all the reports like I
have done then something like "application/reportTemplate+xml" on
metadata url might do, The idea being that a client could read in the
list which templates it wanted to configure/save.

Slightly more chatty but maybe also sensible (but I think maybe more
complicated to implement), would be to use
"application/reportTemplate+xml" on the dataset url (with the
restricted list of dataelements).

a small nitpick that mime-types are generally not camelCase.

a bad java habit ....

Yes, but we could have the client configure/save the XSLT on the server and
on next calls get that response only. Sadly, javarosa-xforms uses text/xml
and nothing more specific :frowning: . Some mobile client could nicely send an XSLT
for JSON and get that representation from the server for reportTempate??

No .. I think the most sensible would just be to map this in dhis web
api to .../api/forms (maybe even in anticipation of a proper Form
object). Then text/xml might be fine. Though you would need
something like application/xforms+xml to get an xforms rendition.

···

On 16 July 2012 17:38, Saptarshi Purkayastha <sunbiz@gmail.com> wrote:

On 16 July 2012 15:21, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

---
Regards,
Saptarshi PURKAYASTHA

My Tech Blog: http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE

Bob

>
> ---
> Regards,
> Saptarshi PURKAYASTHA
>
> My Tech Blog: http://sunnytalkstech.blogspot.com
> You Live by CHOICE, Not by CHANCE
>
>
> On 16 July 2012 12:57, Bob Jolliffe <bobjolliffe@gmail.com> wrote:
>>
>> Sharing some thoughts about using the web-api for facility reporting
>> ....
>>
>> Setting up a client to produce datavalueset reports using the web api
>> gets a bit complicated when you are using categorycombos. The problem
>> is that you can retrieve the metadata for a dataset, but that just
>> gives you the list of dataelements to report - but finding out which
>> categoryoptioncombos are required for each dataelement involves quite
>> a bit more querying of the api.
>>
>> Creating an sdmx style data structure definition is difficult because
>> of the "raggedness" of our datasets (they are not neat datacubes with
>> uniform dimensionality). So another way to approach this is to
>> acquire report templates for each datavalueset - ie. retrieve the
>> template from dhis and the client is then only required to configure
>> itself to provide the values for each row in the template.
>>
>> The web api doesn't provide these directly, but they are easy enough
>> to generate off the metadata. At a minimum you require the datasets,
>> the dataelements and the categoryoptioncombos. If the client is a
>> facility based system (like openmrs) then its really not necessary to
>> get the list of 1000's of orgunits.
>>
>> Using the attached xsl and the url below is my first stab at this:
>>
>> curl -v -X GET -u admin:district
>>
>>
>> "DHIS2 App Hub;
>> > xsltproc dxf2template.xslt - |xmllint --format -
>>
>> The resulting output (also attached) is I think the minimum
>> information required to fully configure a datavalueset producer based
>> on a template, using dxf2, sdmx, csv, xforms or what have you. I am
>> assuming that a facility already 'knows' its facility identifier or
>> code.
>>
>> Bob
>>
>> _______________________________________________
>> Mailing list: DHIS 2 developers in Launchpad
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : DHIS 2 developers in Launchpad
>> More help : ListHelp - Launchpad Help
>>
>

If you just want the specification for a single datavalueset template
then you really do just need the list of dataelements for that
datavalueset. Notice that you do already have that list of required
dataelements in the dataset xml - unfortunately this list format (it
is generic for all identifiable objects) doesn't contain the
categorycombo so a bit of duplication is required.

Yes, I was thinking if this duplication can be avoided. So only the required
disAggregations are received...

There should. I was assuming "numeric" but I guess that is not fair.

Although the type might be dataType like you did in the next xslt, it'd
still not be able to do the xforms/javarosa-style xforms because its
datatype and not viewtype. Like I could represent a multi-select through
checkbox or list etc. So, that case of having view-representation is still
missing.

Thinking some more, this information is missing in my simple
representation anyway. By focussing just on the categoryoptioncombos
there is missing information about the actual categories and category
options. Obliging a client to do the whole mcdonalds thing is a tall
order ...

···

On 16 July 2012 17:38, Saptarshi Purkayastha <sunbiz@gmail.com> wrote:

On 16 July 2012 15:21, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

I agree that the client's life could be made easier if such a
transform was done on the server side. I just put this out as a
feeler to see if people who are building dxf2 clients might find it
useful. Of course its not a big problem to do on the client side and
has the advantage of not needing to build consensus around what a
"standard" representation should be. But such a representation
available from the server could make building clients even easier.

xslt is fairly memory intensive because its in-memory. The newest XSLT 3.0
standard brings streaming, but that's not what you are using. This means
that clients need a lot of memory on large transforms and I'm thinking of
mobile when I was referring to be able to get this transform done on the
server-side.

Where would you map it? If you want to get all the reports like I
have done then something like "application/reportTemplate+xml" on
metadata url might do, The idea being that a client could read in the
list which templates it wanted to configure/save.

Slightly more chatty but maybe also sensible (but I think maybe more
complicated to implement), would be to use
"application/reportTemplate+xml" on the dataset url (with the
restricted list of dataelements).

a small nitpick that mime-types are generally not camelCase.
Yes, but we could have the client configure/save the XSLT on the server and
on next calls get that response only. Sadly, javarosa-xforms uses text/xml
and nothing more specific :frowning: . Some mobile client could nicely send an XSLT
for JSON and get that representation from the server for reportTempate??

---
Regards,
Saptarshi PURKAYASTHA

My Tech Blog: http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE

Bob

>
> ---
> Regards,
> Saptarshi PURKAYASTHA
>
> My Tech Blog: http://sunnytalkstech.blogspot.com
> You Live by CHOICE, Not by CHANCE
>
>
> On 16 July 2012 12:57, Bob Jolliffe <bobjolliffe@gmail.com> wrote:
>>
>> Sharing some thoughts about using the web-api for facility reporting
>> ....
>>
>> Setting up a client to produce datavalueset reports using the web api
>> gets a bit complicated when you are using categorycombos. The problem
>> is that you can retrieve the metadata for a dataset, but that just
>> gives you the list of dataelements to report - but finding out which
>> categoryoptioncombos are required for each dataelement involves quite
>> a bit more querying of the api.
>>
>> Creating an sdmx style data structure definition is difficult because
>> of the "raggedness" of our datasets (they are not neat datacubes with
>> uniform dimensionality). So another way to approach this is to
>> acquire report templates for each datavalueset - ie. retrieve the
>> template from dhis and the client is then only required to configure
>> itself to provide the values for each row in the template.
>>
>> The web api doesn't provide these directly, but they are easy enough
>> to generate off the metadata. At a minimum you require the datasets,
>> the dataelements and the categoryoptioncombos. If the client is a
>> facility based system (like openmrs) then its really not necessary to
>> get the list of 1000's of orgunits.
>>
>> Using the attached xsl and the url below is my first stab at this:
>>
>> curl -v -X GET -u admin:district
>>
>>
>> "DHIS2 App Hub;
>> > xsltproc dxf2template.xslt - |xmllint --format -
>>
>> The resulting output (also attached) is I think the minimum
>> information required to fully configure a datavalueset producer based
>> on a template, using dxf2, sdmx, csv, xforms or what have you. I am
>> assuming that a facility already 'knows' its facility identifier or
>> code.
>>
>> Bob
>>
>> _______________________________________________
>> Mailing list: DHIS 2 developers in Launchpad
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : DHIS 2 developers in Launchpad
>> More help : ListHelp - Launchpad Help
>>
>