API 2.25: Return Full Property Set

Dear all,

I am having a bit of an issue when querying metadata from the api in json format and further processing them in kettle (Pentaho DI): DHIS2 only returns those properties in the json-package that are not NULL. However, the json converter in kettle needs the full property-set in order to create a tabular structure from the data (don't know the reason for that).

Is there a way to force the api to return the full property set, e.g. a secret flag? I'd actually expect to get the full list - even if the property is NULL -, at least when I specify a list of properties in the field-filter ...

Of course, I could also ask the kettle-guys to be a bit more flexible, but normally they are not as agile as the DHIS2-DEVs :wink:

Best regards,

Uwe

Hi Uwe
Have a look here

https://ci.dhis2.org/docs/master/en/developer/html/webapi_metadata_field_filter.html

and specifically the use of the ?fields=:all paramater.

Regards,

Jason

···

On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser uwe@wahser.de wrote:

Dear all,

I am having a bit of an issue when querying metadata from the api in json format and further processing them in kettle (Pentaho DI): DHIS2 only returns those properties in the json-package that are not NULL. However, the json converter in kettle needs the full property-set in order to create a tabular structure from the data (don’t know the reason for that).

Is there a way to force the api to return the full property set, e.g. a secret flag? I’d actually expect to get the full list - even if the property is NULL -, at least when I specify a list of properties in the field-filter …

Of course, I could also ask the kettle-guys to be a bit more flexible, but normally they are not as agile as the DHIS2-DEVs :wink:

Best regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Thanks a lot, Jason, I will try that. I tried 'fields=' expecting this would be the same as 'fields=:all' , which is obviously wrong.

That's a very useful information - do you mind including it into the manual?

Regards, Uwe

···

---

Jason Pickering <jason.p.pickering@gmail.com> hat am 16. Dezember 2016 um 10:13 geschrieben:

Hi Uwe
Have a look here

https://ci.dhis2.org/docs/master/en/developer/html/webapi_metadata_field_filter.html

and specifically the use of the ?fields=:all paramater.

Regards,
Jason

On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser <uwe@wahser.de> wrote:

> Dear all,
>
> I am having a bit of an issue when querying metadata from the api in json
> format and further processing them in kettle (Pentaho DI): DHIS2 only
> returns those properties in the json-package that are not NULL. However,
> the json converter in kettle needs the full property-set in order to create
> a tabular structure from the data (don't know the reason for that).
>
> Is there a way to force the api to return the full property set, e.g. a
> secret flag? I'd actually expect to get the full list - even if the
> property is NULL -, at least when I specify a list of properties in the
> field-filter ...
>
> Of course, I could also ask the kettle-guys to be a bit more flexible, but
> normally they are not as agile as the DHIS2-DEVs :wink:
>
> Best regards,
>
> Uwe
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to : dhis2-users@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help : https://help.launchpad.net/ListHelp
>

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Hi Uwe,
I think the manual is correct, as it specifies how the paramater should be called.

In general, we discourage actually using that “all” paramater, because things can change from version to version (as you have noticed). Its better to make explicit calls for the fields which you require for each object, using something like /api/26/indicators?fields=id,name

where each required field is specified explicitly. This should also decrease the payloads which the ETL process would need to work with. You might want to have a look at that approach.

I would be happy to update the manual, but not really sure what is missing in this case?

Regards,

Jason

···

On Fri, Dec 16, 2016 at 8:24 AM, Uwe Wahser uwe@wahser.de wrote:

Thanks a lot, Jason, I will try that. I tried ‘fields=’ expecting this would be the same as ‘fields=:all’ , which is obviously wrong.

That’s a very useful information - do you mind including it into the manual?

Regards, Uwe


Jason Pickering jason.p.pickering@gmail.com hat am 16. Dezember 2016 um 10:13 geschrieben:

Hi Uwe

Have a look here

https://ci.dhis2.org/docs/master/en/developer/html/webapi_metadata_field_filter.html

and specifically the use of the ?fields=:all paramater.

Regards,

Jason

On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser uwe@wahser.de wrote:

Dear all,

I am having a bit of an issue when querying metadata from the api in json

format and further processing them in kettle (Pentaho DI): DHIS2 only

returns those properties in the json-package that are not NULL. However,

the json converter in kettle needs the full property-set in order to create

a tabular structure from the data (don’t know the reason for that).

Is there a way to force the api to return the full property set, e.g. a

secret flag? I’d actually expect to get the full list - even if the

property is NULL -, at least when I specify a list of properties in the

field-filter …

Of course, I could also ask the kettle-guys to be a bit more flexible, but

normally they are not as agile as the DHIS2-DEVs :wink:

Best regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Sorry, on a closer look it only seems to work for empty collections. Properties of the object itself (e.g. email, comments of organizationUnits) are still left out.

The floor is open again for other suggestions :slight_smile:

Regards, Uwe

···

Uwe Wahser <uwe@wahser.de> hat am 16. Dezember 2016 um 10:24 geschrieben:

Thanks a lot, Jason, I will try that. I tried 'fields=' expecting this would be the same as 'fields=:all' , which is obviously wrong.

That's a very useful information - do you mind including it into the manual?

Regards, Uwe

---

> Jason Pickering <jason.p.pickering@gmail.com> hat am 16. Dezember 2016 um 10:13 geschrieben:
>
>
> Hi Uwe
> Have a look here
>
> https://ci.dhis2.org/docs/master/en/developer/html/webapi_metadata_field_filter.html
>
> and specifically the use of the ?fields=:all paramater.
>
> Regards,
> Jason
>
>
> On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser <uwe@wahser.de> wrote:
>
> > Dear all,
> >
> > I am having a bit of an issue when querying metadata from the api in json
> > format and further processing them in kettle (Pentaho DI): DHIS2 only
> > returns those properties in the json-package that are not NULL. However,
> > the json converter in kettle needs the full property-set in order to create
> > a tabular structure from the data (don't know the reason for that).
> >
> > Is there a way to force the api to return the full property set, e.g. a
> > secret flag? I'd actually expect to get the full list - even if the
> > property is NULL -, at least when I specify a list of properties in the
> > field-filter ...
> >
> > Of course, I could also ask the kettle-guys to be a bit more flexible, but
> > normally they are not as agile as the DHIS2-DEVs :wink:
> >
> > Best regards,
> >
> > Uwe
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~dhis2-users
> > Post to : dhis2-users@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~dhis2-users
> > More help : https://help.launchpad.net/ListHelp
> >
>
>
>
> --
> Jason P. Pickering
> email: jason.p.pickering@gmail.com
> tel:+46764147049

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

Hi Uwe,
Could you give a specific example from the demo server with an API call, and what is not working as you expect?

Regards,

Jason

···

On Fri, Dec 16, 2016 at 8:48 AM, Uwe Wahser uwe@wahser.de wrote:

Sorry, on a closer look it only seems to work for empty collections. Properties of the object itself (e.g. email, comments of organizationUnits) are still left out.

The floor is open again for other suggestions :slight_smile:

Regards, Uwe

Uwe Wahser uwe@wahser.de hat am 16. Dezember 2016 um 10:24 geschrieben:

Thanks a lot, Jason, I will try that. I tried ‘fields=’ expecting this would be the same as ‘fields=:all’ , which is obviously wrong.

That’s a very useful information - do you mind including it into the manual?

Regards, Uwe


Jason Pickering jason.p.pickering@gmail.com hat am 16. Dezember 2016 um 10:13 geschrieben:

Hi Uwe

Have a look here

https://ci.dhis2.org/docs/master/en/developer/html/webapi_metadata_field_filter.html

and specifically the use of the ?fields=:all paramater.

Regards,

Jason

On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser uwe@wahser.de wrote:

Dear all,

I am having a bit of an issue when querying metadata from the api in json

format and further processing them in kettle (Pentaho DI): DHIS2 only

returns those properties in the json-package that are not NULL. However,

the json converter in kettle needs the full property-set in order to create

a tabular structure from the data (don’t know the reason for that).

Is there a way to force the api to return the full property set, e.g. a

secret flag? I’d actually expect to get the full list - even if the

property is NULL -, at least when I specify a list of properties in the

field-filter …

Of course, I could also ask the kettle-guys to be a bit more flexible, but

normally they are not as agile as the DHIS2-DEVs :wink:

Best regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Hi Jason,

simple example:
https://play.dhis2.org/demo/api/organisationUnits.json?paging=false&filter=level:eq:3&fields=name,comment

What I'd expect would be a json package with name & comment for ALL organisationUnits, even when the comment is NULL. What I get is a comment property for Badjia (which I created for the example), for all the other I only get the name property:
[{"name":"Badjia","comment":"This is a comment."},{"name":"Bagruwa"},{"name":"Baoma"}...]

Converting this package in kettle would result to an error message that 50 (or whatever) instances of name were found, but only 1 instance of comment. kettle expects always the complete set of properties for all objects. Not sure how R or others handle this ...

If there was a query parameter like returnNull=true , that could give back the null properties like {"name":"Bagruwa","comment":""}, it would be possible to work with these packages. Actually I had expected this to be the default behaviour, when specifying fields.

Best regards,

Uwe

···

---

Jason Pickering <jason.p.pickering@gmail.com> hat am 16. Dezember 2016 um 12:05 geschrieben:

Hi Uwe,
Could you give a specific example from the demo server with an API call,
and what is not working as you expect?

Regards,
Jason

On Fri, Dec 16, 2016 at 8:48 AM, Uwe Wahser <uwe@wahser.de> wrote:

> Sorry, on a closer look it only seems to work for empty collections.
> Properties of the object itself (e.g. email, comments of organizationUnits)
> are still left out.
>
> The floor is open again for other suggestions :slight_smile:
>
> Regards, Uwe
>
> > Uwe Wahser <uwe@wahser.de> hat am 16. Dezember 2016 um 10:24
> geschrieben:
> >
> >
> > Thanks a lot, Jason, I will try that. I tried 'fields=' expecting this
> would be the same as 'fields=:all' , which is obviously wrong.
> >
> > That's a very useful information - do you mind including it into the
> manual?
> >
> > Regards, Uwe
> >
> > ---
> >
> > > Jason Pickering <jason.p.pickering@gmail.com> hat am 16. Dezember
> 2016 um 10:13 geschrieben:
> > >
> > >
> > > Hi Uwe
> > > Have a look here
> > >
> > > https://ci.dhis2.org/docs/master/en/developer/html/
> webapi_metadata_field_filter.html
> > >
> > > and specifically the use of the ?fields=:all paramater.
> > >
> > > Regards,
> > > Jason
> > >
> > >
> > > On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser <uwe@wahser.de> wrote:
> > >
> > > > Dear all,
> > > >
> > > > I am having a bit of an issue when querying metadata from the api in
> json
> > > > format and further processing them in kettle (Pentaho DI): DHIS2 only
> > > > returns those properties in the json-package that are not NULL.
> However,
> > > > the json converter in kettle needs the full property-set in order to
> create
> > > > a tabular structure from the data (don't know the reason for that).
> > > >
> > > > Is there a way to force the api to return the full property set,
> e.g. a
> > > > secret flag? I'd actually expect to get the full list - even if the
> > > > property is NULL -, at least when I specify a list of properties in
> the
> > > > field-filter ...
> > > >
> > > > Of course, I could also ask the kettle-guys to be a bit more
> flexible, but
> > > > normally they are not as agile as the DHIS2-DEVs :wink:
> > > >
> > > > Best regards,
> > > >
> > > > Uwe
> > > >
> > > > _______________________________________________
> > > > Mailing list: https://launchpad.net/~dhis2-users
> > > > Post to : dhis2-users@lists.launchpad.net
> > > > Unsubscribe : https://launchpad.net/~dhis2-users
> > > > More help : https://help.launchpad.net/ListHelp
> > > >
> > >
> > >
> > >
> > > --
> > > Jason P. Pickering
> > > email: jason.p.pickering@gmail.com
> > > tel:+46764147049
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~dhis2-users
> > Post to : dhis2-users@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~dhis2-users
> > More help : https://help.launchpad.net/ListHelp
>

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Hi Uwe,

OK, I sort of see the problem you have, but am pretty sure its not a DHIS2 problem. Are you saying that this is new behaviour, and it was not like this before? I just checked an old instance I have which is running 2.17, and it is exactly the same, so not sure this has changed any time in the recent past.

Maybe you should try and use CSV?

https://play.dhis2.org/demo/api/organisationUnits.csv?paging=false&filter=level:eq:3&fields=name,comment

does produce an extra column (with NULLS).

From the R standpoint (as well as other languages I think), its really up to you to parse the payload as you need it, but its handled quite easily for this case in R. Other times, the nested lists need to be flattened, if that is what you require, but I do not see this as DHIS2 issue really. It would make little sense to include all of these NULLs where they do not exist, as it could drastically increase the size of the payload. That is why they are not included, except where not null.

image

···

On Fri, Dec 16, 2016 at 10:41 AM, Uwe Wahser uwe@wahser.de wrote:

Hi Uwe,

Could you give a specific example from the demo server with an API call,

and what is not working as you expect?

Regards,

Jason

On Fri, Dec 16, 2016 at 8:48 AM, Uwe Wahser uwe@wahser.de wrote:

Sorry, on a closer look it only seems to work for empty collections.

Properties of the object itself (e.g. email, comments of organizationUnits)

are still left out.

The floor is open again for other suggestions :slight_smile:

Regards, Uwe

Uwe Wahser uwe@wahser.de hat am 16. Dezember 2016 um 10:24

geschrieben:

Thanks a lot, Jason, I will try that. I tried ‘fields=’ expecting this

would be the same as ‘fields=:all’ , which is obviously wrong.

That’s a very useful information - do you mind including it into the

manual?

Regards, Uwe


Jason Pickering jason.p.pickering@gmail.com hat am 16. Dezember

2016 um 10:13 geschrieben:

Hi Uwe

Have a look here

https://ci.dhis2.org/docs/master/en/developer/html/

webapi_metadata_field_filter.html

and specifically the use of the ?fields=:all paramater.

Regards,

Jason

On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser uwe@wahser.de wrote:

Dear all,

I am having a bit of an issue when querying metadata from the api in

json

format and further processing them in kettle (Pentaho DI): DHIS2 only

returns those properties in the json-package that are not NULL.

However,

the json converter in kettle needs the full property-set in order to

create

a tabular structure from the data (don’t know the reason for that).

Is there a way to force the api to return the full property set,

e.g. a

secret flag? I’d actually expect to get the full list - even if the

property is NULL -, at least when I specify a list of properties in

the

field-filter …

Of course, I could also ask the kettle-guys to be a bit more

flexible, but

normally they are not as agile as the DHIS2-DEVs :wink:

Best regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049

Hi Jason,

simple example:

https://play.dhis2.org/demo/api/organisationUnits.json?paging=false&filter=level:eq:3&fields=name,comment

What I’d expect would be a json package with name & comment for ALL organisationUnits, even when the comment is NULL. What I get is a comment property for Badjia (which I created for the example), for all the other I only get the name property:

[{“name”:“Badjia”,“comment”:“This is a comment.”},{“name”:“Bagruwa”},{“name”:“Baoma”}…]

Converting this package in kettle would result to an error message that 50 (or whatever) instances of name were found, but only 1 instance of comment. kettle expects always the complete set of properties for all objects. Not sure how R or others handle this …

If there was a query parameter like returnNull=true , that could give back the null properties like {“name”:“Bagruwa”,“comment”:""}, it would be possible to work with these packages. Actually I had expected this to be the default behaviour, when specifying fields.

Best regards,

Uwe


Jason Pickering jason.p.pickering@gmail.com hat am 16. Dezember 2016 um 12:05 geschrieben:

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Hi Jason,

It’s not new - I had that problem already last year, but managed to work around. Now I wanted to use more properties and the workaround doesn’t work for that.

I had decided against csv back then, but forgot why. Must have been a good reason, because normally I prefer csv :slight_smile:

I also think that it is not a DHIS issue and that the kettle converter should be able to do that. But DHIS developers are usually more responsive :slight_smile:

I was hoping that it wouldn’t be a big deal, as it works with csv. Implementing it as a parameter would also give the choice of using that feature if needed, thus avoiding inflated payloads per default.

Anyway, I think I’ll enjoy our Xmas party now. Maybe I that gives me better ideas on Monday :smiley:

Thanks for looking at it,

Uwe

http://www.aqua-mail.com

···

On Fri, Dec 16, 2016 at 10:41 AM, Uwe Wahser uwe@wahser.de wrote:

Hi Uwe,

Could you give a specific example from the demo server with an API call,

and what is not working as you expect?

Regards,

Jason

On Fri, Dec 16, 2016 at 8:48 AM, Uwe Wahser uwe@wahser.de wrote:

Sorry, on a closer look it only seems to work for empty collections.

Properties of the object itself (e.g. email, comments of organizationUnits)

are still left out.

The floor is open again for other suggestions :slight_smile:

Regards, Uwe

Uwe Wahser uwe@wahser.de hat am 16. Dezember 2016 um 10:24

geschrieben:

Thanks a lot, Jason, I will try that. I tried ‘fields=’ expecting this

would be the same as ‘fields=:all’ , which is obviously wrong.

That’s a very useful information - do you mind including it into the

manual?

Regards, Uwe


Jason Pickering jason.p.pickering@gmail.com hat am 16. Dezember

2016 um 10:13 geschrieben:

Hi Uwe

Have a look here

https://ci.dhis2.org/docs/master/en/developer/html/

webapi_metadata_field_filter.html

and specifically the use of the ?fields=:all paramater.

Regards,

Jason

On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser uwe@wahser.de wrote:

Dear all,

I am having a bit of an issue when querying metadata from the api in

json

format and further processing them in kettle (Pentaho DI): DHIS2 only

returns those properties in the json-package that are not NULL.

However,

the json converter in kettle needs the full property-set in order to

create

a tabular structure from the data (don’t know the reason for that).

Is there a way to force the api to return the full property set,

e.g. a

secret flag? I’d actually expect to get the full list - even if the

property is NULL -, at least when I specify a list of properties in

the

field-filter …

Of course, I could also ask the kettle-guys to be a bit more

flexible, but

normally they are not as agile as the DHIS2-DEVs :wink:

Best regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049

Hi Jason,

simple example:

https://play.dhis2.org/demo/api/organisationUnits.json?paging=false&filter=level:eq:3&fields=name,comment

What I’d expect would be a json package with name & comment for ALL organisationUnits, even when the comment is NULL. What I get is a comment property for Badjia (which I created for the example), for all the other I only get the name property:

[{“name”:“Badjia”,“comment”:“This is a comment.”},{“name”:“Bagruwa”},{“name”:“Baoma”}…]

Converting this package in kettle would result to an error message that 50 (or whatever) instances of name were found, but only 1 instance of comment. kettle expects always the complete set of properties for all objects. Not sure how R or others handle this …

If there was a query parameter like returnNull=true , that could give back the null properties like {“name”:“Bagruwa”,“comment”:""}, it would be possible to work with these packages. Actually I had expected this to be the default behaviour, when specifying fields.

Best regards,

Uwe


Jason Pickering jason.p.pickering@gmail.com hat am 16. Dezember 2016 um 12:05 geschrieben:

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Dear Uwe,

This is a json issue. Recently i had guidance from lars that the xml returns even the empty properties.

Apparently, suggestions in the link below talks of leaving out properties with null values.

https://google.github.io/styleguide/jsoncstyleguide.xml

The only work around:

  1. Try using xml format ( if ok with U)

  2. You will need to take back missing properties within your codes ( probably your previous work around)

···

On Dec 16, 2016 15:33, “Uwe Wahser” uwe@wahser.de wrote:

Hi Jason,

It’s not new - I had that problem already last year, but managed to work around. Now I wanted to use more properties and the workaround doesn’t work for that.

I had decided against csv back then, but forgot why. Must have been a good reason, because normally I prefer csv :slight_smile:

I also think that it is not a DHIS issue and that the kettle converter should be able to do that. But DHIS developers are usually more responsive :slight_smile:

I was hoping that it wouldn’t be a big deal, as it works with csv. Implementing it as a parameter would also give the choice of using that feature if needed, thus avoiding inflated payloads per default.

Anyway, I think I’ll enjoy our Xmas party now. Maybe I that gives me better ideas on Monday :smiley:

Thanks for looking at it,

Uwe

Sent with AquaMail for Android

http://www.aqua-mail.com

On 16 December 2016 12:54:17 Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Uwe,

OK, I sort of see the problem you have, but am pretty sure its not a DHIS2 problem. Are you saying that this is new behaviour, and it was not like this before? I just checked an old instance I have which is running 2.17, and it is exactly the same, so not sure this has changed any time in the recent past.

Maybe you should try and use CSV?

https://play.dhis2.org/demo/api/organisationUnits.csv?paging=false&filter=level:eq:3&fields=name,comment

does produce an extra column (with NULLS).

From the R standpoint (as well as other languages I think), its really up to you to parse the payload as you need it, but its handled quite easily for this case in R. Other times, the nested lists need to be flattened, if that is what you require, but I do not see this as DHIS2 issue really. It would make little sense to include all of these NULLs where they do not exist, as it could drastically increase the size of the payload. That is why they are not included, except where not null.


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

On Fri, Dec 16, 2016 at 10:41 AM, Uwe Wahser uwe@wahser.de wrote:

Hi Uwe,

Could you give a specific example from the demo server with an API call,

and what is not working as you expect?

Regards,

Jason

On Fri, Dec 16, 2016 at 8:48 AM, Uwe Wahser uwe@wahser.de wrote:

Sorry, on a closer look it only seems to work for empty collections.

Properties of the object itself (e.g. email, comments of organizationUnits)

are still left out.

The floor is open again for other suggestions :slight_smile:

Regards, Uwe

Uwe Wahser uwe@wahser.de hat am 16. Dezember 2016 um 10:24

geschrieben:

Thanks a lot, Jason, I will try that. I tried ‘fields=’ expecting this

would be the same as ‘fields=:all’ , which is obviously wrong.

That’s a very useful information - do you mind including it into the

manual?

Regards, Uwe


Jason Pickering jason.p.pickering@gmail.com hat am 16. Dezember

2016 um 10:13 geschrieben:

Hi Uwe

Have a look here

https://ci.dhis2.org/docs/master/en/developer/html/

webapi_metadata_field_filter.html

and specifically the use of the ?fields=:all paramater.

Regards,

Jason

On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser uwe@wahser.de wrote:

Dear all,

I am having a bit of an issue when querying metadata from the api in

json

format and further processing them in kettle (Pentaho DI): DHIS2 only

returns those properties in the json-package that are not NULL.

However,

the json converter in kettle needs the full property-set in order to

create

a tabular structure from the data (don’t know the reason for that).

Is there a way to force the api to return the full property set,

e.g. a

secret flag? I’d actually expect to get the full list - even if the

property is NULL -, at least when I specify a list of properties in

the

field-filter …

Of course, I could also ask the kettle-guys to be a bit more

flexible, but

normally they are not as agile as the DHIS2-DEVs :wink:

Best regards,

Uwe


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049


Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

Jason P. Pickering

email: jason.p.pickering@gmail.com

tel:+46764147049

Hi Jason,

simple example:

https://play.dhis2.org/demo/api/organisationUnits.json?paging=false&filter=level:eq:3&fields=name,comment

What I’d expect would be a json package with name & comment for ALL organisationUnits, even when the comment is NULL. What I get is a comment property for Badjia (which I created for the example), for all the other I only get the name property:

[{“name”:“Badjia”,“comment”:“This is a comment.”},{“name”:“Bagruwa”},{“name”:“Baoma”}…]

Converting this package in kettle would result to an error message that 50 (or whatever) instances of name were found, but only 1 instance of comment. kettle expects always the complete set of properties for all objects. Not sure how R or others handle this …

If there was a query parameter like returnNull=true , that could give back the null properties like {“name”:“Bagruwa”,“comment”:""}, it would be possible to work with these packages. Actually I had expected this to be the default behaviour, when specifying fields.

Best regards,

Uwe


Jason Pickering jason.p.pickering@gmail.com hat am 16. Dezember 2016 um 12:05 geschrieben:


Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

Hi Jason & Ocaya,

thanks for your input. Just to follow up: Kettle just released a new version (7.0) last month. According to their forum it includes a fix for the bug. Didn't have time to try this out, yet, but that seems to be the solution then.

For the record: the xml api doesn't seem to bring the full feature set either. Also I feel that the overhead of redundant strings is much higher than in json. I also decided against csv back then, because I couldn't figure out how to return deep structures (e.g. Parent ID of an OrgUnit).

Have a wonderful Christmas (if you have) and a great start into the new year!

Regards - Uwe

···

---

Am 16.12.2016 um 22:56 schrieb Ocaya Stephen:

Dear Uwe,

This is a json issue. Recently i had guidance from lars that the xml returns even the empty properties.

Apparently, suggestions in the link below talks of leaving out properties with null values.

https://google.github.io/styleguide/jsoncstyleguide.xml

The only work around:

1. Try using xml format ( if ok with U)
2. You will need to take back missing properties within your codes ( probably your previous work around)

On Dec 16, 2016 15:33, "Uwe Wahser" <uwe@wahser.de > <mailto:uwe@wahser.de>> wrote:

    Hi Jason,

    It's not new - I had that problem already last year, but managed
    to work around. Now I wanted to use more properties and the
    workaround doesn't work for that.

    I had decided against csv back then, but forgot why. Must have
    been a good reason, because normally I prefer csv :slight_smile:

    I also think that it is not a DHIS issue and that the kettle
    converter should be able to do that. But DHIS developers are
    usually more responsive :slight_smile:

    I was hoping that it wouldn't be a big deal, as it works with csv.
    Implementing it as a parameter would also give the choice of using
    that feature if needed, thus avoiding inflated payloads per default.

    Anyway, I think I'll enjoy our Xmas party now. Maybe I that gives
    me better ideas on Monday :smiley:

    Thanks for looking at it,

    Uwe

    Sent with AquaMail for Android
    http://www.aqua-mail.com

    On 16 December 2016 12:54:17 Jason Pickering > <jason.p.pickering@gmail.com <mailto:jason.p.pickering@gmail.com>> > wrote:

    Hi Uwe,

    OK, I sort of see the problem you have, but am pretty sure its
    not a DHIS2 problem. Are you saying that this is new behaviour,
    and it was not like this before? I just checked an old instance
    I have which is running 2.17, and it is exactly the same, so not
    sure this has changed any time in the recent past.

    Maybe you should try and use CSV?

    https://play.dhis2.org/demo/api/organisationUnits.csv?paging=false&filter=level:eq:3&fields=name,comment
    <https://play.dhis2.org/demo/api/organisationUnits.csv?paging=false&filter=level:eq:3&fields=name,comment>

    does produce an extra column (with NULLS).

    From the R standpoint (as well as other languages I think), its
    really up to you to parse the payload as you need it, but its
    handled quite easily for this case in R. Other times, the nested
    lists need to be flattened, if that is what you require, but I do
    not see this as DHIS2 issue really. It would make little sense
    to include all of these NULLs where they do not exist, as it
    could drastically increase the size of the payload. That is why
    they are not included, except where not null.

    Inline image 1

    On Fri, Dec 16, 2016 at 10:41 AM, Uwe Wahser <uwe@wahser.de >> <mailto:uwe@wahser.de>> wrote:

        Hi Jason,

        simple example:
        https://play.dhis2.org/demo/api/organisationUnits.json?paging=false&filter=level:eq:3&fields=name,comment
        <https://play.dhis2.org/demo/api/organisationUnits.json?paging=false&filter=level:eq:3&fields=name,comment>

        What I'd expect would be a json package with name & comment
        for ALL organisationUnits, even when the comment is NULL.
        What I get is a comment property for Badjia (which I created
        for the example), for all the other I only get the name property:
        [{"name":"Badjia","comment":"This is a
        comment."},{"name":"Bagruwa"},{"name":"Baoma"}...]

        Converting this package in kettle would result to an error
        message that 50 (or whatever) instances of name were found,
        but only 1 instance of comment. kettle expects always the
        complete set of properties for all objects. Not sure how R or
        others handle this ...

        If there was a query parameter like returnNull=true , that
        could give back the null properties like
        {"name":"Bagruwa","comment":""}, it would be possible to work
        with these packages. Actually I had expected this to be the
        default behaviour, when specifying fields.

        Best regards,

        Uwe

        ---

        > Jason Pickering <jason.p.pickering@gmail.com
        <mailto:jason.p.pickering@gmail.com>> hat am 16. Dezember
        2016 um 12:05 geschrieben:
        >
        > Hi Uwe,
        > Could you give a specific example from the demo server with
        an API call,
        > and what is not working as you expect?
        >
        > Regards,
        > Jason
        >
        > On Fri, Dec 16, 2016 at 8:48 AM, Uwe Wahser <uwe@wahser.de >> <mailto:uwe@wahser.de>> wrote:
        >
        > > Sorry, on a closer look it only seems to work for empty
        collections.
        > > Properties of the object itself (e.g. email, comments of
        organizationUnits)
        > > are still left out.
        > >
        > > The floor is open again for other suggestions :slight_smile:
        > >
        > > Regards, Uwe
        > >
        > > > Uwe Wahser <uwe@wahser.de <mailto:uwe@wahser.de>> hat
        am 16. Dezember 2016 um 10:24 >> > > geschrieben:
        > > >
        > > > Thanks a lot, Jason, I will try that. I tried 'fields='
        expecting this
        > > would be the same as 'fields=:all' , which is obviously
        wrong.
        > > >
        > > > That's a very useful information - do you mind
        including it into the
        > > manual?
        > > >
        > > > Regards, Uwe
        > > >
        > > > ---
        > > >
        > > > > Jason Pickering <jason.p.pickering@gmail.com
        <mailto:jason.p.pickering@gmail.com>> hat am 16. Dezember
        > > 2016 um 10:13 geschrieben:
        > > > >
        > > > > Hi Uwe
        > > > > Have a look here
        > > > >
        > > > > https://ci.dhis2.org/docs/master/en/developer/html/
        <https://ci.dhis2.org/docs/master/en/developer/html/>
        > > webapi_metadata_field_filter.html
        > > > >
        > > > > and specifically the use of the ?fields=:all paramater.
        > > > >
        > > > > Regards,
        > > > > Jason
        > > > >
        > > > > On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser >> <uwe@wahser.de <mailto:uwe@wahser.de>> wrote:
        > > > >
        > > > > > Dear all,
        > > > > >
        > > > > > I am having a bit of an issue when querying
        metadata from the api in
        > > json
        > > > > > format and further processing them in kettle
        (Pentaho DI): DHIS2 only
        > > > > > returns those properties in the json-package that
        are not NULL.
        > > However,
        > > > > > the json converter in kettle needs the full
        property-set in order to
        > > create
        > > > > > a tabular structure from the data (don't know the
        reason for that).
        > > > > >
        > > > > > Is there a way to force the api to return the full
        property set,
        > > e.g. a
        > > > > > secret flag? I'd actually expect to get the full
        list - even if the
        > > > > > property is NULL -, at least when I specify a list
        of properties in
        > > the
        > > > > > field-filter ...
        > > > > >
        > > > > > Of course, I could also ask the kettle-guys to be a
        bit more
        > > flexible, but
        > > > > > normally they are not as agile as the DHIS2-DEVs :wink:
        > > > > >
        > > > > > Best regards,
        > > > > >
        > > > > > Uwe
        > > > > >
        > > > > > _______________________________________________
        > > > > > Mailing list: https://launchpad.net/~dhis2-users
        <https://launchpad.net/~dhis2-users>
        > > > > > Post to : dhis2-users@lists.launchpad.net
        <mailto:dhis2-users@lists.launchpad.net>
        > > > > > Unsubscribe : https://launchpad.net/~dhis2-users
        <https://launchpad.net/~dhis2-users>
        > > > > > More help : https://help.launchpad.net/ListHelp
        <https://help.launchpad.net/ListHelp>
        > > > > >
        > > > >
        > > > > --
        > > > > Jason P. Pickering
        > > > > email: jason.p.pickering@gmail.com
        <mailto:jason.p.pickering@gmail.com>
        > > > > tel:+46764147049 <tel:%2B46764147049>
        > > >
        > > > _______________________________________________
        > > > Mailing list: https://launchpad.net/~dhis2-users
        <https://launchpad.net/~dhis2-users>
        > > > Post to : dhis2-users@lists.launchpad.net
        <mailto:dhis2-users@lists.launchpad.net>
        > > > Unsubscribe : https://launchpad.net/~dhis2-users
        <https://launchpad.net/~dhis2-users>
        > > > More help : https://help.launchpad.net/ListHelp
        <https://help.launchpad.net/ListHelp>
        > >
        >
        > --
        > Jason P. Pickering
        > email: jason.p.pickering@gmail.com
        <mailto:jason.p.pickering@gmail.com>
        > tel:+46764147049 <tel:%2B46764147049>

    -- Jason P. Pickering
    email: jason.p.pickering@gmail.com
    <mailto:jason.p.pickering@gmail.com>
    tel:+46764147049 <tel:+46%2076%20414%2070%2049>

    _______________________________________________
    Mailing list: https://launchpad.net/~dhis2-users
    <https://launchpad.net/~dhis2-users>
    Post to : dhis2-users@lists.launchpad.net
    <mailto:dhis2-users@lists.launchpad.net>
    Unsubscribe : https://launchpad.net/~dhis2-users
    <https://launchpad.net/~dhis2-users>
    More help : https://help.launchpad.net/ListHelp
    <https://help.launchpad.net/ListHelp>