posting dhis2 metadata

jason · 15 February 2018 05:02

Hi Isaiah,

Please post your question back to the user list and I will answer there.

Regards,

Jason

···

On Thu, Feb 15, 2018 at 5:45 AM, Isaiah Nyabuto isaiahnyabuto@gmail.com wrote:

Hi Jason,

I am following your guide on bootstrapping dhis2. Please see my steps below and assist.

Thank you.
---------- Forwarded message ----------
From: Isaiah Nyabuto isaiahnyabuto@gmail.com
Date: Thu, Feb 15, 2018 at 7:40 AM
Subject: Re: [Dhis2-users] posting dhis2 metadata
To: Aaron C White aaroncwhite@gmail.com

Hi Aaron,

Thank you. I am also following Jason examples on bootstrapping dhis2. I am using httr.

Firstly, this is how my original df looks like.

head(org)                           name uid code parent shortname 1 Wikithuki Dispensary Bc371uxfYD5 16925 vpiZlB5cdK7 Wikithuki Dispensary 2 Gemifa Medical Clinic Z7Lf1n3Fcpi 22907 mEZBjj4fLBT Gemifa 3 Olympus Medical Clinic NAbQdHuaJFK 22906 PeEcMfBchbr Olympus 4 Masaba Community Health Unit tYwHeLSnHyb 600616 QR2K4k8VKF1 Masaba CHU 5 Gatunga Health Centre cmay4WXRmhA 12034 IcAegRcGuV6 Gatunga Health Centre 6 Cfw Kimangaru Clinic E52UsoBxJVS 11966 vEAFtHnPDrX CFW Kimangaru Clinic description openingdate closedate comment featuretype coordinates url 1 NA NA POINT [37.95263,-0.77819] 2 Clinic NA NA NONE [34.391914,0.059575] 3 Clinic NA NA NONE [34.3994,0.059575] 4 Community Unit NA NA NONE [34.46411,0.19358] 5 NA NA POINT [37.9423,-0.29656] 6 NA NA POINT [37.53009,-0.54506] contactperson address email phonenumber 1 NA NA NA NA 2 NA NA NA NA 3 NA NA NA NA 4 NA NA NA NA 5 NA NA NA NA 6 NA NA NA NA

I followed the following steps to prepare my df for import.

Removed all the NAs and columns with entire NAs.

head(new_org)                           name uid code parent shortname 1 Wikithuki Dispensary Bc371uxfYD5 16925 vpiZlB5cdK7 Wikithuki Dispensary 2 Gemifa Medical Clinic Z7Lf1n3Fcpi 22907 mEZBjj4fLBT Gemifa 3 Olympus Medical Clinic NAbQdHuaJFK 22906 PeEcMfBchbr Olympus 4 Masaba Community Health Unit tYwHeLSnHyb 600616 QR2K4k8VKF1 Masaba CHU 5 Gatunga Health Centre cmay4WXRmhA 12034 IcAegRcGuV6 Gatunga Health Centre 6 Cfw Kimangaru Clinic E52UsoBxJVS 11966 vEAFtHnPDrX CFW Kimangaru Clinic description comment featuretype coordinates url 1 POINT [37.95263,-0.77819] 2 Clinic NONE [34.391914,0.059575] 3 Clinic NONE [34.3994,0.059575] 4 Community Unit NONE [34.46411,0.19358] 5 POINT [37.9423,-0.29656] 6 POINT [37.53009,-0.54506]

Then post the data as follows.

dest.url ← “http://localhost/dhis/”

url ← paste0(dest.url, “api/metadata?importStrategy=CREATE&atomicMode=NONE”)

r ← POST(url, authenticate(dest.user, dest.pass),

body = toJSON(new_org, auto_unbox = TRUE),

content_type_json())

assertthat::assert_that(r$status_code==200)

The body looks like.

jsonlite::toJSON(me, auto_unbox = TRUE) [{"name":"Wikithuki Dispensary","uid":"Bc371uxfYD5","code":"16925","parent":"vpiZlB5cdK7","shortname":"Wikithuki Dispensary","description":"","comment":"","featuretype":"POINT","coordinates":"[37.95263,-0.77819]","url":""},{"name":"Gemifa Medical Clinic","uid":"Z7Lf1n3Fcpi","code":"22907","parent":"mEZBjj4fLBT","shortname":"Gemifa","description":"Clinic","comment":"","featuretype":"NONE","coordinates":"[34.391914,0.059575]","url":""},{"name":"Olympus Medical Clinic","uid":"NAbQdHuaJFK","code":"22906","parent":"PeEcMfBchbr","shortname":"Olympus","description":"Clinic","comment":"","featuretype":"NONE","coordinates":"[34.3994,0.059575]","url":""},{"name":"Masaba Community Health Unit","uid":"tYwHeLSnHyb","code":"600616","parent":"QR2K4k8VKF1","shortname":"Masaba CHU","description":"Community Unit","comment":"","featuretype":"NONE","coordinates":"[34.46411,0.19358]","url":""},{"name":"Gatunga Health Centre","uid":"cmay4WXRmhA","code":"12034","parent":"IcAegRcGuV6","shortname":"Gatunga Health Centre","description":"","comment":"","featuretype":"POINT","coordinates":"[37.9423,-0.29656]","url":""},{"name":"Cfw Kimangaru Clinic","uid":"E52UsoBxJVS","code":"11966","parent":"vEAFtHnPDrX","shortname":"CFW Kimangaru Clinic","description":"","comment":"","featuretype":"POINT","coordinates":"[37.53009,-0.54506]","url":""}]

What could be the problem here? I get status 200 and still it doesn’t import.

Any helps will be appreciated.

Thank you

Isaiah Nyabuto
+254 729 572 616

Bsc. Health Records Management & Informatics, JKUAT.

–

–
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

On Thu, Feb 15, 2018 at 1:41 AM, Aaron C White aaroncwhite@gmail.com wrote:

Oh, and just a side note: the metadata endpoint will respond with a 200 always I believe. The response will give you an import summary that describes if it actually accepted the payload or not.

Isaiah Nyabuto
+254 729 572 616

Bsc. Health Records Management & Informatics, JKUAT.

–

Aaron C. White

919.522.5585

On Wed, Feb 14, 2018 at 4:49 PM, Isaiah Nyabuto isaiahnyabuto@gmail.com wrote:

Dear Aaron,

I am following your guide to on bootstrapping dhis2 with R.

I am trying to post organisation units metadata to my dhis2 instance. My payload looks like; “api/metadata?import strategy=FALSE&atomicMode=None”

I don’t seam to get status 201. I get code 200, with no metadata posted.

What could be wrong ?

Thank you.

==========================
Isaiah Nyabuto
+254729572616
Bsc. Health Records Management and Informatics

On Jan 29, 2018 21:01, “Aaron C White” aaroncwhite@gmail.com wrote:

Isaiah,

I’ve found the best way is to convert rows of a CSV into JSON structure and then post to the appropriate endpoint using httr.

I’ve made a slew of helper functions that make it easy for me to do along with some markdown files starting to explain things. When I started working with dhis2, I was an analyst and most comfortable using dataframes, so this kind of fits that perspective. I’ve learned a lot along the way and plan to refactor things at some point. Eventually, I’d like to make it a real R package- just haven’t had the time!

https://github.com/aaroncwhite/dhis2R

Feel free to PM me for more detail/explanation.

Aaron

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

–

Aaron C. White

919.522.5585

On Mon, Jan 29, 2018 at 12:24 PM, Jason Pickering jason.p.pickering@gmail.com wrote:

Hi Isaiah,

Yes, R is definitely capable of creating a file which you can import into DHIS2.

If you want to use CSV, be sure to have your file with the "row.names=FALSE " parameter and be sure to convert all “NA” values to with " na = ‘’ "

You can also post the file directly to DHIS2. Have a look at https://github.com/jason-p-pickering/dhis2-data-munging/blob/master/bootstrap/bootstrap.Rmd or more details of how to do this in the section “Loading the data elements”

Regards,

Jason

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp

On Mon, Jan 29, 2018 at 6:04 PM, Timothy Harding tharding@baosystems.com wrote:

Isaiah:

org_kqmh_list_test.csv:

Has an extra column

1, 2, 3, etc… should not be in there unless the OUs are named 1, 2, 3.

Probably should be LF instead of CRLF:

but I’m not sure it makes a difference here.

Also you’ve got a lot of NA’s in there, not sure those are needed unless you are setting something to be NA, best just have , rather than ,NA,

Good luck!

Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+46764147049

–

Timothy Harding
Sr. Systems Analyst, BAO Systems

+1 202-536-1541 | tharding@baosystems.com | http://www.baosystems.com | Skype: hardingt@gmail.com | 2900 K Street, Suite 406, Washington D.C. 20007

On Mon, Jan 29, 2018 at 11:30 AM, Isaiah Nyabuto isaiahnyabuto@gmail.com wrote:

Hi all,

Thanks Timothy for sharing this. I am trying to import organisation units metadata in csv file with UTF-8 encoding. When I import the file with Metadata import in dhis2, the file is not imported at all and I do get the summary report.

I wrote the csv file with R function write.table.

Its funny enough that when I write the same file using Excel and save as csv file with UTF-8, it works well.

I do not understand what could be the problem here, has anyone tried to write a csv file with an R function that can successfully be imported in the dhis2?

Looking for your help. I have attached my csv files in this email.

Thanks

==========================
Isaiah Nyabuto
+254729572616
Bsc. Health Records Management and Informatics

On Jan 17, 2018 17:51, “Thomas Lavreys” thomaslavreys@gmail.com wrote:

Thanks for the in depth info! It will surely help along the way. I’ve only started with some testers, but I will probably run into some more issues when I load my real data.

Thanks again!!

Kind regards,

Thomas

From: Timothy Harding [mailto:tharding@baosystems.com]
Sent: Wednesday, January 17, 2018 3:35 PM
To: Thomas Lavreys thomaslavreys@gmail.com
Cc: Adebusoye Anifalaje busoye@baosystems.com; dhis2-users dhis2-users@lists.launchpad.net
Subject: Re: [Dhis2-users] Data element not found or not accessible

Hello Thomas!

I am glad you have solved your issue! I noticed a few gotchas so here are a couple of things to note when importing csv files in DHIS 2:

The first row

dataelement Data element not found or not accessible

is a ** funny error**, because it is actually telling you something about a single row, found here in the documentation:

"dataelement","period","orgunit","catoptcombo","attroptcombo","value","storedby","lastupd","comment"

DHIS 2 is literally complaining about the fact that it cannot find a data element with the uid “dataelement”, which is what is assumes you are trying to load in that top row. You can try yourself by loading a file with only this top line, the error will be the same, 1 ignored row, data element not found or not accessible. It is safe to ignore this error, I’d be worried if there was a UID called dataelement, which is actually a valid UID! Just extremely unlikely to pop up from the random UID generator.

Ignored

Unlike the metadata importer, the data importer will report that it ignored every row on a dry run. It can make testing difficult because you don’t know if it will actually import correctly or not as it reports back that every row was ignored. Two rules of thumb here:

The actual errored out row will be listed in the “Conflicts” section, even in a dry run

Sadly, some errors you can only find on a real run, so always upload your files with a comment, so you can find them later via the DB to delete them if something goes wrong and all else fails

File Type

UTF-8 works well and I use it for all my CSV uploads. Be wary of EXCEL’s CSV saving habits though, especially on a Mac. Your file should be UTF-8 without a BOM (a byte order marker, DHIS 2 doesn’t really care, as it will error out on row one anyway (see above)) AND it should be standard unix line feeds (LF). Excel for the Mac will default to saving it with circa 1990’s CRs only AND a BOM (unless you are editing a CSV, in which case it will keep the settings it found when it opened it). Easiest way to check is to grab a copy of Notepad++ (windows) or BBedit (mac).

BBedit:

and

Notepad++

and

Hope this helps!

Timothy Harding
Sr. Systems Analyst, BAO Systems

+1 202-536-1541 | tharding@baosystems.com | http://www.baosystems.com | Skype: hardingt@gmail.com | 2900 K Street, Suite 406, Washington D.C. 20007

On Wed, Jan 17, 2018 at 9:11 AM, Thomas Lavreys thomaslavreys@gmail.com wrote:

I found the issue. The file was saved as a csv utf-8, which is different to a normal csv (aparrently).

Thanks for the advice!

Kind regards,

Thomas

From: Adebusoye Anifalaje [mailto:busoye@baosystems.com]
Sent: Wednesday, January 17, 2018 12:31 PM

To: Thomas Lavreys thomaslavreys@gmail.com
Cc: dhis2-users@lists.launchpad.net
Subject: Re: [Dhis2-users] Data element not found or not accessible

Hi Thomas,

I aslo see that you are using version 2.23, it is possible that the csv column structure is different for this version. There might be someone on this forum who is currently working on this version and can provide input on that.

Busoye Anifalaje (PhD)
Director of Services (Principal), BAO Systems

UK: +44 7901-740-757 | US: +1 682-307-0986|

busoye@baosystems.com | http://www.baosystems.com |

Skype: busoye | 2900 K Street, Suite 406, Washington D.C. 20007

On 17 Jan 2018, at 12:23, Thomas Lavreys thomaslavreys@gmail.com wrote:

I tried it anyway, and it doesn’t recognize it:

Import summary

Import count

Type

Count

Imported

0

Updated

0

Deleted

0

Ignored

13

Conflicts

Element

Description

dataelement

Data element not found or not accessible

bjDvmb4bfuf

Category option combo not found or not accessible

So I’m guessing I need to find the catoptioncombo ID somewhere.

Regards,

Thomas

From: Adebusoye Anifalaje [mailto:busoye@baosystems.com]
Sent: Wednesday, January 17, 2018 12:11 PM
To: Thomas Lavreys thomaslavreys@gmail.com
Cc: dhis2-users@lists.launchpad.net
Subject: Re: [Dhis2-users] Data element not found or not accessible

Hi Thomas,

Can you try specifying the catoptioncombo even if it is default? This was an issue with one of the version builds a while ago.

Let’s know if that helps.

Cheers

**
<image001.png>
**

Busoye Anifalaje (PhD)
Director of Services (Principal), BAO Systems

UK: +44 7901-740-757 | US: +1 682-307-0986|

busoye@baosystems.com | http://www.baosystems.com |

Skype: busoye | 2900 K Street, Suite 406, Washington D.C. 20007

On 17 Jan 2018, at 12:02, Thomas Lavreys thomaslavreys@gmail.com wrote:

Dear,

I get the message “Data element not found or not accessible” when trying to import a csv file. I looked at some old threads, but failed to find an answer.

The csv file is attached (with headers, but it gave the same answer without headers).

Attached also the API page of the data element, the sharing of the data element (I have superuser access), and the orgunit page to confirm the ID. Also a screenshot of the import summary and my DHIS2 instance info.

I guess I might be missing something but I can’t figure out what exactly.

Thank you in advance for your help.

Kind regards,

Thomas Lavreys

Thomas Lavreys, MSc, PgC Trop Med

Sustainability Department SOCFINCO

Maisha Consulting bvba

Bisschoppenhof 9

3150 Tildonk

Belgium

**

Tel Socfin: +32-25-080050

Tel Maisha: +32-16-850242

Cell phone: +32-474-717641

Fax: +32-16-850242

<TEST.csv>_______________________________________________
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help : https://help.launchpad.net/ListHelp

Mailing list: https://launchpad.net/~dhis2-users

Post to : dhis2-users@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-users

More help : https://help.launchpad.net/ListHelp