async metadata import in 2.27

Hi everyone,

I am trying to import xml metadata (org units) into 2.27, and async=true query parameter does not seem to make a difference - and it still goes in synchronously. It is in the documentation for 2.27 (and 2.26 and 2.25 - https://docs.dhis2.org/2.26/en/developer/html/dhis2_developer_manual_full.html#webapi_metadata_import), and I am using api version 26. Is it something i am doing wrong?

This is the curl command i am using:

curl -X POST ‘https://domain/api/26/metadata?atomicMode=NONE&async=true’ --data-binary @file.xml -H ‘Content-Type:application/xml’

Thanks!

Vlad

Hi

It should work like that. Would you mind writing a JIRA issue? jira.dhis2.org

···

On Thu, Aug 3, 2017 at 12:03 PM, Vladimer Shioshvili vshioshvili@gmail.com wrote:

Hi everyone,

I am trying to import xml metadata (org units) into 2.27, and async=true query parameter does not seem to make a difference - and it still goes in synchronously. It is in the documentation for 2.27 (and 2.26 and 2.25 - https://docs.dhis2.org/2.26/en/developer/html/dhis2_developer_manual_full.html#webapi_metadata_import), and I am using api version 26. Is it something i am doing wrong?

This is the curl command i am using:

curl -X POST ‘https://domain/api/26/metadata?atomicMode=NONE&async=true’ --data-binary @file.xml -H ‘Content-Type:application/xml’

Thanks!

Vlad


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

Hi Morten,

I did create a JIRA ticket (https://jira.dhis2.org/browse/DHIS2-2023) and have not heard back, i really don’t think it’s working as advertised/intended, unless I am the one doing something wrong with my calls:

  1. I get the OU hierarchy from demo site:

curl -u ‘admin:district’ -X GET ‘https://play.dhis2.org/demo/api/26/metadata.xml?organisationUnits=true’ > play_ou.xml

  1. i use the same file to upload:

curl -u ‘admin:district’ -H ‘Content-Type:application/xml’ -X POST ‘https://play.dhis2.org/demo/api/26/metadata?atomicMode=NONE&async=true’ --data-binary @play_ou.xml -v

it takes about 45 seconds to come back with this:

  • Trying 52.30.174.183…

  • Connected to play.dhis2.org (52.30.174.183) port 443 (#0)

  • TLS 1.2 connection using TLS_ECDHE_RSA_WITH_RC4_128_SHA

  • Server certificate: play.dhis2.org

  • Server certificate: RapidSSL SHA256 CA - G3

  • Server certificate: GeoTrust Global CA

  • Server auth using Basic with user ‘admin’

POST /demo/api/26/metadata?atomicMode=NONE&async=true HTTP/1.1

Host: play.dhis2.org

Authorization: Basic YWRtaW46ZGlzdHJpY3Q=

User-Agent: curl/7.43.0

Accept: /

Content-Type:application/xml

Content-Length: 1568899

Expect: 100-continue

< HTTP/1.1 100 Continue

  • We are completely uploaded and fine

< HTTP/1.1 204 No Content

< Server: nginx/1.4.6 (Ubuntu)

< Date: Tue, 22 Aug 2017 10:46:17 GMT

< Connection: keep-alive

< X-XSS-Protection: 1; mode=block

< X-Frame-Options: SAMEORIGIN

< X-Content-Type-Options: nosniff

< Set-Cookie: JSESSIONID=3A6F3850230C09C81BDB67310398FA68; Path=/demo/; HttpOnly

<

After I get the response back, when i do the status: curl -u ‘admin:district’ -X GET https://play.dhis2.org/demo/api/26/system/tasks/METADATA_IMPORT

I get this back:

[{“uid”:“lqOCBbHOTew”,“level”:“INFO”,“category”:“METADATA_IMPORT”,“time”:“2017-08-22T10:46:17.866”,“message”:“(admin) Import:Done took 43.66 seconds”,“completed”:true},{“uid”:“y2dBsPZrwMX”,“level”:“INFO”,“category”:“METADATA_IMPORT”,“time”:“2017-08-22T10:45:34.428”,“message”:“(admin) Updating 1332 object(s) of type OrganisationUnit”,“completed”:false},{“uid”:“GuGaKcJAToU”,“level”:“INFO”,“category”:“METADATA_IMPORT”,“time”:“2017-08-22T10:45:34.209”,“message”:“(admin) Import:Start”,“completed”:false}]

It takes about 43 seconds to import the file - POST does not come back until import is actually done. Documentation has this:

async
false, true
Asynchronous import, returns immediately with a Location header pointing to the location of the importReport.

And there is no location returned with the 204 (which is not the right HTTP code anyway).

I am also concerned about the time - when doing this with some PEPFAR countries, hierarchy with 14 thousand OUs took 8 hours to import.

Thanks,

Vlad

···

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Aug 3, 2017 at 12:03 PM, Vladimer Shioshvili vshioshvili@gmail.com wrote:

Hi everyone,

I am trying to import xml metadata (org units) into 2.27, and async=true query parameter does not seem to make a difference - and it still goes in synchronously. It is in the documentation for 2.27 (and 2.26 and 2.25 - https://docs.dhis2.org/2.26/en/developer/html/dhis2_developer_manual_full.html#webapi_metadata_import), and I am using api version 26. Is it something i am doing wrong?

This is the curl command i am using:

curl -X POST ‘https://domain/api/26/metadata?atomicMode=NONE&async=true’ --data-binary @file.xml -H ‘Content-Type:application/xml’

Thanks!

Vlad


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Hi Vlad

Yes, this is on the list of todo :wink: hopefully should have time this week (or it will have to wait for next week)

···

On Tue, Aug 22, 2017 at 12:53 PM, Vladimer Shioshvili vshioshvili@gmail.com wrote:

Hi Morten,

I did create a JIRA ticket (https://jira.dhis2.org/browse/DHIS2-2023) and have not heard back, i really don’t think it’s working as advertised/intended, unless I am the one doing something wrong with my calls:

  1. I get the OU hierarchy from demo site:

curl -u ‘admin:district’ -X GET ‘https://play.dhis2.org/demo/api/26/metadata.xml?organisationUnits=true’ > play_ou.xml

  1. i use the same file to upload:

curl -u ‘admin:district’ -H ‘Content-Type:application/xml’ -X POST ‘https://play.dhis2.org/demo/api/26/metadata?atomicMode=NONE&async=true’ --data-binary @play_ou.xml -v

it takes about 45 seconds to come back with this:

  • Trying 52.30.174.183…
  • TLS 1.2 connection using TLS_ECDHE_RSA_WITH_RC4_128_SHA
  • Server certificate: RapidSSL SHA256 CA - G3
  • Server certificate: GeoTrust Global CA
  • Server auth using Basic with user ‘admin’

POST /demo/api/26/metadata?atomicMode=NONE&async=true HTTP/1.1

Host: play.dhis2.org

Authorization: Basic YWRtaW46ZGlzdHJpY3Q=

User-Agent: curl/7.43.0

Accept: /

Content-Type:application/xml

Content-Length: 1568899

Expect: 100-continue

< HTTP/1.1 100 Continue

  • We are completely uploaded and fine

< HTTP/1.1 204 No Content

< Server: nginx/1.4.6 (Ubuntu)

< Date: Tue, 22 Aug 2017 10:46:17 GMT

< Connection: keep-alive

< X-XSS-Protection: 1; mode=block

< X-Frame-Options: SAMEORIGIN

< X-Content-Type-Options: nosniff

< Set-Cookie: JSESSIONID=3A6F3850230C09C81BDB67310398FA68; Path=/demo/; HttpOnly

<

After I get the response back, when i do the status: curl -u ‘admin:district’ -X GET https://play.dhis2.org/demo/api/26/system/tasks/METADATA_IMPORT

I get this back:

[{“uid”:“lqOCBbHOTew”,“level”:“INFO”,“category”:“METADATA_IMPORT”,“time”:“2017-08-22T10:46:17.866”,“message”:“(admin) Import:Done took 43.66 seconds”,“completed”:true},{“uid”:“y2dBsPZrwMX”,“level”:“INFO”,“category”:“METADATA_IMPORT”,“time”:“2017-08-22T10:45:34.428”,“message”:“(admin) Updating 1332 object(s) of type OrganisationUnit”,“completed”:false},{“uid”:“GuGaKcJAToU”,“level”:“INFO”,“category”:“METADATA_IMPORT”,“time”:“2017-08-22T10:45:34.209”,“message”:“(admin) Import:Start”,“completed”:false}]

It takes about 43 seconds to import the file - POST does not come back until import is actually done. Documentation has this:

async
false, true
Asynchronous import, returns immediately with a Location header pointing to the location of the importReport.

And there is no location returned with the 204 (which is not the right HTTP code anyway).

I am also concerned about the time - when doing this with some PEPFAR countries, hierarchy with 14 thousand OUs took 8 hours to import.

Thanks,

Vlad

On Sun, Aug 6, 2017 at 1:44 PM Morten Olav Hansen morten@dhis2.org wrote:

Hi

It should work like that. Would you mind writing a JIRA issue? jira.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

Morten Olav Hansen

Senior Engineer, DHIS 2

University of Oslo

http://www.dhis2.org

On Thu, Aug 3, 2017 at 12:03 PM, Vladimer Shioshvili vshioshvili@gmail.com wrote:

Hi everyone,

I am trying to import xml metadata (org units) into 2.27, and async=true query parameter does not seem to make a difference - and it still goes in synchronously. It is in the documentation for 2.27 (and 2.26 and 2.25 - https://docs.dhis2.org/2.26/en/developer/html/dhis2_developer_manual_full.html#webapi_metadata_import), and I am using api version 26. Is it something i am doing wrong?

This is the curl command i am using:

curl -X POST ‘https://domain/api/26/metadata?atomicMode=NONE&async=true’ --data-binary @file.xml -H ‘Content-Type:application/xml’

Thanks!

Vlad


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp