Events API - content length mismatch

Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.

This is the request (should return about 700 events):

api/events.json?orgUnit=M4nRM5O3mH8&ouMode=DESCENDANTS&programStage=wmSb9TmevMn&startDate=2013-02-17&endDate=2016-02-17&status=COMPLETED&skipPaging=true

These are the error messages:

Chrome: “ERR_CONTENT_LENGTH_MISMATCH”

Curl: curl: (18) transfer closed with 1557104 bytes remaining to read

node.js request.get: simply loads the first part of the events object as a string

Any advice?

Olav

hey

I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).

···

On Thursday, 18 February 2016, Olav Poppe olav.poppe@me.com wrote:

Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.

This is the request (should return about 700 events):

api/events.json?orgUnit=M4nRM5O3mH8&ouMode=DESCENDANTS&programStage=wmSb9TmevMn&startDate=2013-02-17&endDate=2016-02-17&status=COMPLETED&skipPaging=true

These are the error messages:

Chrome: “ERR_CONTENT_LENGTH_MISMATCH”

Curl: curl: (18) transfer closed with 1557104 bytes remaining to read

node.js request.get: simply loads the first part of the events object as a string

Any advice?

Olav


Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org

Hm, never seen that either… do you have any idea of how much data it could be? do you have a very large number of data values for each event? if you are running this from a node js script, why not just use paging, and do multiple requests… should be easy enough to do

···

On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:

hey

I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).

On Thursday, 18 February 2016, Olav Poppe olav.poppe@me.com wrote:

Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.

This is the request (should return about 700 events):

api/events.json?orgUnit=M4nRM5O3mH8&ouMode=DESCENDANTS&programStage=wmSb9TmevMn&startDate=2013-02-17&endDate=2016-02-17&status=COMPLETED&skipPaging=true

These are the error messages:

Chrome: “ERR_CONTENT_LENGTH_MISMATCH”

Curl: curl: (18) transfer closed with 1557104 bytes remaining to read

node.js request.get: simply loads the first part of the events object as a string

Any advice?

Olav

Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Morten

Thanks both. I guess I will use paging, but I assume it should not be like this…? The total size would be about 2 Mb (last attempt I got about 475 of the events in 1,3 Mb).

Olav

···

On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:

hey

I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).

On Thursday, 18 February 2016, Olav Poppe olav.poppe@me.com wrote:

Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.

This is the request (should return about 700 events):

api/events.json?orgUnit=M4nRM5O3mH8&ouMode=DESCENDANTS&programStage=wmSb9TmevMn&startDate=2013-02-17&endDate=2016-02-17&status=COMPLETED&skipPaging=true

These are the error messages:

Chrome: “ERR_CONTENT_LENGTH_MISMATCH”

Curl: curl: (18) transfer closed with 1557104 bytes remaining to read

node.js request.get: simply loads the first part of the events object as a string

Any advice?

Olav

Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Morten

Yeah, it sounds weird… 2mb is not a big payload, is there anything in the server logs?

···

On Thu, Feb 18, 2016 at 3:15 PM, Olav Poppe olav.poppe@me.com wrote:

Thanks both. I guess I will use paging, but I assume it should not be like this…? The total size would be about 2 Mb (last attempt I got about 475 of the events in 1,3 Mb).
Olav

  1. feb. 2016 kl. 09.02 skrev Morten Olav Hansen morten@dhis2.org:

Hm, never seen that either… do you have any idea of how much data it could be? do you have a very large number of data values for each event? if you are running this from a node js script, why not just use paging, and do multiple requests… should be easy enough to do

On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:

hey

I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).

On Thursday, 18 February 2016, Olav Poppe olav.poppe@me.com wrote:

Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.

This is the request (should return about 700 events):

api/events.json?orgUnit=M4nRM5O3mH8&ouMode=DESCENDANTS&programStage=wmSb9TmevMn&startDate=2013-02-17&endDate=2016-02-17&status=COMPLETED&skipPaging=true

These are the error messages:

Chrome: “ERR_CONTENT_LENGTH_MISMATCH”

Curl: curl: (18) transfer closed with 1557104 bytes remaining to read

node.js request.get: simply loads the first part of the events object as a string

Any advice?

Olav

Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Morten

Morten

Hi, nothing in the DHIS logs, but in the nginx log: turned out I had wrong permissions on “…/nginx/proxy_temp/” for some reason. Never noticed any issues caused by that before, but fixing permissions solved this issue.

Thanks!

Olav

···

On Thu, Feb 18, 2016 at 3:15 PM, Olav Poppe olav.poppe@me.com wrote:

Thanks both. I guess I will use paging, but I assume it should not be like this…? The total size would be about 2 Mb (last attempt I got about 475 of the events in 1,3 Mb).
Olav

  1. feb. 2016 kl. 09.02 skrev Morten Olav Hansen morten@dhis2.org:

Hm, never seen that either… do you have any idea of how much data it could be? do you have a very large number of data values for each event? if you are running this from a node js script, why not just use paging, and do multiple requests… should be easy enough to do

On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:

hey

I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).

On Thursday, 18 February 2016, Olav Poppe olav.poppe@me.com wrote:

Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.

This is the request (should return about 700 events):

api/events.json?orgUnit=M4nRM5O3mH8&ouMode=DESCENDANTS&programStage=wmSb9TmevMn&startDate=2013-02-17&endDate=2016-02-17&status=COMPLETED&skipPaging=true

These are the error messages:

Chrome: “ERR_CONTENT_LENGTH_MISMATCH”

Curl: curl: (18) transfer closed with 1557104 bytes remaining to read

node.js request.get: simply loads the first part of the events object as a string

Any advice?

Olav

Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Morten

Morten

Makes sense :slight_smile:
if the response is small enough it fits into memory and doesn’t need to write to the tmp file in that folder

When buffering is enabled, nginx receives a response from the proxied server as soon as possible, saving it into the buffers set by the proxy_buffer_size and proxy_buffers directives. If the whole response does not fit into memory, a part of it can be saved to a temporary file on the disk. Writing to temporary files is controlled by the proxy_max_temp_file_size and proxy_temp_file_write_size directives.

···

On Thu, Feb 18, 2016 at 12:12 PM, Olav Poppe olav.poppe@me.com wrote:

Hi, nothing in the DHIS logs, but in the nginx log: turned out I had wrong permissions on “…/nginx/proxy_temp/” for some reason. Never noticed any issues caused by that before, but fixing permissions solved this issue.

Thanks!

Olav

  1. feb. 2016 kl. 09.40 skrev Morten Olav Hansen morten@dhis2.org:

Yeah, it sounds weird… 2mb is not a big payload, is there anything in the server logs?

On Thu, Feb 18, 2016 at 3:15 PM, Olav Poppe olav.poppe@me.com wrote:

Thanks both. I guess I will use paging, but I assume it should not be like this…? The total size would be about 2 Mb (last attempt I got about 475 of the events in 1,3 Mb).
Olav

  1. feb. 2016 kl. 09.02 skrev Morten Olav Hansen morten@dhis2.org:

Hm, never seen that either… do you have any idea of how much data it could be? do you have a very large number of data values for each event? if you are running this from a node js script, why not just use paging, and do multiple requests… should be easy enough to do


Morten

On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:

hey

I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).

On Thursday, 18 February 2016, Olav Poppe olav.poppe@me.com wrote:

Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.

This is the request (should return about 700 events):

api/events.json?orgUnit=M4nRM5O3mH8&ouMode=DESCENDANTS&programStage=wmSb9TmevMn&startDate=2013-02-17&endDate=2016-02-17&status=COMPLETED&skipPaging=true

These are the error messages:

Chrome: “ERR_CONTENT_LENGTH_MISMATCH”

Curl: curl: (18) transfer closed with 1557104 bytes remaining to read

node.js request.get: simply loads the first part of the events object as a string

Any advice?

Olav

Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Morten

Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org

Thanks, yes, then it does make sense!

Olav

···

On Thu, Feb 18, 2016 at 12:12 PM, Olav Poppe olav.poppe@me.com wrote:

Hi, nothing in the DHIS logs, but in the nginx log: turned out I had wrong permissions on “…/nginx/proxy_temp/” for some reason. Never noticed any issues caused by that before, but fixing permissions solved this issue.

Thanks!

Olav

  1. feb. 2016 kl. 09.40 skrev Morten Olav Hansen morten@dhis2.org:

Yeah, it sounds weird… 2mb is not a big payload, is there anything in the server logs?

On Thu, Feb 18, 2016 at 3:15 PM, Olav Poppe olav.poppe@me.com wrote:

Thanks both. I guess I will use paging, but I assume it should not be like this…? The total size would be about 2 Mb (last attempt I got about 475 of the events in 1,3 Mb).
Olav

  1. feb. 2016 kl. 09.02 skrev Morten Olav Hansen morten@dhis2.org:

Hm, never seen that either… do you have any idea of how much data it could be? do you have a very large number of data values for each event? if you are running this from a node js script, why not just use paging, and do multiple requests… should be easy enough to do


Morten

On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:

hey

I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).

On Thursday, 18 February 2016, Olav Poppe olav.poppe@me.com wrote:

Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.

This is the request (should return about 700 events):

api/events.json?orgUnit=M4nRM5O3mH8&ouMode=DESCENDANTS&programStage=wmSb9TmevMn&startDate=2013-02-17&endDate=2016-02-17&status=COMPLETED&skipPaging=true

These are the error messages:

Chrome: “ERR_CONTENT_LENGTH_MISMATCH”

Curl: curl: (18) transfer closed with 1557104 bytes remaining to read

node.js request.get: simply loads the first part of the events object as a string

Any advice?

Olav

Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Morten

Pierre Dane

Jembi Health Systems

Software Development Manager

tel: +27 (0)21 701 0939

cel: +27 (0)83 680 8274

email: pierre@jembi.org

web: www.jembi.org

Very odd misconfiguration error for nginx though. Did you install
through some package manager (apt, yum, ...) or was this manually
unpacked and configured? If there is an error in the standard ubuntu
install for example, its an important issue to be aware of.

···

On 18 February 2016 at 11:12, Olav Poppe <olav.poppe@me.com> wrote:

Hi, nothing in the DHIS logs, but in the nginx log: turned out I had wrong
permissions on "…/nginx/proxy_temp/" for some reason. Never noticed any
issues caused by that before, but fixing permissions solved this issue.

Thanks!

Olav

18. feb. 2016 kl. 09.40 skrev Morten Olav Hansen <morten@dhis2.org>:

Yeah, it sounds weird.. 2mb is not a big payload, is there anything in the
server logs?

On Thu, Feb 18, 2016 at 3:15 PM, Olav Poppe <olav.poppe@me.com> wrote:

Thanks both. I guess I will use paging, but I assume it should not be like
this..? The total size would be about 2 Mb (last attempt I got about 475 of
the events in 1,3 Mb).

Olav

18. feb. 2016 kl. 09.02 skrev Morten Olav Hansen <morten@dhis2.org>:

Hm, never seen that either.. do you have any idea of how much data it
could be? do you have a very large number of data values for each event? if
you are running this from a node js script, why not just use paging, and do
multiple requests.. should be easy enough to do

On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane <pierre@jembi.org> wrote:

hey

I haven't run into this but you may have to page:( or you could try a
sqlview which won't have a size restriction (you would have to use a
function as tracker events can't be accessed with raw sql).

On Thursday, 18 February 2016, Olav Poppe <olav.poppe@me.com> wrote:

Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a
small number of events, everything works fine. However, for larger datasets,
it fails to load the full "events" object.

This is the request (should return about 700 events):

api/events.json?orgUnit=M4nRM5O3mH8&ouMode=DESCENDANTS&programStage=wmSb9TmevMn&startDate=2013-02-17&endDate=2016-02-17&status=COMPLETED&skipPaging=true

These are the error messages:
Chrome: "ERR_CONTENT_LENGTH_MISMATCH"
Curl: curl: (18) transfer closed with 1557104 bytes remaining to read
node.js request.get: simply loads the first part of the events object as
a string

Any advice?

Olav

--
Pierre Dane

Jembi Health Systems
Software Development Manager
tel: +27 (0)21 701 0939
cel: +27 (0)83 680 8274
email: pierre@jembi.org
web: www.jembi.org

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

--
Morten

--
Morten

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp

I was running this locally, so on a mac with nginx installed via a mac package manager (homebrew). So hopefully not an issue on any production servers.

···
  1. feb. 2016 kl. 14.17 skrev Bob Jolliffe bobjolliffe@gmail.com:

Very odd misconfiguration error for nginx though. Did you install
through some package manager (apt, yum, …) or was this manually
unpacked and configured? If there is an error in the standard ubuntu
install for example, its an important issue to be aware of.

On 18 February 2016 at 11:12, Olav Poppe olav.poppe@me.com wrote:

Hi, nothing in the DHIS logs, but in the nginx log: turned out I had wrong
permissions on “…/nginx/proxy_temp/” for some reason. Never noticed any
issues caused by that before, but fixing permissions solved this issue.

Thanks!

Olav

  1. feb. 2016 kl. 09.40 skrev Morten Olav Hansen morten@dhis2.org:

Yeah, it sounds weird… 2mb is not a big payload, is there anything in the
server logs?

On Thu, Feb 18, 2016 at 3:15 PM, Olav Poppe olav.poppe@me.com wrote:

Thanks both. I guess I will use paging, but I assume it should not be like
this…? The total size would be about 2 Mb (last attempt I got about 475 of
the events in 1,3 Mb).

Olav

  1. feb. 2016 kl. 09.02 skrev Morten Olav Hansen morten@dhis2.org:

Hm, never seen that either… do you have any idea of how much data it
could be? do you have a very large number of data values for each event? if
you are running this from a node js script, why not just use paging, and do
multiple requests… should be easy enough to do

On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:

hey

I haven’t run into this but you may have to page:( or you could try a
sqlview which won’t have a size restriction (you would have to use a
function as tracker events can’t be accessed with raw sql).

On Thursday, 18 February 2016, Olav Poppe olav.poppe@me.com wrote:

Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a
small number of events, everything works fine. However, for larger datasets,
it fails to load the full “events” object.

This is the request (should return about 700 events):

api/events.json?orgUnit=M4nRM5O3mH8&ouMode=DESCENDANTS&programStage=wmSb9TmevMn&startDate=2013-02-17&endDate=2016-02-17&status=COMPLETED&skipPaging=true

These are the error messages:
Chrome: “ERR_CONTENT_LENGTH_MISMATCH”
Curl: curl: (18) transfer closed with 1557104 bytes remaining to read
node.js request.get: simply loads the first part of the events object as
a string

Any advice?

Olav


Pierre Dane

Jembi Health Systems
Software Development Manager
tel: +27 (0)21 701 0939
cel: +27 (0)83 680 8274
email: pierre@jembi.org
web: www.jembi.org


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Morten


Morten


Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp