Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.
This is the request (should return about 700 events):
I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).
Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.
This is the request (should return about 700 events):
Hm, never seen that either… do you have any idea of how much data it could be? do you have a very large number of data values for each event? if you are running this from a node js script, why not just use paging, and do multiple requests… should be easy enough to do
···
On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:
hey
I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).
Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.
This is the request (should return about 700 events):
Thanks both. I guess I will use paging, but I assume it should not be like this…? The total size would be about 2 Mb (last attempt I got about 475 of the events in 1,3 Mb).
Olav
···
On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:
hey
I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).
Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.
This is the request (should return about 700 events):
Yeah, it sounds weird… 2mb is not a big payload, is there anything in the server logs?
···
On Thu, Feb 18, 2016 at 3:15 PM, Olav Poppe olav.poppe@me.com wrote:
Thanks both. I guess I will use paging, but I assume it should not be like this…? The total size would be about 2 Mb (last attempt I got about 475 of the events in 1,3 Mb).
Olav
Hm, never seen that either… do you have any idea of how much data it could be? do you have a very large number of data values for each event? if you are running this from a node js script, why not just use paging, and do multiple requests… should be easy enough to do
–
On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:
hey
I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).
Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.
This is the request (should return about 700 events):
Hi, nothing in the DHIS logs, but in the nginx log: turned out I had wrong permissions on “…/nginx/proxy_temp/” for some reason. Never noticed any issues caused by that before, but fixing permissions solved this issue.
Thanks!
Olav
···
On Thu, Feb 18, 2016 at 3:15 PM, Olav Poppe olav.poppe@me.com wrote:
Thanks both. I guess I will use paging, but I assume it should not be like this…? The total size would be about 2 Mb (last attempt I got about 475 of the events in 1,3 Mb).
Olav
Hm, never seen that either… do you have any idea of how much data it could be? do you have a very large number of data values for each event? if you are running this from a node js script, why not just use paging, and do multiple requests… should be easy enough to do
–
On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:
hey
I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).
Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.
This is the request (should return about 700 events):
Makes sense
if the response is small enough it fits into memory and doesn’t need to write to the tmp file in that folder
When buffering is enabled, nginx receives a response from the proxied server as soon as possible, saving it into the buffers set by the proxy_buffer_size and proxy_buffers directives. If the whole response does not fit into memory, a part of it can be saved to a temporary file on the disk. Writing to temporary files is controlled by the proxy_max_temp_file_size and proxy_temp_file_write_size directives.
···
On Thu, Feb 18, 2016 at 12:12 PM, Olav Poppe olav.poppe@me.com wrote:
Hi, nothing in the DHIS logs, but in the nginx log: turned out I had wrong permissions on “…/nginx/proxy_temp/” for some reason. Never noticed any issues caused by that before, but fixing permissions solved this issue.
Yeah, it sounds weird… 2mb is not a big payload, is there anything in the server logs?
–
On Thu, Feb 18, 2016 at 3:15 PM, Olav Poppe olav.poppe@me.com wrote:
Thanks both. I guess I will use paging, but I assume it should not be like this…? The total size would be about 2 Mb (last attempt I got about 475 of the events in 1,3 Mb).
Olav
Hm, never seen that either… do you have any idea of how much data it could be? do you have a very large number of data values for each event? if you are running this from a node js script, why not just use paging, and do multiple requests… should be easy enough to do
–
Morten
On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:
hey
I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).
Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.
This is the request (should return about 700 events):
On Thu, Feb 18, 2016 at 12:12 PM, Olav Poppe olav.poppe@me.com wrote:
Hi, nothing in the DHIS logs, but in the nginx log: turned out I had wrong permissions on “…/nginx/proxy_temp/” for some reason. Never noticed any issues caused by that before, but fixing permissions solved this issue.
Yeah, it sounds weird… 2mb is not a big payload, is there anything in the server logs?
–
On Thu, Feb 18, 2016 at 3:15 PM, Olav Poppe olav.poppe@me.com wrote:
Thanks both. I guess I will use paging, but I assume it should not be like this…? The total size would be about 2 Mb (last attempt I got about 475 of the events in 1,3 Mb).
Olav
Hm, never seen that either… do you have any idea of how much data it could be? do you have a very large number of data values for each event? if you are running this from a node js script, why not just use paging, and do multiple requests… should be easy enough to do
–
Morten
On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:
hey
I haven’t run into this but you may have to page:( or you could try a sqlview which won’t have a size restriction (you would have to use a function as tracker events can’t be accessed with raw sql).
Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a small number of events, everything works fine. However, for larger datasets, it fails to load the full “events” object.
This is the request (should return about 700 events):
Very odd misconfiguration error for nginx though. Did you install
through some package manager (apt, yum, ...) or was this manually
unpacked and configured? If there is an error in the standard ubuntu
install for example, its an important issue to be aware of.
···
On 18 February 2016 at 11:12, Olav Poppe <olav.poppe@me.com> wrote:
Hi, nothing in the DHIS logs, but in the nginx log: turned out I had wrong
permissions on "…/nginx/proxy_temp/" for some reason. Never noticed any
issues caused by that before, but fixing permissions solved this issue.
Thanks!
Olav
18. feb. 2016 kl. 09.40 skrev Morten Olav Hansen <morten@dhis2.org>:
Yeah, it sounds weird.. 2mb is not a big payload, is there anything in the
server logs?
On Thu, Feb 18, 2016 at 3:15 PM, Olav Poppe <olav.poppe@me.com> wrote:
Thanks both. I guess I will use paging, but I assume it should not be like
this..? The total size would be about 2 Mb (last attempt I got about 475 of
the events in 1,3 Mb).
Olav
18. feb. 2016 kl. 09.02 skrev Morten Olav Hansen <morten@dhis2.org>:
Hm, never seen that either.. do you have any idea of how much data it
could be? do you have a very large number of data values for each event? if
you are running this from a node js script, why not just use paging, and do
multiple requests.. should be easy enough to do
On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane <pierre@jembi.org> wrote:
hey
I haven't run into this but you may have to page:( or you could try a
sqlview which won't have a size restriction (you would have to use a
function as tracker events can't be accessed with raw sql).
On Thursday, 18 February 2016, Olav Poppe <olav.poppe@me.com> wrote:
Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a
small number of events, everything works fine. However, for larger datasets,
it fails to load the full "events" object.
This is the request (should return about 700 events):
These are the error messages:
Chrome: "ERR_CONTENT_LENGTH_MISMATCH"
Curl: curl: (18) transfer closed with 1557104 bytes remaining to read
node.js request.get: simply loads the first part of the events object as
a string
Any advice?
Olav
--
Pierre Dane
Jembi Health Systems
Software Development Manager
tel: +27 (0)21 701 0939
cel: +27 (0)83 680 8274
email: pierre@jembi.org
web: www.jembi.org
I was running this locally, so on a mac with nginx installed via a mac package manager (homebrew). So hopefully not an issue on any production servers.
Very odd misconfiguration error for nginx though. Did you install
through some package manager (apt, yum, …) or was this manually
unpacked and configured? If there is an error in the standard ubuntu
install for example, its an important issue to be aware of.
Hi, nothing in the DHIS logs, but in the nginx log: turned out I had wrong
permissions on “…/nginx/proxy_temp/” for some reason. Never noticed any
issues caused by that before, but fixing permissions solved this issue.
Yeah, it sounds weird… 2mb is not a big payload, is there anything in the
server logs?
On Thu, Feb 18, 2016 at 3:15 PM, Olav Poppe olav.poppe@me.com wrote:
Thanks both. I guess I will use paging, but I assume it should not be like
this…? The total size would be about 2 Mb (last attempt I got about 475 of
the events in 1,3 Mb).
Hm, never seen that either… do you have any idea of how much data it
could be? do you have a very large number of data values for each event? if
you are running this from a node js script, why not just use paging, and do
multiple requests… should be easy enough to do
On Thu, Feb 18, 2016 at 2:57 PM, Pierre Dane pierre@jembi.org wrote:
hey
I haven’t run into this but you may have to page:( or you could try a
sqlview which won’t have a size restriction (you would have to use a
function as tracker events can’t be accessed with raw sql).
Hi devs,
I’m trying to fetch events from the API (DHIS 2.22 build 21829). For a
small number of events, everything works fine. However, for larger datasets,
it fails to load the full “events” object.
This is the request (should return about 700 events):
These are the error messages:
Chrome: “ERR_CONTENT_LENGTH_MISMATCH”
Curl: curl: (18) transfer closed with 1557104 bytes remaining to read
node.js request.get: simply loads the first part of the events object as
a string
Any advice?
Olav
–
Pierre Dane
Jembi Health Systems
Software Development Manager
tel: +27 (0)21 701 0939
cel: +27 (0)83 680 8274
email: pierre@jembi.org
web: www.jembi.org
