409 Errors in Tracker Capture During Program Enrollment

Hi all,
Please read the thread below regarding Tracker Capture App and server errors. We have upgraded the server to a T5 package and are still experiencing issues. In particular, users cannot download reserved UIDs during peak hours with heavy server traffic. Here is a link to the same problem logged into JIRA. Any advice or feedback on similar experiences would be greatly appreciated.

Many thanks,

Aimee Edmondo, MPH | Informatics Associate
aedmondo@akros.com | +260.976.076524

image

image

···

---------- Forwarded message ----------
From: Aimee Edmondo aedmondo@akros.com
Date: Wed, Nov 22, 2017 at 10:55 AM
Subject: 409 Errors in Tracker Capture During Program Enrollment
To: dhis2-devs@lists.launchpad.net

Greetings developers,

In Zambia we are implementing tracker capture and have repeatedly come across the following issue, which is resulting in a non-functional system. Any advice or assistance would be greatly appreciated. This problem has also been logged into JIRA.

App: Android Tracker Capture v 0.34 and 0.40

Tablet: Alcatel Pixi4

DHIS2 Tracker: v2.25

Server: BAO ST4

When enrolling a tracked entity in a new program, the system gives a 409 error. These errors create a backlog on the server and other functions cannot be performed (ex: cannot sync data or generate reserved UIDs). Server usage is at 100% during peak hours (9:00 to 17:00 Central African Time) despite limited functionality. All user roles have been changed to allow them to have access to all permissions. All programs have been changed to repeatable. The 409 errors and issues persist even after these changes. Server access log below:

::ffff:197.220.196.195 - LSibongile [22/Nov/2017:08:38:20 +0000] “POST /api/events/ HTTP/1.1” 409 480 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - MMusosa [22/Nov/2017:08:38:20 +0000] “POST /api/enrollments/ HTTP/1.1” 409 494 “-” “okhttp/2.7.2” “-”

::ffff:41.223.117.53 - - [22/Nov/2017:08:38:19 +0000] “GET /dhis-web-maintenance-user/index.action HTTP/1.1” 200 2725 "https:/…

::ffff:41.223.118.69 - ESINKALA [22/Nov/2017:08:38:19 +0000] "GET /api/enrollments?skipPaging=true&ouMode=ACCESSIBLE&trackedEn…

::ffff:197.220.196.195 - NBanda [22/Nov/2017:08:38:19 +0000] “POST /api/enrollments/ HTTP/1.1” 500 226 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - LSibongile [22/Nov/2017:08:38:19 +0000] “POST /api/events/ HTTP/1.1” 409 480 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - MMusosa [22/Nov/2017:08:38:18 +0000] “POST /api/enrollments/ HTTP/1.1” 409 494 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - MMusosa [22/Nov/2017:08:38:17 +0000] “POST /api/enrollments/ HTTP/1.1” 409 494 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - LSibongile [22/Nov/2017:08:38:17 +0000] “POST /api/events/ HTTP/1.1” 409 480 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - NBanda [22/Nov/2017:08:38:17 +0000] “POST /api/enrollments/ HTTP/1.1” 500 226 “-” “okhttp/2.7.2” “-”

::ffff:41.223.117.53 - PMwambazi [22/Nov/2017:08:38:17 +0000] "GET /api/events/woasxx8PBT8?fields=created%2ClastUpdated HTTP/1…

::ffff:197.220.196.195 - MMusosa [22/Nov/2017:08:38:16 +0000] “POST /api/enrollments/ HTTP/1.1” 409 494 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - LSibongile [22/Nov/2017:08:38:16 +0000] “POST /api/events/ HTTP/1.1” 409 480 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - NBanda [22/Nov/2017:08:38:15 +0000] “POST /api/enrollments/ HTTP/1.1” 500 226 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - LSibongile [22/Nov/2017:08:38:15 +0000] “POST /api/events/ HTTP/1.1” 409 480 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - MMusosa [22/Nov/2017:08:38:15 +0000] “POST /api/enrollments/ HTTP/1.1” 409 494 “-” “okhttp/2.7.2” “-”

::ffff:41.223.117.53 - PMwambazi [22/Nov/2017:08:38:14 +0000] “POST /api/events/ HTTP/1.1” 200 247 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - MMusosa [22/Nov/2017:08:38:13 +0000] “POST /api/enrollments/ HTTP/1.1” 409 494 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - LSibongile [22/Nov/2017:08:38:13 +0000] “POST /api/events/ HTTP/1.1” 409 480 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - NBanda [22/Nov/2017:08:38:13 +0000] “POST /api/enrollments/ HTTP/1.1” 500 226 “-” “okhttp/2.7.2” “-”

See additional attachments to this email, including CPU usage and error logs.Inline image 1

Inline image 2

Aimee Edmondo, MPH | Informatics Associate
aedmondo@akros.com | +260.976.076524

Hi Aimee,

We experienced this in Uganda last year. The solution was to do load balancing of tomcat and web servers using apache 2.4 and above (load balancing)

We also implemented postgresql replication to take advantage of read only queries for analytics.

You may need extra servers or lxc containers to achieve it.

Alex

image

image

···

On Tuesday, November 28, 2017, Aimee Edmondo aedmondo@akros.com wrote:

Hi all,
Please read the thread below regarding Tracker Capture App and server errors. We have upgraded the server to a T5 package and are still experiencing issues. In particular, users cannot download reserved UIDs during peak hours with heavy server traffic. Here is a link to the same problem logged into JIRA. Any advice or feedback on similar experiences would be greatly appreciated.

Many thanks,

Aimee Edmondo, MPH | Informatics Associate
aedmondo@akros.com | +260.976.076524

---------- Forwarded message ----------
From: Aimee Edmondo aedmondo@akros.com
Date: Wed, Nov 22, 2017 at 10:55 AM
Subject: 409 Errors in Tracker Capture During Program Enrollment
To: dhis2-devs@lists.launchpad.net

Greetings developers,

In Zambia we are implementing tracker capture and have repeatedly come across the following issue, which is resulting in a non-functional system. Any advice or assistance would be greatly appreciated. This problem has also been logged into JIRA.

App: Android Tracker Capture v 0.34 and 0.40

Tablet: Alcatel Pixi4

DHIS2 Tracker: v2.25

Server: BAO ST4

When enrolling a tracked entity in a new program, the system gives a 409 error. These errors create a backlog on the server and other functions cannot be performed (ex: cannot sync data or generate reserved UIDs). Server usage is at 100% during peak hours (9:00 to 17:00 Central African Time) despite limited functionality. All user roles have been changed to allow them to have access to all permissions. All programs have been changed to repeatable. The 409 errors and issues persist even after these changes. Server access log below:

::ffff:197.220.196.195 - LSibongile [22/Nov/2017:08:38:20 +0000] “POST /api/events/ HTTP/1.1” 409 480 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - MMusosa [22/Nov/2017:08:38:20 +0000] “POST /api/enrollments/ HTTP/1.1” 409 494 “-” “okhttp/2.7.2” “-”

::ffff:41.223.117.53 - - [22/Nov/2017:08:38:19 +0000] “GET /dhis-web-maintenance-user/index.action HTTP/1.1” 200 2725 "https:/…

::ffff:41.223.118.69 - ESINKALA [22/Nov/2017:08:38:19 +0000] "GET /api/enrollments?skipPaging=true&ouMode=ACCESSIBLE&trackedEn…

::ffff:197.220.196.195 - NBanda [22/Nov/2017:08:38:19 +0000] “POST /api/enrollments/ HTTP/1.1” 500 226 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - LSibongile [22/Nov/2017:08:38:19 +0000] “POST /api/events/ HTTP/1.1” 409 480 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - MMusosa [22/Nov/2017:08:38:18 +0000] “POST /api/enrollments/ HTTP/1.1” 409 494 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - MMusosa [22/Nov/2017:08:38:17 +0000] “POST /api/enrollments/ HTTP/1.1” 409 494 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - LSibongile [22/Nov/2017:08:38:17 +0000] “POST /api/events/ HTTP/1.1” 409 480 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - NBanda [22/Nov/2017:08:38:17 +0000] “POST /api/enrollments/ HTTP/1.1” 500 226 “-” “okhttp/2.7.2” “-”

::ffff:41.223.117.53 - PMwambazi [22/Nov/2017:08:38:17 +0000] "GET /api/events/woasxx8PBT8?fields=created%2ClastUpdated HTTP/1…

::ffff:197.220.196.195 - MMusosa [22/Nov/2017:08:38:16 +0000] “POST /api/enrollments/ HTTP/1.1” 409 494 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - LSibongile [22/Nov/2017:08:38:16 +0000] “POST /api/events/ HTTP/1.1” 409 480 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - NBanda [22/Nov/2017:08:38:15 +0000] “POST /api/enrollments/ HTTP/1.1” 500 226 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - LSibongile [22/Nov/2017:08:38:15 +0000] “POST /api/events/ HTTP/1.1” 409 480 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - MMusosa [22/Nov/2017:08:38:15 +0000] “POST /api/enrollments/ HTTP/1.1” 409 494 “-” “okhttp/2.7.2” “-”

::ffff:41.223.117.53 - PMwambazi [22/Nov/2017:08:38:14 +0000] “POST /api/events/ HTTP/1.1” 200 247 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - MMusosa [22/Nov/2017:08:38:13 +0000] “POST /api/enrollments/ HTTP/1.1” 409 494 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - LSibongile [22/Nov/2017:08:38:13 +0000] “POST /api/events/ HTTP/1.1” 409 480 “-” “okhttp/2.7.2” “-”

::ffff:197.220.196.195 - NBanda [22/Nov/2017:08:38:13 +0000] “POST /api/enrollments/ HTTP/1.1” 500 226 “-” “okhttp/2.7.2” “-”

See additional attachments to this email, including CPU usage and error logs.Inline image 1

Inline image 2

Aimee Edmondo, MPH | Informatics Associate
aedmondo@akros.com | +260.976.076524


Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET | HISP Uganda

Kampala

Uganda
+256 774149 775, + 256 759 800161

Skype ID: talexie

IT Consultant (Servers, Networks and Security, Health Information Systems - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant

"I don’t want to be anything other than what I have been - one tree hill "