Pull data out of mobile device from DHIS2 Android

Implementation - Implémentation Mobile
1

Hi All!

We would like to have the ability to pull data out of a device in order to format & import the data in another method, if/when syncing is not possible.

For a reason we are still investigating, it appears that users have entered data into their device but are unable to sync it up to the server. (The same user newly logged-in device can sync data just fine, and unfortunately sync configuration does not resolve the 500 error that we are getting when trying to sync in the original device.)

Is there a location that we can find and make sense of data (ideally specifically the data that has been saved and not synced) in the local memory of the device, so that we can pull it out and push it into our servers manually?

Thanks!

2

Hi @Matthew_Boddie,

I am sorry to hear about your issues at TEI sync, if we can help you investigating the issue please give us more information about it. Have you tried syncing TEIs one by one (only the modified ones)? In this way, you can narrow down the failing one.

App data is stored in the internal database (SQLite) in the app folder, which is protected and cannot be exported by default. We have thought about giving a way to export this database for debugging purposes, but nothing has been implemented right now because of security concerns.

1 Like
3

@vgarciabnz appreciate the response! good to know re the inability to export; this makes sense.

Actually our experience is more and more interesting as we dive into it. Here’s what we know so far:
Specific Users, who were able to sync data from the same programs with the same kinds of selections last month, are unable to sync entered data now. These specific users have synced configuration, to no avail. These usernames were also used and logged into across multiple distinct devices, and experience the same lack of ability to sync information. This is across 2 different event programs, 1 which has 2 categories into a categoryCombo, the other which uses default catOptionCombo. The users which are unable to sync on mobile still have the ability to fully enter and save data through the Capture App, on Browser

4

Some other things we’ve noticed:

Not all of the users have the same pieces of metadata in their API. Example 1 has a CreatedBy section (and strangely it notes that the user was createdBy itself…), and a user section. I’ve tried through import/export to modify and update these values in both Example 1 and 2, however no edits seem to be made after passing them in.

Example 1

{“lastUpdated”:“2021-08-10T16:07:35.975”,“id”:“rIvR57tQtxQ”,“href”:“https://vectorlinkcollect.org/api/users/rIvR57tQtxQ",“created”:“2021-08-10T16:07:35.975”,“name”:"Wiawso Community GH Mobile”,“displayName”:“Wiawso Community GH Mobile”,“externalAccess”:false,“surname”:“GH Mobile”,“firstName”:“Wiawso Community”,“favorite”:false,“access”:{“read”:true,“update”:true,“externalize”:true,“delete”:true,“write”:true,“manage”:true},“userCredentials”:{“lastUpdated”:“2023-08-29T17:54:04.054”,“id”:“PxH5kEpNlVw”,“created”:“2021-08-10T16:07:35.576”,“name”:“Wiawso Community GH Mobile”,“lastLogin”:“2023-08-29T17:54:04.053”,“displayName”:“Wiawso Community GH Mobile”,“externalAuth”:false,“externalAccess”:false,“disabled”:false,“twoFA”:false,“passwordLastUpdated”:“2021-08-10T16:07:37.288”,“invitation”:false,“selfRegistered”:false,“favorite”:false,“username”:“GH-WIA-1”,“userInfo”:{“id”:“rIvR57tQtxQ”},“access”:{“read”:true,“update”:true,“externalize”:true,“delete”:true,“write”:true,“manage”:true},“lastUpdatedBy”:{“displayName”:“Wiawso Community GH Mobile”,“name”:“Wiawso Community GH Mobile”,“id”:“rIvR57tQtxQ”,“username”:“GH-WIA-1”},“sharing”:{“external”:false},“createdBy”:{“displayName”:“Wiawso Community GH Mobile”,“name”:“Wiawso Community GH Mobile”,“id”:“rIvR57tQtxQ”,“username”:“GH-WIA-1”},“user”:{“displayName”:“Wiawso Community GH Mobile”,“name”:“Wiawso Community GH Mobile”,“id”:“rIvR57tQtxQ”,“username”:“GH-WIA-1”},“favorites”:,“cogsDimensionConstraints”:,“catDimensionConstraints”:,“translations”:,“userGroupAccesses”:,“attributeValues”:,“userRoles”:[{“id”:“Rry05ioKFiN”},{“id”:“cZmXa0qWK8I”}],“userAccesses”:},“sharing”:{“external”:false},“favorites”:,“teiSearchOrganisationUnits”:,“translations”:,“organisationUnits”:[{“id”:“JjT8hqi7T2T”}],“dataViewOrganisationUnits”:[{“id”:“JjT8hqi7T2T”}],“userGroupAccesses”:,“attributeValues”:,“userGroups”:[{“id”:“O5KW6iofw0H”},{“id”:“UcbGln8fRgI”},{“id”:“S0mJfuacgJr”}],“userAccesses”:}

Example 2

{“lastUpdated”:“2023-08-29T18:43:36.056”,“id”:“kgfAuTIDW1G”,“href”:“https://vectorlinkcollect.org/api/users/kgfAuTIDW1G",“created”:“2021-08-10T16:07:35.977”,“name”:"Wiawso Community GH Mobile”,“displayName”:“Wiawso Community GH Mobile”,“externalAccess”:false,“surname”:“GH Mobile”,“firstName”:“Wiawso Community”,“favorite”:false,“access”:{“read”:true,“update”:true,“externalize”:true,“delete”:true,“write”:true,“manage”:true},“userCredentials”:{“lastUpdated”:“2023-08-29T18:26:16.701”,“id”:“J3I6vOl2dFB”,“created”:“2023-08-29T18:26:16.701”,“name”:“Wiawso Community GH Mobile”,“lastLogin”:“2023-08-29T18:06:42.182”,“displayName”:“Wiawso Community GH Mobile”,“externalAuth”:false,“externalAccess”:false,“disabled”:false,“twoFA”:false,“passwordLastUpdated”:“2023-08-29T18:26:16.701”,“invitation”:false,“selfRegistered”:false,“favorite”:false,“username”:“GH-WIA-2”,“userInfo”:{“id”:“kgfAuTIDW1G”},“access”:{“read”:true,“update”:true,“externalize”:true,“delete”:true,“write”:true,“manage”:true},“lastUpdatedBy”:{“displayName”:“Matthew Boddie”,“name”:“Matthew Boddie",“id”:“mqCwaTdzxy2”,“username”:"matthew_boddie@abtassoc.com”},“sharing”:{“external”:false},“favorites”:,“cogsDimensionConstraints”:,“catDimensionConstraints”:,“translations”:,“userGroupAccesses”:,“attributeValues”:,“userRoles”:[{“id”:“Rry05ioKFiN”},{“id”:“cZmXa0qWK8I”}],“userAccesses”:},“sharing”:{“external”:false},“favorites”:,“teiSearchOrganisationUnits”:,“translations”:,“organisationUnits”:[{“id”:“JjT8hqi7T2T”}],“dataViewOrganisationUnits”:[{“id”:“JjT8hqi7T2T”}],“userGroupAccesses”:,“attributeValues”:,“userGroups”:[{“id”:“O5KW6iofw0H”},{“id”:“UcbGln8fRgI”},{“id”:“S0mJfuacgJr”}],“userAccesses”:}

Error Message from Mobile when we try and sync:

{”httpStatus”:”Internal Server Error”,”httpStatusCode:”500”,”status”:”ERROR”}
5

Hello @Matthew_Boddie .

What you are asking about the extraction of data would go against the security principles while desgining the app. But Victor already explained that and it seems clear.

How did you manage to extract the API info? Are you using a proxy? Or you are using the Training App? Because if the second you could actually extract the inform from the DB.

In any case, what I would do is to perform a granular sync and try to analyse the 500 on the server logs which should probably guide you. With the information you are providing, I think there might be a wrong metadata on the device (as a new one can sync without problems).

6

@jaime.bosque appreciate the response. Yes with Flipper and a training app we were able to extract the data and import it into a server for testing.

One premise to make clear is that, actually, across devices, a user is unable to sync data. I.e. we heard reports of the error (in field office), and multiple devices replicated the issue (in home office). Strangely, some (not all) of the users that we do test will, the following day, become functional. For these newfound ability to sync users, their at least appears to be no change/it was confirmed that noone attempted to make a change to the user in anyway (other than simply logging in).

If by granular sync you mean simply attempting to sync a single event and looking at the app logs, it’s only the {”httpStatus”:”Internal Server Error”,”httpStatusCode:”500”,”status”:”ERROR”} that we receive. But I feel like I’m misunderstanding here. We’ve tried and as of yet unsuccessfully searched for a correlating error log in the dhis2.log from the backend, but will keep searching.

7

Hello @Matthew_Boddie

This seems awkward indeed and then makes troubleshooting even more difficult. Might it be realted to a network issue?

For that part, yes, I mean clicking on a single item and inspecting the logs at the server level. You should be able to identify the call and trace the error. You should be able to see the API call and the response on the server. If you could identify the API call I would recommend performing it isolated from the device (thus is putting it directly on the browser or via any API program you might use i.e. postman). If you get the 500 constantly it means there is an error and you could play modifying the call to understand where it might be.

Cheers.

8

@jaime.bosque pulling and pushing the sync from mobile into browser was functional, and indeed there are no Problems with the user on a laptop at all. Internet was also a first thought of ours, however we’ve now replicates this in one user in 3 countries and 2 US states across a myriad of high speed connections and devices.

We’re still on this obviously l, will keep all updated.

9

If by network you mean the server itself, that well could be, however we’d expect to be seeing errors more generally and not just related to android. We are syncing quite a large volume of single events daily, and would likely have seen other effects.

Update: We have gone through and confirmed that there is no account for these data sync failures in the logs at all.

10

Hi @Matthew_Boddie .

This seems challenging to troubleshoot for what you are saying. The 500 error is very generic but I think it should be output in the logs. I am thinking that it also could be related to the server load, maybe? Not sure, please let us know if you need help although not sure what else we could do.

1 Like
11

Hi @jaime.bosque very much agree! We’ve had some good progress on this, very much due to @chase.freeman helping us out.

[DHIS2-14021] - Jira is where Chase has outlined it and created a new ticket here: [DHIS2-15818] - Jira

We are experiencing what I think is the fallout of DHIS2-14021: Created by - UsersDONE being fixed. We were on 2.37.8 (affected version) and are experiencing a problem where some users do NOT have “createdBy” or “user” in their “userCredentials”. My assumption is that this situation was caused by the aforementioned bug. Now on 2.37.9.1 If a super user (or other user) updates that account, those fields are not created. If the user updates their own account, they ARE CREATED.
We have upgraded to 2.37.9.1 and now the downstream affect of a user that does NOT have the createdBy or user fields in userCredentials is that those users cannot POST events to the server - we discovered this while using the android capture app abut the results are the same when POSTing using postman – I didn’t test the browser but expect the same since its also just a POST.
The real challenge with this is that the error returned from the server only returns a generic 500 “internal server error” code/response and there is no logging for it. I had to you our application monitoring tools to capture the “nullPointerException” in the stack trace.

12

@jaime.bosque - I believe the key point that we’ve arrived at is that whatever the fix was from 2.37.8 → 2.37.9 in that linked ticket removed the createdBy and user fields for affected users with the assumption that the next time the user logged in, it would populate with that user’s credentials.However, on android, those fields are not being updated upon login so when they go to send data, the POST request errors out as described in the ticket’s comment

1 Like
13

Hello, @chase.freeman . Thanks a lot for the clarification and reporting the issue, I will watch it to see as this is more like a backend issue. However, why did you assume that those fields will be populated upon logging in? Was this mentioned somewhere I might not be aware?

14

It is not mentioned anywhere - I am just just trying to piece together what is happening. Becuase, when the user logs in in the browser in the new version, it updates their account with “last login” which I am guessing is what prompts dhis2 to assign the “user” field (which I think is the necessary one for POSTing events) and the createdBy (which again is the user itself which if I understand is the original bug, which appears not to be fixed).

Anyway, on android, when the user logs in on the device, it is not sending that update to the server and thus the “user” and “createdBy” fields do not exist which causes the POSTs to fail silently.

So in our case, if our users login via the browser, then log out/log in on the device, they can now POST data.

And yes, it does seem more of a backend issue. And most of this is conjecture based on my understanding of the system, plus what we’re seeing in our use case.