Grouped events in Maps do not show correctly

Dear All

When using the Maps app, we have noticed that grouped events sometimes do not display properly. When there are a low number of events to group together, the grouping works. However, when a larger number (a few hundred) events are entered for the same or similar coordinates, then the grouping feature does not work and the events are simply not displayed at all. The map remains blank. However, if you then switch to individual events being displayed, the data shows. I have attached a screenshot of the error when grouping events.

Map%20error%20-%20grouping%20samples

I checked this on the Play instance and it cannot be replicated. I was wondering whether anyone could assist me in understanding this error.

Much appreciated!

DHIS 2.30 build revision 552c60a

2 Likes

Terence,

Firstly, note the error message: “ERROR: Input geometry has unknown (0) SRID”.

Secondly, an SRID in postGIS is a Spatial Reference ID - in general the same number as the EPSG code (e.g. =4326 for the “WGS-84” datum).

Thirdly, so your problem seem to be rooted in one of two things:

  • Some of your event data might be stored with an SRID of 0, which technically has no meaning (it’s like a “NULL” value for SRIDs, and will cause problems) and other data with e.g. SRID 4326 (the standard supported by DHIS2). If you retrieve a mix of geopositions, some with SRID=0 and some with SRID=4326, Postgis will throw an error.

OR

  • the SRID number is somehow dropped or disregarded when DHIS2 is retrieving and grouping more than X events (i.e. it’s a bug).

In terms of the first possibility, note that you should be able to run an SRID update script using .STSrid=4326; on any postgis geometry field.

My 2c worth
Calle

2 Likes

Hi, the reason you get different results when you have a low number of a events is that we use a different cluster algorithm in the browser. If you have more than 2000 events we do the clustering on the server where the error occurs.

I agree with Calle, there seems to be a missing SRID for the database column where the event coordinates are stored. The SRID should be 4326.

Bjørn,
Maps app developer

2 Likes

@Terence_Scott, which coordinate field do you use? “Event location” or another?

1 Like

Hi @Calle_Hedberg and @Bjorn_Sandvik

Thank you very much for your responses. Unfortunately, I am a noob when it comes to these things so I unfortunately am unsure what you mean by this. So it seems from what I understood (and the most likely option from what Calle mentioned) is that we are missing the SRID for some of the event data. You mentioned using an update script, but I am not sure how one would go about this.

Bjorn, in response to your question, we have been using “Event location” for the coordinate field when drawing the maps.

I am not sure if this error could have originated in the way that we have been importing back-data into the event program. We have been doing mass imports of data, and have been importing the information using both lat/long as well as the “Point coordinates” field. I have attached an example of the import data that we have been using also if that helps at all.

Event import CSV file example.csv (414 Bytes)

I appreciate your help so far and your rapid input!

2 Likes

Terence,

If I understand you correctly with regard to that CSV import file: you created a CSV file from external data that had values in BOTH the “old” (legacy) longitude and latitude fields AND in the new Geometry field?

Or is that CSV sample file a re-export from the system?

Also, just to confirm: you DO have access directly to the database?

Regards
Calle

2 Likes

Hi Calle,

So the CSV import file was created based directly on an exported file from the system. I.e. we simply changed values, but left the format exactly as is - as that was the only means we could figure out to import the back-data for the time being.

So to answer your question, Yes, we imported using both the Legacy (Lat/Long) fields as well as the Geometry field: Point (long lat). We did this, as when we attempted to upload using only the legacy field, the coordinates were not saved.

I am a superuser on the front end, but do not have direct access to the backend, but this can be arranged if needed. However, I have no specific programming/computer language understanding.

2 Likes

Terence,

Apologies, I was leading you off into the wilderness above with that STSrid partial statement.

Ask your db admin to run the following small script on your database, to ensure that the SRID of the programstageinstance table is 4326 (the default geographic WGS-84 graticule, the only one supported by DHIS2):

SELECT UpdateGeometrySRID(‘programstageinstance’,‘geometry’,4326);

You can use the same for any other DHIS2 table with a “geometry” column, just change the table name.

NOTE: I’m not guaranteeing that the above will fix your problem, but at least it will ensure that the SRID is correct. If that does NOT fix the problem, then it looks like a server-side bug to me - so write a JIRA issue for it.
@Bjorn_Sandvik is only working on the Maps app etc (UI side), but he would know to whom such a JIRA bug should be assigned.

Regards
Calle

2 Likes

Fantastic! Thank you very much Calle. I will ask them to run it and will keep you updated as to whether this would solve the issue.

Thanks for your patience and assistance!

2 Likes

@Terence_Scott

It’s OK. This is not the forum to educate people on GIS - and the GIS knowledge among DHIS2 users are generally low - but for those interested, note the following:

DHIS2 only supports SRID / EPSG 4326 - which is equivalent to the geographical graticule (earth model) that often are referred to as WGS-84 (World Geodetic System of 1984).
2.
Geographical longitudes goes either East from the 0 meridian (in school you might have heard of it as the “Greenwich Meridian”) to +180 deg in the Pacific, or West from the 0 deg meridian to -180 deg in the Pacific. Geographical latitude from 0 degrees (equator) to +90 degrees North (the true North Pole) and from 0 deg (equator) to -90 degrees (true South Pole) in the southern hemisphere.

DHIS2, like nearly all GIS systems, is also only accepting coordinates in the DECIMAL DEGREES format - not e.g. Decimal degrees, minutes, and seconds that many learned about in school (navigation is heavily influenced by it’s marine origins and mariners used deg-min-seconds, but that’s another story). So all coordinates you capture or import into DHIS2 must be from -180.000000 to +180.000000 for longitudes and from -90.000000 to +90.000000 degrees for latitudes (six decimals are equivalent to approximately 50cm accuracy, which is more than sufficient for health applications)

If you find that you have coordinates like 170543 or something like that, it means those coordinates are coming from a PROJECTED map - and you must PROJECT such coordinates back to geographical coordinates before using/importing them into DHIS2.

What this means e.g. for Africa is that west of approximately Togo, you will have a negative longitude (west of 0 deg) and a positive latitude (north of the Equator). From Togo and eastwards, all longitudes are positive - BUT latitudes will be positive north of the equator and negative south of the Equator. Asian countries are all having positive longitudes and (north of the equator) positive latitudes. This basic understanding is what I often find lacking among users, especially data capturers - so if they capture coordinates manually you get a lot of East/west values longitude values mixed up due to the minus sign, and a lot of north/south values mixed up too.

Some countries use local variants of WGS-84 to increase accuracy within their borders - like South Africa is using an official datum called Hartebeeshoek - but the differential between those and WGS-84 are usually less than 1-2 metres and thus irrelevant for health purposes.

For DHIS2 version up to 2.29, standard coordinates were generally stored in two fields called “longitude” and “latitude”. The first step in the shift to using PostGIS geometry happened when the “COORDINATE” data type was introduced. The second step from 2.30 was to replace all those long/lat fields in various key tables with a “geometry” field. In 2.30, only the three tables “organisationunit”, “map”, and “programinstance” still have the lat/long fields) - in 2.31.1 it’s only the first two left, while “programinstance”, “programstageinstance”, and “trackedentityinstance” tables now have a “geometry” field.

During this transition it is not a surprise that bugs are creeping in or mistakes made - for instance I picked up in Nov-18 that the the maps and GIS apps did not display coordinate data elements of data type “COORDINATE”, the apps only displayed e.g. events using the default coordinates. That bug was fixed, and they had to back-port it several versions, indicating that nobody had picked it during the last 1-2 years - which again reflects the fact that the GIS functionality in DHIS2, just like GIS functionalities in every other health information system I’ve looked at over the last 20-25 years, are under-utilised. Which again reflect the fact that most health managers and health personnel are not familiar with the analysis and interpretation of spatial data and maps. That again reflect the fact that many many Low and Middle income countries historically have restricted access to maps and spatial data…

Finally, also note that the DHIS2 user interface for geographical data is poor/inconsistent:

  • While the global standard for storing and displaying coordinates is longitude-latitude, and DHIS2 stores them like that, it often DISPLAY them as latitude-longitude.
  • While the words latitude and longitude are displayed for coordinate fields when those fields are empty, the words disappear after values are entered, so when a user reviews or edits coordinates it’s easy to mix them up.
  • There are no features to limiting coordinates to a specific range.
  • There are no features for e.g. limiting point coordinates to fall within a specific area, like within a country
  • You cannot use geometry fields in program rules
    and so on. It all boils down to a lot of garbage coordinates being captured or imported, causing problems which in turn reduces the useability and thus usage.

Nuff’ said - that was today’s sermon from the spatial pulpit :wink:

Regards
Calle

2 Likes

Hi @Calle_Hedberg

Apologies for the delay, but we were finally able to run the script as suggested and it worked like a charm!

Thanks so much for your input and help!

2 Likes

@Terence_Scott

Nice to see that dhis-jurassic-park creatures like me are still able to come up with a solution now and then ;-o

Regards
Calle

2 Likes

Hi All

I am back with the same issue again. The script that @Calle_Hedberg mentioned did help us for a short while. However, soon after this, we encountered several errors when attempting to capture event data, as the event data would not save and we received an internal error 500. The event import also did not work and gave the same error. Seemingly, the instance was unable to communicate with the server wrt event capture. We then upgraded the build version and we were then able to successfully import events. However, we have reverted to the issue relating to the grouping of events on a map. When moving above a certain number of events, the grouping function apparently uses a different cluster algorithm.

BAO (@dhuser) (our server hosts) then looked into the issue and came to the following conclusions:

  • Coordinates are valid via SELECT uid,geometry FROM programstageinstance WHERE (st_x(geometry) NOT BETWEEN -180 AND 180) OR (st_y(geometry) NOT BETWEEN -90 AND 90); (0 results)

  • The geometry column in CSV import can be ignored

  • Transferring the CSV into a JSON payload matching to a play.dhis2.org/2.30 = same problem (event attached - first row of your CSV)

  • When setting the programstageinstance table to SRID 0 SELECT UpdateGeometrySRID('programstageinstance','geometry',4326);

    • You can import events

    • You cannot visualize clusters (“ERROR: Input geometry has unknown (0) SRID”)

  • When setting the programstageinstance table to SRID 4326 SELECT UpdateGeometrySRID('programstageinstance','geometry',0);

    • You cannot import events (“ERROR: Geometry SRID (0) does not match column SRID (4326)”)

    • You can visualize clusters

We looked through some of the suggested JIRA fixes ( DHIS2-6325 , DHIS2-6575 ) for what could be related to this issue and it seems that these have been potentially corrected in version 2.32.

We are currently hesitant to upgrade to 2.32 at the moment and would like to therefore ask whether this is in fact the case, as this is a major problem for the depiction of our event data.

Any help would once again be appreciated.

1 Like