2.34.2, 2.35.10 Continuous Analytics Duplicating Events, omitting others

In 2.34.2 on our Sandbox Server, I have the following read-out of an event report:
image

When I pull the csv for this report, I get:

Event Program stage Event date Geometry Longitude Latitude Organisation unit name Organisation unit code Organisation unit GLO- Name of respondent
gvKaxTWigQO sZpSadHhAXb 00:00.0 0 0 Benin Test Village BJ- NhNb0IUbF5V Test
v6N49ZEDUIV sZpSadHhAXb 00:00.0 0 0 Benin Test Village BJ- NhNb0IUbF5V Test (One Fourteen PM EST on Friday, December Eleventh)
gvKaxTWigQO sZpSadHhAXb 00:00.0 0 0 Benin Test Village BJ- NhNb0IUbF5V Test
v6N49ZEDUIV sZpSadHhAXb 00:00.0 0 0 Benin Test Village BJ- NhNb0IUbF5V Test (One Fourteen PM EST on Friday, December Eleventh)

So basically:
4 events are appearing, but in reality they are 2 events (both duplicated, including the event UID)

When I pull the same dimensions in OU and Period and Program (and the same User), it returns 5 events to me. I can confirm that all 5 events are real and accurate.

Some Notes:

  1. When we initially tried to load tables (for the first time), after 26+ hours of grinding, the server shut down. I then did not re-push tables, and instead ran analytics table while skipping regeneration of tables
  2. After the server came back, I also started Continuous Analytics through the Scheduler, with a daily restart at 1:00am

My questions:

  1. Obviously the server shutting down and restarting could have resulted in analytics not completing. Does that mean that it could, in this scenario, duplicate events? Those feel like two different issues altogether in my mind?
  2. Wouldn’t Continuous Analytics, during the nightly run, remove these events, if they existed?

Hi @Matthew_Boddie This one is quite puzzling indeed. Can you provide more information on the dimensions that were selected and a screenshot of the dev console to see the server requests would also be helpful.

1 Like

https://xxx/api/29/analytics/events/query/BLGobttHq7p.json?dimension=pe:LAST_YEAR&dimension=ou:NhNb0IUbF5V&dimension=sZpSadHhAXb.QFfaXNW5tVX&stage=sZpSadHhAXb&displayProperty=NAME&outputType=EVENT&desc=eventdate&pageSize=100&page=1

is the GET request that comes through when pulling a event report

Not sure if the short gif is useful, but attaching along as well.

@Scott I’m gathering the documentation to try and make this as clear as possible, but we just re-tested Continuous Analytics on 2.35.4 and are having the same issue (data gets in quickly and accurately, but then after 24 hours and beyond, can see duplicate data events.

Curious if you’ve heard this happening elsewhere at all? Fairly basic set-up that I have in scheduler, with a 2 year CA analytics tables running with a 60 second delay (full refresh at 1am), and an additional resource tables scheduled at midnight. Again, will come back here soon with more documentation.

Date/Time Summary Notes Notes
6/11/2021 8:21am Imported 3266 Events from Prod to Staging. These were BF Details data for IRS, all in 2021.
6/11/2021 8:30am Events are visible. Now selecting BF Details for IRS for 2021 period, 17944 events present in event reports 9 minutes for visibility Once CA starts, the old is removed from the system/tasks. It does still get logged in the manager.baosystems.com analytics_tables log, and shows time taken and all information related
6/11/2021 8:34am Imported 3026 events (869 created, 2157 updated). Again BF Details for data for IRS, all in 2021
6/11/2021 8:37am Continuous Analytics starts (SMhwurwU5J1)
6/11/2021 8:52am CA ends. Events are visible. Same dimensions above, now 18813 events (869 more visible) 15 minutes for visibility

Import Test passes quite well. Would be good to augment this test with data from other programs, but ultimately IRS Details is also a good test for analytics because of the amount of data that exists in it for 2021. CPU usage does increase notably with the need for a run; that said, importing 3266 and 3026 events didn’t result in any lack of functionality or noticeable latency in the system. Also noteworthy that during the analytics work, no other users were active in staging.

Can be a bit difficult to track CA given system/tasks removes the results once complete. Also would be interested to see how cotinuous data being imported throughout the data would affect partitions being created and piling up leading up to the midnight run. Could be wortwhile, thus, to consider duration of data entry alongside scale of data entry for thorough testing

|6/14/2021 7:50am|Checking Data with same dimensions above. Now 27,798 Events exist. Pulled CSV, found 8985 duplicate event UIDs (18813 events unique)|||

@Scott this is a quick run-down of what I did, and how data started to become duplicated in the analytics of one of our non-prod servers. I’d be happy to take you/anyone through this quickly so that another look at it can be given.

@chase.freeman tagging you here as I know you were interested in the outcome of our analytics testing.

I just tested this again in 2.35.10, and the exact same thing is happening for our instance. I cleared analytics, manually re-ran resource tables through Data Administration, and then let continuous analytics (through scheduler) build out the data. At first this looked good, (data was coming in very fast into analytics, and no signs of issue). Then I came back the next day, and there are two copies of the data that was entered.

I’d very very much love to be running CA for our instance–our analytics run time takes over an hour to complete. How can we get this functional? How can I help figure out what might be going wrong?

cc @Gassim
cc @chase.freeman

@Matthew_Boddie thank you for the detailed report and for the tag. Would I please request that you also share the Catalina log (just in case it might be useful) without including any sensitive info?

I will asking for further support and following up on this, thanks!

@Gassim happy to work on that–not knowledgeable in this area though. Are you speaking of tomcat.catalina.out?

1 Like

Yes, thanks!

Sorry for just now responding; the tomcat.catalina.out for this server doesn’t have much, and nothing related to the continuous analytics. I don’t think it will be too helpful.

I’m going to try and recreate this experience in play over the next week. Hopefully I can get it (not) functioning!

1 Like

I’ve created a scheduler for continuous analytics on play 2.35.10 (DHIS 2 Demo - Sierra Leone) but it’s failing. I’m wondering if this is actually related to this: Continuous Analytics Scheduler Jobs Require a Server Restart
@Gassim

1 Like

Okay great, thanks for checking out! I’m not sure but we can triage this for further support. Would you take screenshots of the errors you’re facing now?

There is ongoing investigation ( Log in - DHIS 2 JIRA, hope you are granted to see )

3 Likes