In 2.34.2 on our Sandbox Server, I have the following read-out of an event report:
When I pull the csv for this report, I get:
Event
Program stage
Event date
Geometry
Longitude
Latitude
Organisation unit name
Organisation unit code
Organisation unit
GLO- Name of respondent
gvKaxTWigQO
sZpSadHhAXb
00:00.0
0
0
Benin Test Village
BJ-
NhNb0IUbF5V
Test
v6N49ZEDUIV
sZpSadHhAXb
00:00.0
0
0
Benin Test Village
BJ-
NhNb0IUbF5V
Test (One Fourteen PM EST on Friday, December Eleventh)
gvKaxTWigQO
sZpSadHhAXb
00:00.0
0
0
Benin Test Village
BJ-
NhNb0IUbF5V
Test
v6N49ZEDUIV
sZpSadHhAXb
00:00.0
0
0
Benin Test Village
BJ-
NhNb0IUbF5V
Test (One Fourteen PM EST on Friday, December Eleventh)
So basically:
4 events are appearing, but in reality they are 2 events (both duplicated, including the event UID)
When I pull the same dimensions in OU and Period and Program (and the same User), it returns 5 events to me. I can confirm that all 5 events are real and accurate.
Some Notes:
When we initially tried to load tables (for the first time), after 26+ hours of grinding, the server shut down. I then did not re-push tables, and instead ran analytics table while skipping regeneration of tables
After the server came back, I also started Continuous Analytics through the Scheduler, with a daily restart at 1:00am
My questions:
Obviously the server shutting down and restarting could have resulted in analytics not completing. Does that mean that it could, in this scenario, duplicate events? Those feel like two different issues altogether in my mind?
Wouldn’t Continuous Analytics, during the nightly run, remove these events, if they existed?
Hi @Matthew_Boddie This one is quite puzzling indeed. Can you provide more information on the dimensions that were selected and a screenshot of the dev console to see the server requests would also be helpful.
@Scott I’m gathering the documentation to try and make this as clear as possible, but we just re-tested Continuous Analytics on 2.35.4 and are having the same issue (data gets in quickly and accurately, but then after 24 hours and beyond, can see duplicate data events.
Curious if you’ve heard this happening elsewhere at all? Fairly basic set-up that I have in scheduler, with a 2 year CA analytics tables running with a 60 second delay (full refresh at 1am), and an additional resource tables scheduled at midnight. Again, will come back here soon with more documentation.
Imported 3266 Events from Prod to Staging. These were BF Details data for IRS, all in 2021.
6/11/2021 8:30am
Events are visible. Now selecting BF Details for IRS for 2021 period, 17944 events present in event reports
9 minutes for visibility
Once CA starts, the old is removed from the system/tasks. It does still get logged in the manager.baosystems.com analytics_tables log, and shows time taken and all information related
6/11/2021 8:34am
Imported 3026 events (869 created, 2157 updated). Again BF Details for data for IRS, all in 2021
6/11/2021 8:37am
Continuous Analytics starts (SMhwurwU5J1)
6/11/2021 8:52am
CA ends. Events are visible. Same dimensions above, now 18813 events (869 more visible)
15 minutes for visibility
Import Test passes quite well. Would be good to augment this test with data from other programs, but ultimately IRS Details is also a good test for analytics because of the amount of data that exists in it for 2021. CPU usage does increase notably with the need for a run; that said, importing 3266 and 3026 events didn’t result in any lack of functionality or noticeable latency in the system. Also noteworthy that during the analytics work, no other users were active in staging.
Can be a bit difficult to track CA given system/tasks removes the results once complete. Also would be interested to see how cotinuous data being imported throughout the data would affect partitions being created and piling up leading up to the midnight run. Could be wortwhile, thus, to consider duration of data entry alongside scale of data entry for thorough testing
|6/14/2021 7:50am|Checking Data with same dimensions above. Now 27,798 Events exist. Pulled CSV, found 8985 duplicate event UIDs (18813 events unique)|||
@Scott this is a quick run-down of what I did, and how data started to become duplicated in the analytics of one of our non-prod servers. I’d be happy to take you/anyone through this quickly so that another look at it can be given.
@chase.freeman tagging you here as I know you were interested in the outcome of our analytics testing.
I just tested this again in 2.35.10, and the exact same thing is happening for our instance. I cleared analytics, manually re-ran resource tables through Data Administration, and then let continuous analytics (through scheduler) build out the data. At first this looked good, (data was coming in very fast into analytics, and no signs of issue). Then I came back the next day, and there are two copies of the data that was entered.
I’d very very much love to be running CA for our instance–our analytics run time takes over an hour to complete. How can we get this functional? How can I help figure out what might be going wrong?
@Matthew_Boddie thank you for the detailed report and for the tag. Would I please request that you also share the Catalina log (just in case it might be useful) without including any sensitive info?
I will asking for further support and following up on this, thanks!
Sorry for just now responding; the tomcat.catalina.out for this server doesn’t have much, and nothing related to the continuous analytics. I don’t think it will be too helpful.
I’m going to try and recreate this experience in play over the next week. Hopefully I can get it (not) functioning!
Okay great, thanks for checking out! I’m not sure but we can triage this for further support. Would you take screenshots of the errors you’re facing now?
I believe the issue on this was elegantly described here [DHIS2-13479] - Jira by @Timothy_Harding1 , and I think his description allowed for a fix to be possible. Much appreciation to all—excited to test out 2.37.9 and confirm this works well for us.
is the GET request that comes through when pulling a event report