SMS Submission failed to data to the Dataset

Gerald_Thomas · 11 July 2022 18:38

Dear All,
In Sierra Leone, we are implementing eIDSR and part of the solution is using an SMS channel through Rapidpro (webhook) and SMS comprehension service to submit data to DHIS2. Our DHIS2 was affected and we had to recreate it again from the backup. The data flow is working as it should:
Mobile tables------------Rapidpro===SMS_Compression_Server-------------DHIS2
From Mobile to SMS_Compression_Server it is OK. In DHIS2, data as SMS commands failed to submit.
Attached are the screenshots of failed SMS. I already tried to re-import them again but this is the error I see in the tomcat:
ul 11 14:51:29 hmis tomcat9[269]: * INFO 2022-07-11 14:51:29,642 No template found (DefaultDataSetNotificationService.java [tomcat-http-256])

Jul 11 14:51:29 hmis tomcat9[269]: * INFO 2022-07-11 14:51:29,655 Complete data set registration import done, total: 1, imported: 1, updated: 0, deleted: 0: 00:00:00.067 (Clock.java [tomcat-http-256])

Gassim · 11 July 2022 18:59

Thank you for sharing! Please check if there is more info we could get from the Catalina.out log (without the sensitive info)?

It might be helpful if you share the steps you are taking right before seeing the issues. Any errors in the Network tab, too?

Thanks!

zubair · 11 July 2022 19:23

@Gerald_Thomas can you share SMS command format?

Gerald_Thomas · 11 July 2022 22:21

Please see the screenshot of the SMS commands for the data elements

Gerald_Thomas · 13 July 2022 02:25

@zubair,
Please can you help?

zubair · 13 July 2022 02:37

@Gerald_Thomas
can you try using pipe ’ | ’ instead of ’ = ’

Gerald_Thomas · 13 July 2022 02:51

Dear Zubair,
I already did but I have 2 hidden data elements which do not need the SMS command and I don’t know how to remove them from the SMS command

Gerald_Thomas · 13 July 2022 02:54

Because those 2 data elements do not have SMS commands don’t know if the changes will work

jaime.bosque · 13 July 2022 07:43

Hey @Gerald_Thomas . How are you sending this messages? Are you using a manual SMS? Any specific application?

zubair · 13 July 2022 07:56

you can assign some code to those datalements and save sms command @Gerald_Thomas

Gerald_Thomas · 13 July 2022 10:39

Dear Zubair,
I have already done that now though. Though they are still failing. Please can I have 30 minutes of your time, just to look through and you make recommendations. It was working before so I don’t think it is anything major.

Gerald_Thomas · 13 July 2022 10:42

Dear Jaime,
There is a third-party system deployed between the Mobile tablets and the DHIS2 server which Rapidpro patched with an SMS compression server to increase the number of SMSes the system can capture at one time.

jaime.bosque · 13 July 2022 11:00

Hi @Gerald_Thomas . Understood. I am saying because the official Android App uses a compression library which is being used at server level to decompress. And if you would be using the same in your third party App you didn’t need to pass by the Dataset command list.

Gerald_Thomas · 13 July 2022 15:49

Dear Jaime,
We are not using the same new app. We are using version 3.1.2

jaime.bosque · 14 July 2022 05:00

I guess that 3.1.2 is the version of the custom app. In any case, as said, if this custom app would be using the new SMS libraries you could get rid of the configuration at the server level, which I am not sure is an option.

Gerald_Thomas · 14 July 2022 22:55

Dear Zubair,
I already added the SMS commands for the remaining 2 data elements and I tried to re-import but I am getting this from the log:

ul 14 22:51:55 hmis tomcat9[313]: * INFO 2022-07-14 22:51:55,607 Simple Local Cache instance created for region:‘userApprovalLevelCache’ (SimpleCacheBuilder.java [tomcat-http-1])
Jul 14 22:52:29 hmis tomcat9[313]: * INFO 2022-07-14 22:52:29,935 No template found (DefaultDataSetNotificationService.java [tomcat-http-2])
Jul 14 22:52:30 hmis tomcat9[313]: * INFO 2022-07-14 22:52:30,345 Sending SMS to [23233395481] (SmsMessageSender.java [tomcat-http-2])
Jul 14 22:52:31 hmis tomcat9[313]: * INFO 2022-07-14 22:52:31,714 Starting validation analysis for orgUnit brfOQ6V3QWO, period 202206, 0 rules: 00:00:00.000 (Clock.java [tomcat-http-6])
Jul 14 22:52:31 hmis tomcat9[313]: * INFO 2022-07-14 22:52:31,757 Initialized validation analysis: 00:00:00.042 (Clock.java [tomcat-http-6])
Jul 14 22:52:31 hmis tomcat9[313]: * INFO 2022-07-14 22:52:31,758 Finished validation analysis, 0 results: 00:00:00.044 (Clock.java [tomcat-http-6])
Jul 14 22:52:32 hmis tomcat9[313]: * INFO 2022-07-14 22:52:32,083 Starting complete data set registration import, options: ImportOptions{idSchemes=IdSchemes{idScheme=IdScheme{identifiableProperty=UID, attribute=null}, dataElementIdScheme=null, categoryOptionComboIdScheme=null, categoryOptionIdScheme=null, orgUnitIdScheme=null, programIdScheme=null, programStageIdScheme=null, trackedEntityIdScheme=null, trackedEntityAttributeIdScheme=null, dataSetIdScheme=null, attributeOptionComboIdScheme=null, programStageInstanceIdScheme=null}, dryRun=false, preheatCache=null, async=false, importStrategy=NEW_AND_UPDATES, mergeMode=REPLACE, skipExistingCheck=false, ignoreEmptyCollection=false, sharing=false, skipNotifications=false, datasetAllowsPeriods=false, strictPeriods=false, strictDataElements=false, strictCategoryOptionCombos=false, strictAttributeOptionCombos=false, strictOrganisationUnits=false, requireCategoryOptionCombo=false, requireAttributeOptionCombo=false, force=false, skipLastUpdated=false}: 00:00:00.000 (Clock.java [tomcat-http-3])
Jul 14 22:52:32 hmis tomcat9[313]: * INFO 2022-07-14 22:52:32,084 Import options: ImportOptions{idSchemes=IdSchemes{idScheme=IdScheme{identifiableProperty=UID, attribute=null}, dataElementIdScheme=null, categoryOptionComboIdScheme=null, categoryOptionIdScheme=null, orgUnitIdScheme=null, programIdScheme=null, programStageIdScheme=null, trackedEntityIdScheme=null, trackedEntityAttributeIdScheme=null, dataSetIdScheme=null, attributeOptionComboIdScheme=null, programStageInstanceIdScheme=null}, dryRun=false, preheatCache=null, async=false, importStrategy=NEW_AND_UPDATES, mergeMode=REPLACE, skipExistingCheck=false, ignoreEmptyCollection=false, sharing=false, skipNotifications=false, datasetAllowsPeriods=false, strictPeriods=false, strictDataElements=false, strictCategoryOptionCombos=false, strictAttributeOptionCombos=false, strictOrganisationUnits=false, requireCategoryOptionCombo=false, requireAttributeOptionCombo=false, force=false, skipLastUpdated=false} (DefaultCompleteDataSetRegistrationExchangeService.java [tomcat-http-3])
Jul 14 22:52:32 hmis tomcat9[313]: * INFO 2022-07-14 22:52:32,085 Data set scheme: IdScheme{identifiableProperty=null, attribute=null}, org unit scheme: IdScheme{identifiableProperty=null, attribute=null}, attribute option combo scheme: IdScheme{identifiableProperty=null, attribute=null} (ImportConfig.java [tomcat-http-3])
Jul 14 22:52:32 hmis tomcat9[313]: * INFO 2022-07-14 22:52:32,141 No template found (DefaultDataSetNotificationService.java [tomcat-http-3])
Jul 14 22:52:32 hmis tomcat9[313]: * INFO 2022-07-14 22:52:32,151 Complete data set registration import done, total: 1, imported: 1, updated: 0, deleted: 0: 00:00:00.068 (Clock.java [tomcat-http-3])

Gerald_Thomas · 14 July 2022 22:57

Dear Zubair,
What is the view from the logs I really don’t know if it is working?

Jul 15 01:39:05 hmis tomcat9[313]: * INFO 2022-07-15 01:39:05,689 No template found (DefaultDataSetNotificationService.java [tomcat-http-2])
Jul 15 01:39:06 hmis tomcat9[313]: * INFO 2022-07-15 01:39:06,108 Sending SMS to [23276162670] (SmsMessageSender.java [tomcat-http-2])

Gerald_Thomas · 19 July 2022 20:53

Dear all,
Getting a new error today, please can some help? i getting really frustrated here:

Jul 19 20:48:18 hmis tomcat9[520]: * INFO 2022-07-19 20:48:18,138 Sending SMS to [23276710495] (SmsMessageSender.java [taskScheduler-22])
Jul 19 20:48:18 hmis tomcat9[520]: * ERROR 2022-07-19 20:48:18,185 Client error 422 Unprocessable Entity (SimplisticHttpGetGateWay.java [taskScheduler-22])
Jul 19 20:48:18 hmis tomcat9[520]: * ERROR 2022-07-19 20:48:18,186 SMS failed, failure cause: failed (SmsMessageSender.java [taskScheduler-22])
Jul 19 20:48:18 hmis tomcat9[520]: * INFO 2022-07-19 20:48:18,186 [Level: INFO, category: SMS_SEND, time: Tue Jul 19 20:48:18 UTC 2022, message: Message sending failed] (InMemoryNotifier.java [taskScheduler-22])
Jul 19 20:48:18 hmis tomcat9[520]: * INFO 2022-07-19 20:48:18,186 ‘system-process’ create org.hisp.dhis.sms.outbound.OutboundSms (AuditLogUtil.java [taskScheduler-22])

Gassim · 20 July 2022 19:18

Dear @Gerald_Thomas,
Sorry you are getting these errors and unable to solve the issue. Is this an issue connecting the SMS_Compression_Server to DHIS2? Would it work if the data flow was as @jaime.bosque suggested (if I understood correctly): Mobile tables------------Rapidpro===(compression library used by the official Android App)-------------DHIS2

Or for debugging sake would it work if it was: Mobile tables------------Rapidpro===---------DHIS2? My last question is because it might help to pinpoint exactly what works and what exactly isn’t working.

Thanks! Hope we’ll find a solution the soonest.

Gerald_Thomas · 21 July 2022 05:49

The reason for the SMS/compression server is that SMS for one dataset will be more than 160 characters, and it needs to be processed for DHIS2 to understand the SMS. The Rapidpro aggregates the mobile network operators (MNOs) services using USSD, collecting my SMSes and moving them to the SMS/compression server. DHIS2, was built to send SMS but not to receive SMS. With the SMS commands, you can tweak it to send SMS.
DHIS2 received those SMSes from the Mobile devices; an acknowledgement SMS is sent back to the Mobile devices that update the form of the mobile device. Finally, approval is only available in 2.31.9 and lower which is why we cannot upgrade our instances because this works best for us since internet coverage is low countrywide.