There seems to be at least two bugs related to the import of event-data in CSV format introduced in 2.17.
Firstly, a dry run importing around 29,000 events (~250,000 trackervalue records) into 2.18 (version as of 09 Mar 2015) did not seem to work - no feedback within reasonable time. NOTE that the import with dry run set to NO worked fine, but it was quite slow - so there is a possibility the dry run actually works BUT since it gives no indication of activity for a LOONG time it is perceived as not working. If that is the case, it is important to add a counter or similar to the user messages (e.g. “1000 events imported” … “2000 events imported” etc).
Secondly, DHIS2 has a confusing set of standards for uppercase and lowercase keywords. When you export some event data to CSV, it will export the content of “ProvidedElsewhere” as it is stored: e.g. “FALSE” as an uppercase word. If you try to import the same file, it is rejected and the tomcat log indicates that only “false” and “true” are accepted (lowercase).
Thirdly, from a more “cosmetic” perspective, I find it annoying that a tomcat log entry is produced for EVERY event imported - it would be better if only “abnormal” issues are logged otherwise any abnormalities or warnings drowns in the “info about every event record” clutter.
FYI I have some database scripts that I used to import events into our program - 300k records in 10 mins or so. if you need to bulk insert events let me know and I can share. p
FYI I have some database scripts that I used to import events into our program - 300k records in 10 mins or so. if you need to bulk insert events let me know and I can share. p
There seems to be at least two bugs related to the import of event-data in CSV format introduced in 2.17.
Firstly, a dry run importing around 29,000 events (~250,000 trackervalue records) into 2.18 (version as of 09 Mar 2015) did not seem to work - no feedback within reasonable time. NOTE that the import with dry run set to NO worked fine, but it was quite slow - so there is a possibility the dry run actually works BUT since it gives no indication of activity for a LOONG time it is perceived as not working. If that is the case, it is important to add a counter or similar to the user messages (e.g. “1000 events imported” … “2000 events imported” etc).
Secondly, DHIS2 has a confusing set of standards for uppercase and lowercase keywords. When you export some event data to CSV, it will export the content of “ProvidedElsewhere” as it is stored: e.g. “FALSE” as an uppercase word. If you try to import the same file, it is rejected and the tomcat log indicates that only “false” and “true” are accepted (lowercase).
Thirdly, from a more “cosmetic” perspective, I find it annoying that a tomcat log entry is produced for EVERY event imported - it would be better if only “abnormal” issues are logged otherwise any abnormalities or warnings drowns in the “info about every event record” clutter.
Unless it was enhanced during the last 2-3 days, then it is still very slow compared to other data imports (e.g importing 5.6 mill datavalue records take around 10 minutes on my laptop - importing ~29,000 events with 250k event values took about twice that)
FYI I have some database scripts that I used to import events into our program - 300k records in 10 mins or so. if you need to bulk insert events let me know and I can share. p
There seems to be at least two bugs related to the import of event-data in CSV format introduced in 2.17.
Firstly, a dry run importing around 29,000 events (~250,000 trackervalue records) into 2.18 (version as of 09 Mar 2015) did not seem to work - no feedback within reasonable time. NOTE that the import with dry run set to NO worked fine, but it was quite slow - so there is a possibility the dry run actually works BUT since it gives no indication of activity for a LOONG time it is perceived as not working. If that is the case, it is important to add a counter or similar to the user messages (e.g. “1000 events imported” … “2000 events imported” etc).
Secondly, DHIS2 has a confusing set of standards for uppercase and lowercase keywords. When you export some event data to CSV, it will export the content of “ProvidedElsewhere” as it is stored: e.g. “FALSE” as an uppercase word. If you try to import the same file, it is rejected and the tomcat log indicates that only “false” and “true” are accepted (lowercase).
Thirdly, from a more “cosmetic” perspective, I find it annoying that a tomcat log entry is produced for EVERY event imported - it would be better if only “abnormal” issues are logged otherwise any abnormalities or warnings drowns in the “info about every event record” clutter.
Unless it was enhanced during the last 2-3 days, then it is still very slow compared to other data imports (e.g importing 5.6 mill datavalue records take around 10 minutes on my laptop - importing ~29,000 events with 250k event values took about twice that)
FYI I have some database scripts that I used to import events into our program - 300k records in 10 mins or so. if you need to bulk insert events let me know and I can share. p
There seems to be at least two bugs related to the import of event-data in CSV format introduced in 2.17.
Firstly, a dry run importing around 29,000 events (~250,000 trackervalue records) into 2.18 (version as of 09 Mar 2015) did not seem to work - no feedback within reasonable time. NOTE that the import with dry run set to NO worked fine, but it was quite slow - so there is a possibility the dry run actually works BUT since it gives no indication of activity for a LOONG time it is perceived as not working. If that is the case, it is important to add a counter or similar to the user messages (e.g. “1000 events imported” … “2000 events imported” etc).
Secondly, DHIS2 has a confusing set of standards for uppercase and lowercase keywords. When you export some event data to CSV, it will export the content of “ProvidedElsewhere” as it is stored: e.g. “FALSE” as an uppercase word. If you try to import the same file, it is rejected and the tomcat log indicates that only “false” and “true” are accepted (lowercase).
Thirdly, from a more “cosmetic” perspective, I find it annoying that a tomcat log entry is produced for EVERY event imported - it would be better if only “abnormal” issues are logged otherwise any abnormalities or warnings drowns in the “info about every event record” clutter.
I am intrested with the SQL script that you use to insert record to database.
I would be great if you have also a script to isert record from tracker capture program.
Could you share with me please?