Event-data CSV import issues

Hi

There seems to be at least two bugs related to the import of event-data in CSV format introduced in 2.17.

Firstly, a dry run importing around 29,000 events (~250,000 trackervalue records) into 2.18 (version as of 09 Mar 2015) did not seem to work - no feedback within reasonable time. NOTE that the import with dry run set to NO worked fine, but it was quite slow - so there is a possibility the dry run actually works BUT since it gives no indication of activity for a LOONG time it is perceived as not working. If that is the case, it is important to add a counter or similar to the user messages (e.g. “1000 events imported” … “2000 events imported” etc).

Secondly, DHIS2 has a confusing set of standards for uppercase and lowercase keywords. When you export some event data to CSV, it will export the content of “ProvidedElsewhere” as it is stored: e.g. “FALSE” as an uppercase word. If you try to import the same file, it is rejected and the tomcat log indicates that only “false” and “true” are accepted (lowercase).

Thirdly, from a more “cosmetic” perspective, I find it annoying that a tomcat log entry is produced for EVERY event imported - it would be better if only “abnormal” issues are logged otherwise any abnormalities or warnings drowns in the “info about every event record” clutter.

Regards

Calle

···

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


hi Calle

FYI I have some database scripts that I used to import events into our program - 300k records in 10 mins or so. if you need to bulk insert events let me know and I can share. p

···

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


Also I think the performance was enhanced recently.

···

On 11 Mar 2015 23:09, “Pierre Dane” pierre@jembi.org wrote:

hi Calle

FYI I have some database scripts that I used to import events into our program - 300k records in 10 mins or so. if you need to bulk insert events let me know and I can share. p

On 11 Mar 2015, at 23:49, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

There seems to be at least two bugs related to the import of event-data in CSV format introduced in 2.17.

Firstly, a dry run importing around 29,000 events (~250,000 trackervalue records) into 2.18 (version as of 09 Mar 2015) did not seem to work - no feedback within reasonable time. NOTE that the import with dry run set to NO worked fine, but it was quite slow - so there is a possibility the dry run actually works BUT since it gives no indication of activity for a LOONG time it is perceived as not working. If that is the case, it is important to add a counter or similar to the user messages (e.g. “1000 events imported” … “2000 events imported” etc).

Secondly, DHIS2 has a confusing set of standards for uppercase and lowercase keywords. When you export some event data to CSV, it will export the content of “ProvidedElsewhere” as it is stored: e.g. “FALSE” as an uppercase word. If you try to import the same file, it is rejected and the tomcat log indicates that only “false” and “true” are accepted (lowercase).

Thirdly, from a more “cosmetic” perspective, I find it annoying that a tomcat log entry is produced for EVERY event imported - it would be better if only “abnormal” issues are logged otherwise any abnormalities or warnings drowns in the “info about every event record” clutter.

Regards

Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Knut,

Unless it was enhanced during the last 2-3 days, then it is still very slow compared to other data imports (e.g importing 5.6 mill datavalue records take around 10 minutes on my laptop - importing ~29,000 events with 250k event values took about twice that)

regards

calle

···

On 12 March 2015 at 04:11, Knut Staring knutst@gmail.com wrote:

Also I think the performance was enhanced recently.

On 11 Mar 2015 23:09, “Pierre Dane” pierre@jembi.org wrote:

hi Calle

FYI I have some database scripts that I used to import events into our program - 300k records in 10 mins or so. if you need to bulk insert events let me know and I can share. p

On 11 Mar 2015, at 23:49, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

There seems to be at least two bugs related to the import of event-data in CSV format introduced in 2.17.

Firstly, a dry run importing around 29,000 events (~250,000 trackervalue records) into 2.18 (version as of 09 Mar 2015) did not seem to work - no feedback within reasonable time. NOTE that the import with dry run set to NO worked fine, but it was quite slow - so there is a possibility the dry run actually works BUT since it gives no indication of activity for a LOONG time it is perceived as not working. If that is the case, it is important to add a counter or similar to the user messages (e.g. “1000 events imported” … “2000 events imported” etc).

Secondly, DHIS2 has a confusing set of standards for uppercase and lowercase keywords. When you export some event data to CSV, it will export the content of “ProvidedElsewhere” as it is stored: e.g. “FALSE” as an uppercase word. If you try to import the same file, it is rejected and the tomcat log indicates that only “false” and “true” are accepted (lowercase).

Thirdly, from a more “cosmetic” perspective, I find it annoying that a tomcat log entry is produced for EVERY event imported - it would be better if only “abnormal” issues are logged otherwise any abnormalities or warnings drowns in the “info about every event record” clutter.

Regards

Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


It was enhanced a few days ago, yes… and is now in latest 2.17, 2.18 and trunk

···

On Thu, Mar 12, 2015 at 11:31 AM, Calle Hedberg calle.hedberg@gmail.com wrote:

Knut,

Unless it was enhanced during the last 2-3 days, then it is still very slow compared to other data imports (e.g importing 5.6 mill datavalue records take around 10 minutes on my laptop - importing ~29,000 events with 250k event values took about twice that)

regards

calle


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Morten

On 12 March 2015 at 04:11, Knut Staring knutst@gmail.com wrote:

Also I think the performance was enhanced recently.

On 11 Mar 2015 23:09, “Pierre Dane” pierre@jembi.org wrote:

hi Calle

FYI I have some database scripts that I used to import events into our program - 300k records in 10 mins or so. if you need to bulk insert events let me know and I can share. p

On 11 Mar 2015, at 23:49, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi

There seems to be at least two bugs related to the import of event-data in CSV format introduced in 2.17.

Firstly, a dry run importing around 29,000 events (~250,000 trackervalue records) into 2.18 (version as of 09 Mar 2015) did not seem to work - no feedback within reasonable time. NOTE that the import with dry run set to NO worked fine, but it was quite slow - so there is a possibility the dry run actually works BUT since it gives no indication of activity for a LOONG time it is perceived as not working. If that is the case, it is important to add a counter or similar to the user messages (e.g. “1000 events imported” … “2000 events imported” etc).

Secondly, DHIS2 has a confusing set of standards for uppercase and lowercase keywords. When you export some event data to CSV, it will export the content of “ProvidedElsewhere” as it is stored: e.g. “FALSE” as an uppercase word. If you try to import the same file, it is rejected and the tomcat log indicates that only “false” and “true” are accepted (lowercase).

Thirdly, from a more “cosmetic” perspective, I find it annoying that a tomcat log entry is produced for EVERY event imported - it would be better if only “abnormal” issues are logged otherwise any abnormalities or warnings drowns in the “info about every event record” clutter.

Regards

Calle


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg



Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp


Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp


Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg


1 Like

Hi @pierredane

I am intrested with the SQL script that you use to insert record to database.
I would be great if you have also a script to isert record from tracker capture program.
Could you share with me please?

Thanks a lot
Patrick

1 Like