Import error

Support - Assistance technique
1

Hi,

I’m getting the following tomcat log import error when trying to import CSV (attached) - any idea what the root cause is (the tomcat messages does not make sense to me). The primary message seems to be “failed to flush batch handler”, whatever that means:

  • INFO 2015-02-13 14:30:12,895 Importing data values (DefaultDataValueSetService.java [taskScheduler-3])

  • ERROR 2015-02-13 14:30:12,934 java.lang.RuntimeException: Failed to flush BatchHandler

at org.amplecode.quick.batchhandler.AbstractBatchHandler.flush(AbstractBatchHandler.java:360)

at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.saveDataValueSet(DefaultDataValueSetService.java:868)

at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.saveDataValueSetCsv(DefaultDataValueSetService.java:539)

at org.hisp.dhis.importexport.action.util.ImportDataValueTask.run(ImportDataValueTask.java:83)

at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: org.postgresql.util.PSQLException: ERROR: value too long for type character varying(31)

at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)

at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)

at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)

at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:561)

at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:405)

at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:333)

at org.amplecode.quick.batchhandler.AbstractBatchHandler.flush(AbstractBatchHandler.java:342)

… 11 more

(DefaultDataValueSetService.java [taskScheduler-3])

  • INFO 2015-02-13 14:30:12,944 [Level: ERROR, category: DATAVALUE_IMPORT, time: Fri Feb 13 14:30:12 CAT 2015, message: Process failed: Failed to flush BatchHandler] (InMemoryNotifier.java [taskScheduler-3])

  • ERROR 2015-02-13 14:30:19,402 Left side ($summary.conflicts.size()) of ‘>’ operation has null value at /dhis-web-importexport/importSummary.vm[line 35, column 33] (Log4JLogChute.java [http-apr-8080-exec-8])

DHIS2_CSV_tboutcomesX.csv (8.29 KB)

···

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

2

Hi Calle,

the problem is that the StoredBy values are 33 characters long, whereas the max size in database / DHIS 2 is 31.

I can see that we don’t validate the stored by property in the data value import. I will fix that.

The short term fix is to reduce the values in the StoredBy column to 31 or less characters.

The storedby column is supposed to hold usernames. We can extend it a bit so it matches the max username length exactly.

Lars

···

On Fri, Feb 13, 2015 at 1:31 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I’m getting the following tomcat log import error when trying to import CSV (attached) - any idea what the root cause is (the tomcat messages does not make sense to me). The primary message seems to be “failed to flush batch handler”, whatever that means:

  • INFO 2015-02-13 14:30:12,895 Importing data values (DefaultDataValueSetService.java [taskScheduler-3])
  • ERROR 2015-02-13 14:30:12,934 java.lang.RuntimeException: Failed to flush BatchHandler

at org.amplecode.quick.batchhandler.AbstractBatchHandler.flush(AbstractBatchHandler.java:360)

at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.saveDataValueSet(DefaultDataValueSetService.java:868)

at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.saveDataValueSetCsv(DefaultDataValueSetService.java:539)

at org.hisp.dhis.importexport.action.util.ImportDataValueTask.run(ImportDataValueTask.java:83)

at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: org.postgresql.util.PSQLException: ERROR: value too long for type character varying(31)

at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)

at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)

at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)

at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:561)

at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:405)

at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:333)

at org.amplecode.quick.batchhandler.AbstractBatchHandler.flush(AbstractBatchHandler.java:342)

… 11 more

(DefaultDataValueSetService.java [taskScheduler-3])

  • INFO 2015-02-13 14:30:12,944 [Level: ERROR, category: DATAVALUE_IMPORT, time: Fri Feb 13 14:30:12 CAT 2015, message: Process failed: Failed to flush BatchHandler] (InMemoryNotifier.java [taskScheduler-3])
  • ERROR 2015-02-13 14:30:19,402 Left side ($summary.conflicts.size()) of ‘>’ operation has null value at /dhis-web-importexport/importSummary.vm[line 35, column 33] (Log4JLogChute.java [http-apr-8080-exec-8])

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

3

Hi Calle,

The key words to look for in the Tomcat log is “Caused by”

In this case, as Lars says, the storedby string is too long:
Caused by: org.postgresql.util.
PSQLException: ERROR: value too long for type character varying(31)

···

On Fri, Feb 13, 2015 at 1:31 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I’m getting the following tomcat log import error when trying to import CSV (attached) - any idea what the root cause is (the tomcat messages does not make sense to me). The primary message seems to be “failed to flush batch handler”, whatever that means:

  • INFO 2015-02-13 14:30:12,895 Importing data values (DefaultDataValueSetService.java [taskScheduler-3])
  • ERROR 2015-02-13 14:30:12,934 java.lang.RuntimeException: Failed to flush BatchHandler

at org.amplecode.quick.batchhandler.AbstractBatchHandler.flush(AbstractBatchHandler.java:360)

at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.saveDataValueSet(DefaultDataValueSetService.java:868)

at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.saveDataValueSetCsv(DefaultDataValueSetService.java:539)

at org.hisp.dhis.importexport.action.util.ImportDataValueTask.run(ImportDataValueTask.java:83)

at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: org.postgresql.util.PSQLException: ERROR: value too long for type character varying(31)

at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)

at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)

at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)

at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:561)

at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:405)

at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:333)

at org.amplecode.quick.batchhandler.AbstractBatchHandler.flush(AbstractBatchHandler.java:342)

… 11 more

(DefaultDataValueSetService.java [taskScheduler-3])

  • INFO 2015-02-13 14:30:12,944 [Level: ERROR, category: DATAVALUE_IMPORT, time: Fri Feb 13 14:30:12 CAT 2015, message: Process failed: Failed to flush BatchHandler] (InMemoryNotifier.java [taskScheduler-3])
  • ERROR 2015-02-13 14:30:19,402 Left side ($summary.conflicts.size()) of ‘>’ operation has null value at /dhis-web-importexport/importSummary.vm[line 35, column 33] (Log4JLogChute.java [http-apr-8080-exec-8])

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

–

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

4

Guys,

Thanks for rapid response - I obviously saw the “caused by” too, but could not figure out what was too long (stored by not specified). I had no idea that stored by had been limited to 31 characters (the username in users table is e.g. 255 characters).

You learn something every day…

:slight_smile:

Regards

Calle

···

On 13 February 2015 at 15:03, Knut Staring knutst@gmail.com wrote:

Hi Calle,

The key words to look for in the Tomcat log is “Caused by”

In this case, as Lars says, the storedby string is too long:
Caused by: org.postgresql.util.
PSQLException: ERROR: value too long for type character varying(31)

–

On Fri, Feb 13, 2015 at 1:31 PM, Calle Hedberg calle.hedberg@gmail.com wrote:

Hi,

I’m getting the following tomcat log import error when trying to import CSV (attached) - any idea what the root cause is (the tomcat messages does not make sense to me). The primary message seems to be “failed to flush batch handler”, whatever that means:

  • INFO 2015-02-13 14:30:12,895 Importing data values (DefaultDataValueSetService.java [taskScheduler-3])
  • ERROR 2015-02-13 14:30:12,934 java.lang.RuntimeException: Failed to flush BatchHandler

at org.amplecode.quick.batchhandler.AbstractBatchHandler.flush(AbstractBatchHandler.java:360)

at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.saveDataValueSet(DefaultDataValueSetService.java:868)

at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.saveDataValueSetCsv(DefaultDataValueSetService.java:539)

at org.hisp.dhis.importexport.action.util.ImportDataValueTask.run(ImportDataValueTask.java:83)

at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: org.postgresql.util.PSQLException: ERROR: value too long for type character varying(31)

at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)

at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)

at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)

at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:561)

at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:405)

at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:333)

at org.amplecode.quick.batchhandler.AbstractBatchHandler.flush(AbstractBatchHandler.java:342)

… 11 more

(DefaultDataValueSetService.java [taskScheduler-3])

  • INFO 2015-02-13 14:30:12,944 [Level: ERROR, category: DATAVALUE_IMPORT, time: Fri Feb 13 14:30:12 CAT 2015, message: Process failed: Failed to flush BatchHandler] (InMemoryNotifier.java [taskScheduler-3])
  • ERROR 2015-02-13 14:30:19,402 Left side ($summary.conflicts.size()) of ‘>’ operation has null value at /dhis-web-importexport/importSummary.vm[line 35, column 33] (Log4JLogChute.java [http-apr-8080-exec-8])

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

Mailing list: https://launchpad.net/~dhis2-devs

Post to : dhis2-devs@lists.launchpad.net

Unsubscribe : https://launchpad.net/~dhis2-devs

More help : https://help.launchpad.net/ListHelp

Knut Staring

Dept. of Informatics, University of Oslo

Norway: +4791880522

Skype: knutstar

http://dhis2.org

–

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@gmail.com

Skype: calle_hedberg

5

Yes it should be considered a bug.

Fixed now in trunk and 2.18. Trigger new build (takes 45 min).

Lars