Problems with DXF imports

jason · 29 December 2010 06:41

Hi again,
I had sent a message to the group earlier regarding importation of
data from a legacy source by DXF. Previously, I have used direct
inserts into the DHIS2 database, but because of the nature of this
data source, I want to try and use DXF. I am having problems however,
and could use some pointers.

The DXF file is here (382kB)

http://dl.dropbox.com/u/11802917/test_dxf_export8.7z

Basically, everything runs smoothly, up until the last stage when the
data values are to be imported.. When I get the error message posted
at the end of the mail.

If I import the same DXF without the <dataValues> block, it imports
just fine. I really cannot decipher the log message however to
determine what the exact problem is. Any help appreciated.

Thanks,
Jason

* INFO 08:08:50,741 Content-type: text/plain, filename: C:\Users\jason\AppData\
Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12d30795f48__8000
_00000026.tmp (ImportAction.java [2779391@qtp-1027049-15])
* INFO 08:08:50,782 Internal process started (AbstractStatementInternalProcess.
java [Thread-32])
* INFO 08:08:50,782 Importing stream (DefaultImportService.java [Thread-32])
* INFO 08:08:50,784 Importing DXF native stream (DefaultImportService.java [Thr
ead-32])
* INFO 08:08:50,791 Deleted previewed objects (DXFConverter.java [Thread-32])
* WARN 08:08:50,836 Resource Leaking: Could not remove uploaded file 'C:\Users
\jason\AppData\Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12
d30795f48__8000_00000026.tmp'. (CommonsLogger.java [2779391@qtp-1027049-15])
* INFO 08:08:50,857 Imported DataElementCategoryOptions (DXFConverter.java [Thr
ead-32])
* INFO 08:08:50,907 Imported DataElementCategories (DXFConverter.java [Thread-3
2])
* INFO 08:08:50,956 Imported DataElementCategoryCombos (DXFConverter.java [Thre
ad-32])
* INFO 08:08:51,054 Imported DataElementCategoryOptionCombos (DXFConverter.java
[Thread-32])
* INFO 08:08:51,195 Imported CategoryCategoryOption associations (DXFConverter.
java [Thread-32])
* INFO 08:08:51,308 Imported CategoryComboCategory associations (DXFConverter.j
ava [Thread-32])
* INFO 08:08:52,568 Imported DataElements (DXFConverter.java [Thread-32])
* INFO 08:22:23,426 Imported OrganisationUnits (DXFConverter.java [Thread-32])
* INFO 08:22:40,145 Imported OrganisationUnit relationships (DXFConverter.java
[Thread-32])
* INFO 08:22:40,466 Imported OrganisationUnitGroups (DXFConverter.java [Thread-
32])
* INFO 08:23:54,931 Imported OrganisationUnitGroup members (DXFConverter.java [
Thread-32])
* INFO 08:23:55,005 Imported OrganisationUnitGroupSets (DXFConverter.java [Thre
ad-32])
* INFO 08:23:55,175 Imported OrganisationUnitGroupSet members (DXFConverter.jav
a [Thread-32])
* INFO 08:23:55,255 Imported OrganisationUnitLevels (DXFConverter.java [Thread-
32])
* ERROR 08:23:55,419 Internal process failed (AbstractStatementInternalProcess.j
ava [Thread-32])
java.lang.NumberFormatException: null
        at java.lang.Integer.parseInt(Integer.java:417)
        at java.lang.Integer.parseInt(Integer.java:499)
        at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
ValueConverter.java:195)
        at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
(DefaultConverterInvoker.java:45)
        at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
on(AopUtils.java:309)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
inpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:150)
        at org.springframework.transaction.interceptor.TransactionInterceptor.in
voke(TransactionInterceptor.java:110)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
cAopProxy.java:202)
        at $Proxy49.invokeRead(Unknown Source)
        at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
er.java:968)
        at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
ortService.java:185)
        at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
portInternalProcess.java:89)
        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:91)
        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:40)
        at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
25)
        at java.lang.Thread.run(Thread.java:662)
java.lang.NumberFormatException: null
        at java.lang.Integer.parseInt(Integer.java:417)
        at java.lang.Integer.parseInt(Integer.java:499)
        at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
ValueConverter.java:195)
        at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
(DefaultConverterInvoker.java:45)
        at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
on(AopUtils.java:309)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
inpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:150)
        at org.springframework.transaction.interceptor.TransactionInterceptor.in
voke(TransactionInterceptor.java:110)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
cAopProxy.java:202)
        at $Proxy49.invokeRead(Unknown Source)
        at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
er.java:968)
        at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
ortService.java:185)
        at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
portInternalProcess.java:89)
        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:91)
        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:40)
        at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
25)
        at java.lang.Thread.run(Thread.java:662)
Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
may not be in the context. Operation not possible. /dhis-web-importexport/previ
ewForm.vm [line 92, column 157]
Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
may not be in the context. Operation not possible. /dhis-web-importexport/previ
ewForm.vm [line 93, column 161]

Current user:
    admin
Version:
    2.0.5
Build revision:
    2095
Build date:
    2010-11-25 10:47
User agent:
    Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13)
Gecko/20101203 Firefox/3.6.13
External configuration directory:
    C:\Program Files\DHIS2-2.0.5\conf
Environment variable:
    DHIS2_HOME
Database type:
    PostgreSQL
Database name:
    harvard
Database user:
    postgres
Java Opts:
Java version:
    1.6.0_22
Java vendor:
    Sun Microsystems Inc.
OS name:
    Windows 7
OS architecture:
    x86
OS version:
    6.1

···

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+260968395190

jason · 29 December 2010 11:36

Well, of course, as was the case with my previous mail, I saw my
mistake immediately after I sent the mail, which was an incorrect
attribute in the dataValue block (periodid instead of period). After
several more tedious hours of looking at XML, I have something that
works, and plan to share my experience with the group once I am done.
This conversion of legacy data is always something that needs to be
done, but up until now, it has been a process that we have not spent
so much time documenting, especially via the route from a legacy
database to DXF.

A few more impressions and comments as well.

1) Periods...DHIS2 will happily import periods which are actually not
quarters, such as with a startdate of Jan 1 1997 and end date of Jan 1
1998. It would be a good to put validation on this. Of course, the
dates should be valid to begin with, but perhaps we cannot always
trust legacy data.

I made another mistake as well, and had an end date of "1997-09-31"
which is not a valid date. I kept getting an NPE, but could not figure
out what the heck was causing it. The date itself was actually
invalid, so the enddate field ended up being transformed to NULL,
which violates the NOT NULL constraint on the enddate in periods, and
thus the NPE.

A more informative error message would have been helpful. In general,
trying to untangle the error messages that were provided during the
workup to a functional XML file resulted in a small migraine.

2) Users...When considering how to import a big block of data into an
empty database, I could not really figure out how to specify a user,
which is used in the data value attribute "storedBy"

Looking at the DXF XSD it is pretty clear...

<xs:attribute name="storedBy" type="xs:integer"/>
that the ID of the user has to be provided, but there does not appear
to be a way to specify a user actually anywhere else in the XML. I
know in DHIS 1.4, users can be transmitted from one DB to the other. I
was not really sure how to handle this, so I had DHIS2 create the DB
first, and then I had a peek in the users table to see what the
"admin" user's ID actually was. Again..a bit troublesome if you want
to do a bulk import of data from XML, without having to mess around
with the DB.

I plan on adding a bit of a case study to the documentation once I am
done, and will inform the devs once I have had time to do this.

Regards,
Jason

···

On Wed, Dec 29, 2010 at 8:41 AM, Jason Pickering <jason.p.pickering@gmail.com> wrote:

Hi again,
I had sent a message to the group earlier regarding importation of
data from a legacy source by DXF. Previously, I have used direct
inserts into the DHIS2 database, but because of the nature of this
data source, I want to try and use DXF. I am having problems however,
and could use some pointers.

The DXF file is here (382kB)

http://dl.dropbox.com/u/11802917/test_dxf_export8.7z

Basically, everything runs smoothly, up until the last stage when the
data values are to be imported.. When I get the error message posted
at the end of the mail.

If I import the same DXF without the <dataValues> block, it imports
just fine. I really cannot decipher the log message however to
determine what the exact problem is. Any help appreciated.

Thanks,
Jason

* INFO 08:08:50,741 Content-type: text/plain, filename: C:\Users\jason\AppData\
Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12d30795f48__8000
_00000026.tmp (ImportAction.java [2779391@qtp-1027049-15])
* INFO 08:08:50,782 Internal process started (AbstractStatementInternalProcess.
java [Thread-32])
* INFO 08:08:50,782 Importing stream (DefaultImportService.java [Thread-32])
* INFO 08:08:50,784 Importing DXF native stream (DefaultImportService.java [Thr
ead-32])
* INFO 08:08:50,791 Deleted previewed objects (DXFConverter.java [Thread-32])
* WARN 08:08:50,836 Resource Leaking: Could not remove uploaded file 'C:\Users
\jason\AppData\Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12
d30795f48__8000_00000026.tmp'. (CommonsLogger.java [2779391@qtp-1027049-15])
* INFO 08:08:50,857 Imported DataElementCategoryOptions (DXFConverter.java [Thr
ead-32])
* INFO 08:08:50,907 Imported DataElementCategories (DXFConverter.java [Thread-3
2])
* INFO 08:08:50,956 Imported DataElementCategoryCombos (DXFConverter.java [Thre
ad-32])
* INFO 08:08:51,054 Imported DataElementCategoryOptionCombos (DXFConverter.java
[Thread-32])
* INFO 08:08:51,195 Imported CategoryCategoryOption associations (DXFConverter.
java [Thread-32])
* INFO 08:08:51,308 Imported CategoryComboCategory associations (DXFConverter.j
ava [Thread-32])
* INFO 08:08:52,568 Imported DataElements (DXFConverter.java [Thread-32])
* INFO 08:22:23,426 Imported OrganisationUnits (DXFConverter.java [Thread-32])
* INFO 08:22:40,145 Imported OrganisationUnit relationships (DXFConverter.java
[Thread-32])
* INFO 08:22:40,466 Imported OrganisationUnitGroups (DXFConverter.java [Thread-
32])
* INFO 08:23:54,931 Imported OrganisationUnitGroup members (DXFConverter.java [
Thread-32])
* INFO 08:23:55,005 Imported OrganisationUnitGroupSets (DXFConverter.java [Thre
ad-32])
* INFO 08:23:55,175 Imported OrganisationUnitGroupSet members (DXFConverter.jav
a [Thread-32])
* INFO 08:23:55,255 Imported OrganisationUnitLevels (DXFConverter.java [Thread-
32])
* ERROR 08:23:55,419 Internal process failed (AbstractStatementInternalProcess.j
ava [Thread-32])
java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:417)
at java.lang.Integer.parseInt(Integer.java:499)
at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
ValueConverter.java:195)
at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
(DefaultConverterInvoker.java:45)
at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
on(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
inpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.in
voke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
cAopProxy.java:202)
at $Proxy49.invokeRead(Unknown Source)
at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
er.java:968)
at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
ortService.java:185)
at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
portInternalProcess.java:89)
at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:91)
at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:40)
at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
25)
at java.lang.Thread.run(Thread.java:662)
java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:417)
at java.lang.Integer.parseInt(Integer.java:499)
at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
ValueConverter.java:195)
at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
(DefaultConverterInvoker.java:45)
at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
on(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
inpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.in
voke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
cAopProxy.java:202)
at $Proxy49.invokeRead(Unknown Source)
at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
er.java:968)
at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
ortService.java:185)
at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
portInternalProcess.java:89)
at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:91)
at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:40)
at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
25)
at java.lang.Thread.run(Thread.java:662)
Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
may not be in the context. Operation not possible. /dhis-web-importexport/previ
ewForm.vm [line 92, column 157]
Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
may not be in the context. Operation not possible. /dhis-web-importexport/previ
ewForm.vm [line 93, column 161]

Current user:
admin
Version:
2.0.5
Build revision:
2095
Build date:
2010-11-25 10:47
User agent:
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13)
Gecko/20101203 Firefox/3.6.13
External configuration directory:
C:\Program Files\DHIS2-2.0.5\conf
Environment variable:
DHIS2_HOME
Database type:
PostgreSQL
Database name:
harvard
Database user:
postgres
Java Opts:
Java version:
1.6.0_22
Java vendor:
Sun Microsystems Inc.
OS name:
Windows 7
OS architecture:
x86
OS version:
6.1
--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+260968395190

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+260968395190

bobj · 29 December 2010 14:20

Hi Jason

Well, of course, as was the case with my previous mail, I saw my
mistake immediately after I sent the mail, which was an incorrect
attribute in the dataValue block (periodid instead of period).

I was going to say after reading your previous mail that an invalid
attribute name tends to result in this error about number format.

After
several more tedious hours of looking at XML, I have something that
works, and plan to share my experience with the group once I am done.

I also saw your blueprint about validation ... I think many of the
common errors might be best caught with schematron rules rather than
traditional relaxng ot xml schema. Having also mispent many glorious
hours going through the sort of process you have been going through I
agree that some validations would help.

This conversion of legacy data is always something that needs to be
done, but up until now, it has been a process that we have not spent
so much time documenting, especially via the route from a legacy
database to DXF.

A few more impressions and comments as well.

1) Periods...DHIS2 will happily import periods which are actually not
quarters, such as with a startdate of Jan 1 1997 and end date of Jan 1
1998. It would be a good to put validation on this. Of course, the
dates should be valid to begin with, but perhaps we cannot always
trust legacy data.

I have never been too happy about monthly, weekly, quarterly, annual
etc periods being represented by start and end date. Of course for
uniformity of storage they could and perhaps should be persisted like
that, but in the api and dxf it might be better to construct them from
1997-01, 1197-W1, 1997-Q1, 1997 etc. Specifying start-end dates for
months and the like is really looking for validation errors - which
means of course we should test them Period validation blueprint
required.

I made another mistake as well, and had an end date of "1997-09-31"
which is not a valid date. I kept getting an NPE, but could not figure
out what the heck was causing it. The date itself was actually
invalid, so the enddate field ended up being transformed to NULL,
which violates the NOT NULL constraint on the enddate in periods, and
thus the NPE.

A more informative error message would have been helpful. In general,
trying to untangle the error messages that were provided during the
workup to a functional XML file resulted in a small migraine.

Some schematron rules would help here.

2) Users...When considering how to import a big block of data into an
empty database, I could not really figure out how to specify a user,
which is used in the data value attribute "storedBy"

<dataValue dataElement="2" period="37" source="6795" value="17"
storedBy="14" timeStamp="" comment="PATH Harvard Dataset"
categoryOptionCombo="11"></dataValue>

I am not sure we should allow storedBy as an attribute at all. There
isn't obvious use case to import/export users. At least not in the
common data exchange scenarios. I think the most sensible approach
would be to remove the attribute (or at least make it optional, which
I think it actually is but the schema may not reflect this) and set
storedBy to the user performing the import.

Looking at the DXF XSD it is pretty clear...

<xs:attribute name="storedBy" type="xs:integer"/>
that the ID of the user has to be provided, but there does not appear
to be a way to specify a user actually anywhere else in the XML. I
know in DHIS 1.4, users can be transmitted from one DB to the other. I
was not really sure how to handle this, so I had DHIS2 create the DB
first, and then I had a peek in the users table to see what the
"admin" user's ID actually was. Again..a bit troublesome if you want
to do a bulk import of data from XML, without having to mess around
with the DB.

The dxf 1 schema needs a bit of tidying up to reflect some of the
minor changes over the past few months. I've not given it the
attention it deserves because I am still hoping to sift all these
learnings into a new look dxf 2 one of these fine days. Perhaps 2011
will be it ... I know Jo has also been rumbling in this regard.

Cheers
Bob

···

On 29 December 2010 11:36, Jason Pickering <jason.p.pickering@gmail.com> wrote:

I plan on adding a bit of a case study to the documentation once I am
done, and will inform the devs once I have had time to do this.

Regards,
Jason

On Wed, Dec 29, 2010 at 8:41 AM, Jason Pickering > <jason.p.pickering@gmail.com> wrote:

Hi again,
I had sent a message to the group earlier regarding importation of
data from a legacy source by DXF. Previously, I have used direct
inserts into the DHIS2 database, but because of the nature of this
data source, I want to try and use DXF. I am having problems however,
and could use some pointers.

The DXF file is here (382kB)

http://dl.dropbox.com/u/11802917/test_dxf_export8.7z

Basically, everything runs smoothly, up until the last stage when the
data values are to be imported.. When I get the error message posted
at the end of the mail.

If I import the same DXF without the <dataValues> block, it imports
just fine. I really cannot decipher the log message however to
determine what the exact problem is. Any help appreciated.

Thanks,
Jason

* INFO 08:08:50,741 Content-type: text/plain, filename: C:\Users\jason\AppData\
Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12d30795f48__8000
_00000026.tmp (ImportAction.java [2779391@qtp-1027049-15])
* INFO 08:08:50,782 Internal process started (AbstractStatementInternalProcess.
java [Thread-32])
* INFO 08:08:50,782 Importing stream (DefaultImportService.java [Thread-32])
* INFO 08:08:50,784 Importing DXF native stream (DefaultImportService.java [Thr
ead-32])
* INFO 08:08:50,791 Deleted previewed objects (DXFConverter.java [Thread-32])
* WARN 08:08:50,836 Resource Leaking: Could not remove uploaded file 'C:\Users
\jason\AppData\Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12
d30795f48__8000_00000026.tmp'. (CommonsLogger.java [2779391@qtp-1027049-15])
* INFO 08:08:50,857 Imported DataElementCategoryOptions (DXFConverter.java [Thr
ead-32])
* INFO 08:08:50,907 Imported DataElementCategories (DXFConverter.java [Thread-3
2])
* INFO 08:08:50,956 Imported DataElementCategoryCombos (DXFConverter.java [Thre
ad-32])
* INFO 08:08:51,054 Imported DataElementCategoryOptionCombos (DXFConverter.java
[Thread-32])
* INFO 08:08:51,195 Imported CategoryCategoryOption associations (DXFConverter.
java [Thread-32])
* INFO 08:08:51,308 Imported CategoryComboCategory associations (DXFConverter.j
ava [Thread-32])
* INFO 08:08:52,568 Imported DataElements (DXFConverter.java [Thread-32])
* INFO 08:22:23,426 Imported OrganisationUnits (DXFConverter.java [Thread-32])
* INFO 08:22:40,145 Imported OrganisationUnit relationships (DXFConverter.java
[Thread-32])
* INFO 08:22:40,466 Imported OrganisationUnitGroups (DXFConverter.java [Thread-
32])
* INFO 08:23:54,931 Imported OrganisationUnitGroup members (DXFConverter.java [
Thread-32])
* INFO 08:23:55,005 Imported OrganisationUnitGroupSets (DXFConverter.java [Thre
ad-32])
* INFO 08:23:55,175 Imported OrganisationUnitGroupSet members (DXFConverter.jav
a [Thread-32])
* INFO 08:23:55,255 Imported OrganisationUnitLevels (DXFConverter.java [Thread-
32])
* ERROR 08:23:55,419 Internal process failed (AbstractStatementInternalProcess.j
ava [Thread-32])
java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:417)
at java.lang.Integer.parseInt(Integer.java:499)
at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
ValueConverter.java:195)
at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
(DefaultConverterInvoker.java:45)
at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
on(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
inpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.in
voke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
cAopProxy.java:202)
at $Proxy49.invokeRead(Unknown Source)
at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
er.java:968)
at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
ortService.java:185)
at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
portInternalProcess.java:89)
at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:91)
at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:40)
at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
25)
at java.lang.Thread.run(Thread.java:662)
java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:417)
at java.lang.Integer.parseInt(Integer.java:499)
at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
ValueConverter.java:195)
at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
(DefaultConverterInvoker.java:45)
at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
on(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
inpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.in
voke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
cAopProxy.java:202)
at $Proxy49.invokeRead(Unknown Source)
at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
er.java:968)
at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
ortService.java:185)
at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
portInternalProcess.java:89)
at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:91)
at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:40)
at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
25)
at java.lang.Thread.run(Thread.java:662)
Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
may not be in the context. Operation not possible. /dhis-web-importexport/previ
ewForm.vm [line 92, column 157]
Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
may not be in the context. Operation not possible. /dhis-web-importexport/previ
ewForm.vm [line 93, column 161]

Current user:
admin
Version:
2.0.5
Build revision:
2095
Build date:
2010-11-25 10:47
User agent:
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13)
Gecko/20101203 Firefox/3.6.13
External configuration directory:
C:\Program Files\DHIS2-2.0.5\conf
Environment variable:
DHIS2_HOME
Database type:
PostgreSQL
Database name:
harvard
Database user:
postgres
Java Opts:
Java version:
1.6.0_22
Java vendor:
Sun Microsystems Inc.
OS name:
Windows 7
OS architecture:
x86
OS version:
6.1
--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+260968395190

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+260968395190

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

jason · 29 December 2010 15:49

Hi Bob,
Completely agree that the schematron rules would be a good idea. This
is also somewhat related to my blueprint on attribute level validation
(https://blueprints.launchpad.net/dhis2/+spec/regex-validation\) but
not exactly. I must confess I have not done much with schematrons,
and do not really understand their full potential. However, there is
clearly a need for rules both within an attribute itself, as well as
the relationships between them. If a schematron could do this, or part
of it, it would certainly be helpful. The biggest problem I see at the
moment for people aspiring to import data via DXF or other formats is
the extraordinarily arcane error messages that leaves one either
peeking at the source code to attempt to decipher what the problem
might be or desiring to open a beer instead. If these could somehow be
improved, it sure would make this process better. In general, the
entire import process seems rather fragile (my experience with DHIS
1.4 imports as well). If anything is wrong, the whole process bombs
with a bizarre message (which when using DHIS Live, is not even clear
where the log actually is???)

Just a short rebuttal on the importation of users. I actually would
consider the import of users to be quite important. You are correct of
course (as seen in the database) that the storedBy attribute is not
required. However, it is useful to see who has done what, and who
imported the data. It is especially useful in the context of pulling
data in from remote sources, where the contact details of the person
who has transmitted the file is present, so that if follow up is
necessary, it is easily apparent who to contact. I would argue for
more info in this regard, a la DHIS 1.4. Of course, there are security
issues here that need to be sorted out in the context of DHIS2. Just
because a user would get imported into the DB, does not mean they
would have the right to do anything. Sounds like another blueprint,
but I will not bother unless there is a need beyond my own wish list.

Regards,
Jason

···

On Wed, Dec 29, 2010 at 4:20 PM, Bob Jolliffe <bobjolliffe@gmail.com> wrote:

Hi Jason

On 29 December 2010 11:36, Jason Pickering <jason.p.pickering@gmail.com> wrote:

Well, of course, as was the case with my previous mail, I saw my
mistake immediately after I sent the mail, which was an incorrect
attribute in the dataValue block (periodid instead of period).

I was going to say after reading your previous mail that an invalid
attribute name tends to result in this error about number format.

After
several more tedious hours of looking at XML, I have something that
works, and plan to share my experience with the group once I am done.

I also saw your blueprint about validation ... I think many of the
common errors might be best caught with schematron rules rather than
traditional relaxng ot xml schema. Having also mispent many glorious
hours going through the sort of process you have been going through I
agree that some validations would help.

This conversion of legacy data is always something that needs to be
done, but up until now, it has been a process that we have not spent
so much time documenting, especially via the route from a legacy
database to DXF.

A few more impressions and comments as well.

1) Periods...DHIS2 will happily import periods which are actually not
quarters, such as with a startdate of Jan 1 1997 and end date of Jan 1
1998. It would be a good to put validation on this. Of course, the
dates should be valid to begin with, but perhaps we cannot always
trust legacy data.

I have never been too happy about monthly, weekly, quarterly, annual
etc periods being represented by start and end date. Of course for
uniformity of storage they could and perhaps should be persisted like
that, but in the api and dxf it might be better to construct them from
1997-01, 1197-W1, 1997-Q1, 1997 etc. Specifying start-end dates for
months and the like is really looking for validation errors - which
means of course we should test them Period validation blueprint
required.

I made another mistake as well, and had an end date of "1997-09-31"
which is not a valid date. I kept getting an NPE, but could not figure
out what the heck was causing it. The date itself was actually
invalid, so the enddate field ended up being transformed to NULL,
which violates the NOT NULL constraint on the enddate in periods, and
thus the NPE.

A more informative error message would have been helpful. In general,
trying to untangle the error messages that were provided during the
workup to a functional XML file resulted in a small migraine.

Some schematron rules would help here.

2) Users...When considering how to import a big block of data into an
empty database, I could not really figure out how to specify a user,
which is used in the data value attribute "storedBy"

<dataValue dataElement="2" period="37" source="6795" value="17"
storedBy="14" timeStamp="" comment="PATH Harvard Dataset"
categoryOptionCombo="11"></dataValue>

I am not sure we should allow storedBy as an attribute at all. There
isn't obvious use case to import/export users. At least not in the
common data exchange scenarios. I think the most sensible approach
would be to remove the attribute (or at least make it optional, which
I think it actually is but the schema may not reflect this) and set
storedBy to the user performing the import.

Looking at the DXF XSD it is pretty clear...

<xs:attribute name="storedBy" type="xs:integer"/>
that the ID of the user has to be provided, but there does not appear
to be a way to specify a user actually anywhere else in the XML. I
know in DHIS 1.4, users can be transmitted from one DB to the other. I
was not really sure how to handle this, so I had DHIS2 create the DB
first, and then I had a peek in the users table to see what the
"admin" user's ID actually was. Again..a bit troublesome if you want
to do a bulk import of data from XML, without having to mess around
with the DB.

The dxf 1 schema needs a bit of tidying up to reflect some of the
minor changes over the past few months. I've not given it the
attention it deserves because I am still hoping to sift all these
learnings into a new look dxf 2 one of these fine days. Perhaps 2011
will be it ... I know Jo has also been rumbling in this regard.

Cheers
Bob

I plan on adding a bit of a case study to the documentation once I am
done, and will inform the devs once I have had time to do this.

Regards,
Jason

On Wed, Dec 29, 2010 at 8:41 AM, Jason Pickering >> <jason.p.pickering@gmail.com> wrote:

Hi again,
I had sent a message to the group earlier regarding importation of
data from a legacy source by DXF. Previously, I have used direct
inserts into the DHIS2 database, but because of the nature of this
data source, I want to try and use DXF. I am having problems however,
and could use some pointers.

The DXF file is here (382kB)

http://dl.dropbox.com/u/11802917/test_dxf_export8.7z

Basically, everything runs smoothly, up until the last stage when the
data values are to be imported.. When I get the error message posted
at the end of the mail.

If I import the same DXF without the <dataValues> block, it imports
just fine. I really cannot decipher the log message however to
determine what the exact problem is. Any help appreciated.

Thanks,
Jason

* INFO 08:08:50,741 Content-type: text/plain, filename: C:\Users\jason\AppData\
Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12d30795f48__8000
_00000026.tmp (ImportAction.java [2779391@qtp-1027049-15])
* INFO 08:08:50,782 Internal process started (AbstractStatementInternalProcess.
java [Thread-32])
* INFO 08:08:50,782 Importing stream (DefaultImportService.java [Thread-32])
* INFO 08:08:50,784 Importing DXF native stream (DefaultImportService.java [Thr
ead-32])
* INFO 08:08:50,791 Deleted previewed objects (DXFConverter.java [Thread-32])
* WARN 08:08:50,836 Resource Leaking: Could not remove uploaded file 'C:\Users
\jason\AppData\Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12
d30795f48__8000_00000026.tmp'. (CommonsLogger.java [2779391@qtp-1027049-15])
* INFO 08:08:50,857 Imported DataElementCategoryOptions (DXFConverter.java [Thr
ead-32])
* INFO 08:08:50,907 Imported DataElementCategories (DXFConverter.java [Thread-3
2])
* INFO 08:08:50,956 Imported DataElementCategoryCombos (DXFConverter.java [Thre
ad-32])
* INFO 08:08:51,054 Imported DataElementCategoryOptionCombos (DXFConverter.java
[Thread-32])
* INFO 08:08:51,195 Imported CategoryCategoryOption associations (DXFConverter.
java [Thread-32])
* INFO 08:08:51,308 Imported CategoryComboCategory associations (DXFConverter.j
ava [Thread-32])
* INFO 08:08:52,568 Imported DataElements (DXFConverter.java [Thread-32])
* INFO 08:22:23,426 Imported OrganisationUnits (DXFConverter.java [Thread-32])
* INFO 08:22:40,145 Imported OrganisationUnit relationships (DXFConverter.java
[Thread-32])
* INFO 08:22:40,466 Imported OrganisationUnitGroups (DXFConverter.java [Thread-
32])
* INFO 08:23:54,931 Imported OrganisationUnitGroup members (DXFConverter.java [
Thread-32])
* INFO 08:23:55,005 Imported OrganisationUnitGroupSets (DXFConverter.java [Thre
ad-32])
* INFO 08:23:55,175 Imported OrganisationUnitGroupSet members (DXFConverter.jav
a [Thread-32])
* INFO 08:23:55,255 Imported OrganisationUnitLevels (DXFConverter.java [Thread-
32])
* ERROR 08:23:55,419 Internal process failed (AbstractStatementInternalProcess.j
ava [Thread-32])
java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:417)
at java.lang.Integer.parseInt(Integer.java:499)
at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
ValueConverter.java:195)
at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
(DefaultConverterInvoker.java:45)
at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
on(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
inpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.in
voke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
cAopProxy.java:202)
at $Proxy49.invokeRead(Unknown Source)
at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
er.java:968)
at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
ortService.java:185)
at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
portInternalProcess.java:89)
at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:91)
at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:40)
at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
25)
at java.lang.Thread.run(Thread.java:662)
java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:417)
at java.lang.Integer.parseInt(Integer.java:499)
at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
ValueConverter.java:195)
at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
(DefaultConverterInvoker.java:45)
at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
on(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
inpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.in
voke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
cAopProxy.java:202)
at $Proxy49.invokeRead(Unknown Source)
at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
er.java:968)
at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
ortService.java:185)
at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
portInternalProcess.java:89)
at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:91)
at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
(AbstractStatementInternalProcess.java:40)
at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
25)
at java.lang.Thread.run(Thread.java:662)
Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
may not be in the context. Operation not possible. /dhis-web-importexport/previ
ewForm.vm [line 92, column 157]
Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
may not be in the context. Operation not possible. /dhis-web-importexport/previ
ewForm.vm [line 93, column 161]

Current user:
admin
Version:
2.0.5
Build revision:
2095
Build date:
2010-11-25 10:47
User agent:
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13)
Gecko/20101203 Firefox/3.6.13
External configuration directory:
C:\Program Files\DHIS2-2.0.5\conf
Environment variable:
DHIS2_HOME
Database type:
PostgreSQL
Database name:
harvard
Database user:
postgres
Java Opts:
Java version:
1.6.0_22
Java vendor:
Sun Microsystems Inc.
OS name:
Windows 7
OS architecture:
x86
OS version:
6.1
--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+260968395190

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+260968395190

_______________________________________________
Mailing list: DHIS 2 developers in Launchpad
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : DHIS 2 developers in Launchpad
More help : ListHelp - Launchpad Help

--
Jason P. Pickering
email: jason.p.pickering@gmail.com
tel:+260968395190