[Dhis-dev] Export process from 1.4 to 2.0

Hi Lars,
I have made the required changes to DHIS, and now it reports using the
Postgreql backend.

About DHIS 2

Current user:
    admin
Version:
    2.0.1-SNAPSHOT
Build date:
    2009-03-03 17:07
User agent:
    Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6)
Gecko/2009011913 Firefox/3.0.6
External configuration directory:
    c:\dhis2
Environment variable:
    DHIS2_HOME
Database type:
    PostgreSQL
Database name:
    dhis2_zm
Database user:
    dhis

I have tried method (1) on a clean database and obtain the following
errors (from the Tomcat log)

* INFO 14:04:16,440 Environment variable DHIS2_HOME points to
c:\dhis2 (DefaultLocationManager.java [Thread-1])
* INFO 14:04:18,643 Hibernate properties file found:
jar:file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%206.0/webapps/dhis/WEB-INF/lib/dhis-support-hibernate-2.0.1-SNAPSHOT.jar!/hibernate-default.properties
(DefaultHibernateConfigurationProvider.java [Thread-1])
* INFO 14:04:18,643 Hibernate properties file found:
jar:file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%206.0/webapps/dhis/WEB-INF/lib/dhis-web-maintenance-dataadmin.jar!/hibernate.properties
(DefaultHibernateConfigurationProvider.java [Thread-1])
* INFO 14:04:18,658 Hibernate properties file found:
c:\dhis2\hibernate.properties
(DefaultHibernateConfigurationProvider.java [Thread-1])
* INFO 14:04:42,969 Executing startup routine [1 of 11, runlevel 0]:
DataElementDefaultDimensionPopulator
(DefaultStartupRoutineExecutor.java [Thread-1])
* INFO 14:04:59,093 Added default dataelement dimension
(DataElementDefaultDimensionPopulator.java [Thread-1])
* INFO 14:04:59,125 Executing startup routine [2 of 11, runlevel 0]:
DataSetShortNamePopulator (DefaultStartupRoutineExecutor.java
[Thread-1])
* INFO 14:04:59,140 Populated DataSet short name and code
(DataSetShortNamePopulator.java [Thread-1])
* INFO 14:04:59,140 Executing startup routine [3 of 11, runlevel 0]:
CategoryOptionShortNamePopulator (DefaultStartupRoutineExecutor.java
[Thread-1])
* INFO 14:04:59,218 Populated CategoryOption shortname
(CategoryOptionShortNamePopulator.java [Thread-1])
* INFO 14:04:59,218 Executing startup routine [4 of 11, runlevel 0]:
OrganisationUnitHierarchyVerifier (DefaultStartupRoutineExecutor.java
[Thread-1])
* WARN 14:04:59,546 firstResult/maxResults specified with collection
fetch; applying in memory! (QueryTranslatorImpl.java [Thread-1])
* INFO 14:04:59,578 Added organistion unit hierarchy
(OrganisationUnitHierarchyVerifier.java [Thread-1])
* INFO 14:04:59,578 Executing startup routine [5 of 11, runlevel 0]:
OpenHealthDataSourceWriter (DefaultStartupRoutineExecutor.java
[Thread-1])
* INFO 14:04:59,578 Environment variable OPENHEALTH_HOME not set
(OpenHealthDataSourceWriter.java [Thread-1])
* INFO 14:04:59,578 Executing startup routine [6 of 11, runlevel 0]:
ReportTableUpgrader (DefaultStartupRoutineExecutor.java [Thread-1])
* INFO 14:04:59,593 Upgraded report tables (ReportTableUpgrader.java
[Thread-1])
* INFO 14:04:59,593 Executing startup routine [7 of 11, runlevel 1]:
PeriodTypePopulator (DefaultStartupRoutineExecutor.java [Thread-1])
* INFO 14:04:59,625 Executing startup routine [8 of 11, runlevel 2]:
MultiDimensionExpressionUpgrader (DefaultStartupRoutineExecutor.java
[Thread-1])
* INFO 14:04:59,656 Executing startup routine [9 of 11, runlevel 3]:
AggregationTableCreator (DefaultStartupRoutineExecutor.java
[Thread-1])
* INFO 14:04:59,750 Created table aggregateddatavalue
(AggregationTableCreator.java [Thread-1])
* INFO 14:04:59,750 Created table aggregatedindicatorvalue
(AggregationTableCreator.java [Thread-1])
* INFO 14:04:59,828 Created index crosstab on table datavalue
(AggregationTableCreator.java [Thread-1])
* INFO 14:04:59,843 Created table aggregateddatasetcompleteness
(AggregationTableCreator.java [Thread-1])
* INFO 14:04:59,843 Executing startup routine [10 of 11, runlevel 3]:
DataElementCategoryOptionComboStartupRoutine
(DefaultStartupRoutineExecutor.java [Thread-1])
* INFO 14:04:59,906 Verified CategoryOptionCombos
(DataElementCategoryOptionComboStartupRoutine.java [Thread-1])
* INFO 14:04:59,906 Executing startup routine [11 of 11, runlevel
10]: I18nUpgrader (DefaultStartupRoutineExecutor.java [Thread-1])
* INFO 14:05:00,125 I18n Upgrader running using locale engelska
(Storbritannien) (I18nUpgrader.java [Thread-1])
* INFO 14:05:00,187 I81n Upgrader: DataElementCategory (1)
(I18nUpgrader.java [Thread-1])
* INFO 14:05:00,203 I81n Upgrader: DataElementCategoryOption (1)
(I18nUpgrader.java [Thread-1])
* INFO 14:05:00,218 I81n Upgrader: DataElementCategoryCombo (1)
(I18nUpgrader.java [Thread-1])
* INFO 14:05:00,218 All startup routines done
(DefaultStartupRoutineExecutor.java [Thread-1])
* INFO 14:05:02,765 Reading datasource config file from
datasources.xml,
file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%206.0/webapps/openhealth-fp/WEB-INF/classes/datasources.xml
(MondrianOlapQueryProcessor.java [Thread-1])
* INFO 14:05:02,999 xmla output: '<Discover
xmlns="urn:schemas-microsoft-com:xml-analysis">
<RequestType>MDSCHEMA_DIMENSIONS</RequestType>
<Restrictions>
  <RestrictionList>
    <CATALOG_NAME>OpenHealth</CATALOG_NAME>
    <CUBE_NAME>Diseases</CUBE_NAME>
      </RestrictionList>
</Restrictions>
<Properties>
  <PropertyList>
    <DataSourceInfo>OpenHealthData</DataSourceInfo>
    <Catalog>OpenHealth</Catalog>
    <Format>Tabular</Format>
    <Content>SchemaData</Content>
  </PropertyList>
</Properties>
</Discover>
' (MondrianOlapQueryProcessor.java [Thread-1])
17 mar 14:05:12 ERROR [geoserver.global] -
<omitting Geoserver startup messages>

* INFO 14:06:48,946 Imported DataElements
(DefaultDhis14FileImportService.java [Thread-16])
* WARN 14:06:52,837 Value is null for key: '908' (LoggingHashMap.java
[Thread-16])
* ERROR 14:06:52,837 The process threw exception (ProcessExecutor.java
[Thread-16])
java.lang.RuntimeException: Query with RowHandler failed
  at org.hisp.dhis.importexport.dhis14.file.query.IbatisQueryManager.queryWithRowhandler(IbatisQueryManager.java:141)
  at org.hisp.dhis.importexport.dhis14.file.query.IbatisQueryManager.queryWithRowhandler(IbatisQueryManager.java:124)
  at org.hisp.dhis.importexport.dhis14.file.importer.DefaultDhis14FileImportService.importCalculatedDataElements(DefaultDhis14FileImportService.java:346)
  at org.hisp.dhis.importexport.dhis14.file.importer.DefaultDhis14FileImportService.importData(DefaultDhis14FileImportService.java:257)
  at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(ImportInternalProcess.java:84)
  at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute(AbstractStatementInternalProcess.java:85)
  at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute(AbstractStatementInternalProcess.java:37)
  at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:126)
  at java.lang.Thread.run(Unknown Source)
Caused by: com.ibatis.common.jdbc.exception.NestedSQLException:
--- The error occurred in sqlmap/calculatedDataElement.sqlmap.xml.
--- The error occurred while applying a result map.
--- Check the calculatedDataElement.calculatedDataElementResultMap.
--- The error happened while setting a property on the result object.
--- Cause: java.lang.NullPointerException
Caused by: java.lang.NullPointerException
  at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:188)
  at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler(GeneralStatement.java:133)
  at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(SqlMapExecutorDelegate.java:644)
  at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMapSessionImpl.java:121)
  at com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryWithRowHandler(SqlMapClientImpl.java:98)
  at org.hisp.dhis.importexport.dhis14.file.query.IbatisQueryManager.queryWithRowhandler(IbatisQueryManager.java:135)
  ... 8 more
Caused by: java.lang.NullPointerException
  at org.hisp.dhis.expression.DefaultExpressionService.convertExpression(DefaultExpressionService.java:187)
  at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.lang.reflect.Method.invoke(Unknown Source)
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
  at $Proxy35.convertExpression(Unknown Source)
  at org.hisp.dhis.importexport.dhis14.file.rowhandler.CalculatedDataElementRowHandler.handleRow(CalculatedDataElementRowHandler.java:115)
  at com.ibatis.sqlmap.engine.mapping.statement.RowHandlerCallback.handleResultObject(RowHandlerCallback.java:76)
  at com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:395)
  at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:185)
  at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
  at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
  ... 13 more

I would assume this would be caused by the fact I am using a
completely new database.

So essentially, I would like to perform a completely clean
installation from 1.4 to 2.0. We will need to do this on a regular
basis, as 2.0 (at least for the time being in Zambia) will only be
used for reporting and analysis. No data will be entered or modified.
Each quarter, a new import from the master 1.4 database will need to
take place.

Thanks in advance for your help.

I will attempt the second procedure now,and see what happens .

Best regards,
Jason

···

On Wed, Mar 11, 2009 at 12:40 PM, Lars Helge Øverland <larshelge@gmail.com> wrote:

Here are the details of the system..

Current user:
admin
Version:
2.0.1-SNAPSHOT
Build date:
2009-03-03 17:07
User agent:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6)
Gecko/2009011913 Firefox/3.0.6
External configuration directory:
Not set
Environment variable:
DHIS2_HOME
Database type:
H2
Database name:
dhis2
Database user:
sa

I do not have a real database setup yet. I am thinking that this may
have to do with the lack of an external configuration directory.

Let me try and set it, but I assume that there should be a more
graceful error returned. :slight_smile:

Regards,
Jason

Hi Jason,

you need to install DHIS 2 properly:

- Set the DHIS2_HOME environment variable to any folder. In this folder the
system will create a subfolder called "importexport" where the dhis14 import
config file will be stored.

- Also set up a database, your system is now running on the default H2
in-memory database. Install postgresql/mysql, put the attached
hibernate.properties in the DHIS2_HOME directory and modify it to your db
setup.

You could also set the "import type" to analysis to reveal problems in the
DHIS 1.4 datafile related to unique field/foreign key contraints (access is
more lenient that postgres/mysql).

Then try again, let me know if it works.

Lars