DHIS2 Upgrade from 2.28 to 2.31 fails with Flyway migration error

Gaurav_Kumar · 18 July 2019 09:02

I have executed the 2.29 & 2.29 upgrade scripts on our 2.28 DB but still getting migration error on trying to initiate the “2.31.5-SNAPSHOT” war.

ERROR LOG SNIPPET:

* ERROR 2019-07-18 11:49:59,019 Migration of schema "public" to version 2.31.1 - Migrations for release v31 failed! Changes successfully rolled back. (Slf4jLog.java [main])
* WARN  2019-07-18 11:49:59,022 Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.hisp.dhis.setting.SystemSettingStore' defined in URL [jar:file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-service-setting-2.31.5-SNAPSHOT.jar!/META-INF/dhis/beans.xml]: Cannot resolve reference to bean 'sessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flyway' defined in URL [jar:file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-support-db-migration-2.31.5-SNAPSHOT.jar!/META-INF/dhis/beans.xml]: Invocation of init method failed; nested exception is org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException: 
Migration V2_31_1__Migrations_for_release_v31.sql failed
--------------------------------------------------------
SQL State  : 42P01
Error Code : 0
Message    : ERROR: relation "predictorgrouptranslations" does not exist
  Position: 183
Location   : org/hisp/dhis/db/migration/2.31/V2_31_1__Migrations_for_release_v31.sql (/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-support-db-migration-2.31.5-SNAPSHOT.jar!/org/hisp/dhis/db/migration/2.31/V2_31_1__Migrations_for_release_v31.sql)
Line       : 320
Statement  : update predictorgroup o set translations = (select to_jsonb(array_agg(jsonb_build_object('locale',locale,'property', property,'value', value ))) from objecttranslation ot inner join predictorgrouptranslations t on ot.objecttranslationid = t.objecttranslationid  where t.predictorgroupid = o.predictorgroupid )
 (AbstractApplicationContext.java [main])
* ERROR 2019-07-18 11:49:59,027 Context initialization failed (ContextLoader.java [main])
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.hisp.dhis.setting.SystemSettingStore' defined in URL [jar:file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-service-setting-2.31.5-SNAPSHOT.jar!/META-INF/dhis/beans.xml]: Cannot resolve reference to bean 'sessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flyway' defined in URL [jar:file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-support-db-migration-2.31.5-SNAPSHOT.jar!/META-INF/dhis/beans.xml]: Invocation of init method failed; nested exception is org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException: 
Migration V2_31_1__Migrations_for_release_v31.sql failed
--------------------------------------------------------
SQL State  : 42P01
Error Code : 0
Message    : ERROR: relation "predictorgrouptranslations" does not exist
  Position: 183
Location   : org/hisp/dhis/db/migration/2.31/V2_31_1__Migrations_for_release_v31.sql (/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-support-db-migration-2.31.5-SNAPSHOT.jar!/org/hisp/dhis/db/migration/2.31/V2_31_1__Migrations_for_release_v31.sql)
Line       : 320
Statement  : update predictorgroup o set translations = (select to_jsonb(array_agg(jsonb_build_object('locale',locale,'property', property,'value', value ))) from objecttranslation ot inner join predictorgrouptranslations t on ot.objecttranslationid = t.objecttranslationid  where t.predictorgroupid = o.predictorgroupid )

	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1534)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1281)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:551)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:843)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
	at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:816)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404)
	at org.eclipse.jetty.maven.plugin.JettyWebAppContext.startWebapp(JettyWebAppContext.java:323)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520)
	at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart(JettyWebAppContext.java:398)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
	at org.eclipse.jetty.server.Server.start(Server.java:411)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.Server.doStart(Server.java:378)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty(AbstractJettyMojo.java:460)
	at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute(AbstractJettyMojo.java:328)
	at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute(JettyRunWarMojo.java:64)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:956)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:192)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flyway' defined in URL [jar:file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-support-db-migration-2.31.5-SNAPSHOT.jar!/META-INF/dhis/beans.xml]: Invocation of init method failed; nested exception is org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException: 
Migration V2_31_1__Migrations_for_release_v31.sql failed
--------------------------------------------------------
SQL State  : 42P01
Error Code : 0
Message    : ERROR: relation "predictorgrouptranslations" does not exist
  Position: 183
Location   : org/hisp/dhis/db/migration/2.31/V2_31_1__Migrations_for_release_v31.sql (/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-support-db-migration-2.31.5-SNAPSHOT.jar!/org/hisp/dhis/db/migration/2.31/V2_31_1__Migrations_for_release_v31.sql)
Line       : 320
Statement  : update predictorgroup o set translations = (select to_jsonb(array_agg(jsonb_build_object('locale',locale,'property', property,'value', value ))) from objecttranslation ot inner join predictorgrouptranslations t on ot.objecttranslationid = t.objecttranslationid  where t.predictorgroupid = o.predictorgroupid )

	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1631)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:297)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
	... 67 more
Caused by: org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException: 
Migration V2_31_1__Migrations_for_release_v31.sql failed

Ameen · 18 July 2019 09:45

@Gaurav_Kumar Did you upgrade to 2.29 and 2.30 correctly before trying the 2.31.5 war?
It looks like one of the tables which should have been present on a typical 2.30 db is missing in your db. That is why the upgrade is failing. Let us know if you are sure that your db is in a compatible 2.30db state right now, and we can try some options. One way to confirm whether your db is 2.30 compatible is to try and deploy a 2.30 war onto it. If it starts up successfully, we are good.

Gaurav_Kumar · 18 July 2019 10:37

@Ameen thanks for the response, I will try that. I was mistakingly assuming that the upgrade scripts should take care of all the required updates to the DB.

Gaurav_Kumar · 22 July 2019 08:21

Hi @Ameen , now I am getting another error deploying the latest the 2.32.2-Snapshot portal. Can you please advise.

=============================================================================

WARN 2019-07-22 11:12:55,116 Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘org.hisp.dhis.schema.MergeService’ defined in URL [jar:file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-service-schema-2.32.2-SNAPSHOT.jar!/META-INF/dhis/beans.xml]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘org.hisp.dhis.schema.SchemaService’ defined in URL [jar:file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-service-schema-2.32.2-SNAPSHOT.jar!/META-INF/dhis/beans.xml]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘org.hisp.dhis.schema.PropertyIntrospectorService’: Unsatisfied dependency expressed through field ‘sessionFactory’; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘flyway’ defined in URL [jar:file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-support-db-migration-2.32.2-SNAPSHOT.jar!/META-INF/dhis/beans.xml]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Found more than one migration with version 2.31.1

Offenders:

→ /Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-support-db-migration-2.32.2-SNAPSHOT.jar!/org/hisp/dhis/db/migration/2.31/V2_31_1__Migrations_for_release_v31.sql (SQL)

→ /Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-support-db-migration-2.32.2-SNAPSHOT.jar!/org/hisp/dhis/db/migration/2.31/V2_31_1__Migrations_for_release_v31 2.sql (SQL) (AbstractApplicationContext.java [main])

ERROR 2019-07-22 11:12:55,122 Context initialization failed (ContextLoader.java [main])

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘org.hisp.dhis.schema.MergeService’ defined in URL [jar:file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-service-schema-2.32.2-SNAPSHOT.jar!/META-INF/dhis/beans.xml]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘org.hisp.dhis.schema.SchemaService’ defined in URL [jar:file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-service-schema-2.32.2-SNAPSHOT.jar!/META-INF/dhis/beans.xml]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘org.hisp.dhis.schema.PropertyIntrospectorService’: Unsatisfied dependency expressed through field ‘sessionFactory’; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘flyway’ defined in URL [jar:file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-support-db-migration-2.32.2-SNAPSHOT.jar!/META-INF/dhis/beans.xml]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Found more than one migration with version 2.31.1

Offenders:

→ /Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-support-db-migration-2.32.2-SNAPSHOT.jar!/org/hisp/dhis/db/migration/2.31/V2_31_1__Migrations_for_release_v31.sql (SQL)

→ /Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/file:/Users/gaurav/Documents/Gaurav/Work/Source/blackbird/dhis2-core/dhis-2/dhis-web/dhis-web-portal/target/dhis/WEB-INF/lib/dhis-support-db-migration-2.32.2-SNAPSHOT.jar!/org/hisp/dhis/db/migration/2.31/V2_31_1__Migrations_for_release_v31 2.sql (SQL)

Ameen · 22 July 2019 09:54

@Gaurav_Kumar Can you explain how are you deploying? Deploying the war into a tomcat local installation webapps folder? Also, which specific version of 2.31 did you deploy when upgrading? Was it the latest 2.31.4?

Gaurav_Kumar · 23 July 2019 05:44

Hi @Ameen

a. I am using the “mvn clean -DskipTests=true jetty:run-war” command to deploy the dhis-web-portal in my macbook-pro. I am in sync with the latest 2.32.2-snapshot from the DHIS2 repo.

b. Yes I used the latest 2.31.4 version which was deployed successfully with our DB before upgrading to 2.32.

Also, I have seen similar error before in which some of the “.java” files in DHIS2 API couldn’t be compiled as they were showing with the number 2 appended to their name. For example categoryComboStore.java was showing up as “categoryComboStore 2.java”. I got around that error by deleting my local branch and then performing a fresh checkout. Could this error be because of the same reason? if yes, why is it happening ?

Ameen · 23 July 2019 08:55

@Gaurav_Kumar I am not certain as to why that is happening. But usually its related to the build-deploy mechanism. Some IDEs (Netbeans for example) sometimes have such issues where it loads the same classes/resources into the classpath more than once. Do you have the project open in an IDE when you are executing the mvn command above?

Gaurav_Kumar · 23 July 2019 09:10

Hi @Ameen, resolved. Did a fresh build(after pulling/syncing with the 2.32 repo) & cleaned all modules. The 2.32 is now successfully running with our database.
Thank you for the prompt support buddy, appreciated much.