Upgrade 2.34 to 2.35 : Detected failed migration to version 2.35.4 (Remove color set)

the tomcat server starts but the migration fails.

Caused by: org.flywaydb.core.api.FlywayException: Validate failed: Detected failed migration to version 2.35.4 (Remove color set)

this is what I suppose the script running

Since some tables were dropped, I deduced it’s the color table causing the issue

I ran manually in psql the following command : drop table if exists color;

The command outputs this :

ERROR:  cannot drop table color because other objects depend on it
DETAIL:  constraint fk_objecttranslation_colorid on table colortranslations depends on table color

The thing I don’t understand all these tables are empty

dhis2=> select * from colortranslations;
 colorid | objecttranslationid 
(0 rows)

dhis2=> select * from color;
 colorid | uid | code | created | lastupdated | lastupdatedby | color | name | translations 
(0 rows)

I assume the migration that drops the table “forgot” about colortranslations

I manually dropped the colortranslations table

Restarted the server but the migration still fails

drop table if exists colorset;
ERROR:  cannot drop table colorset because other objects depend on it
DETAIL:  constraint fk_objecttranslation_colorsetid on table colorsettranslations depends on table colorset
HINT:  Use DROP ... CASCADE to drop the dependent objects too.

so I dropped also colorsettranslations

Restarted the server but the migration still fails…

Is there something to tell to flyway to relaunch the migration ?

so I delete a flyway record

delete from flyway_schema_history where flyway_schema_history.success = false

now the server starts

* INFO  2020-11-10T13:01:40,066 Executing startup routine [6 of 10, runlevel 11]: ExpressionUpgrader (DefaultStartupRoutineExecutor.java [localhost-startStop-1])
 * ERROR 2020-11-10T13:01:40,184 Upgrading custom data entry form failed: S KPI - Initiation (ExpressionUpgrader.java [localhost-startStop-1])
 java.lang.NumberFormatException: For input string: "ElNWXB4odHy"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:1.8.0_272]
	at java.lang.Integer.parseInt(Integer.java:580) ~[?:1.8.0_272]
	at java.lang.Integer.parseInt(Integer.java:615) ~[?:1.8.0_272]
	at org.hisp.dhis.startup.ExpressionUpgrader.upgradeDataEntryForms(ExpressionUpgrader.java:232) [dhis-service-core-2.35.0.jar:?]
	at org.hisp.dhis.startup.ExpressionUpgrader.executeInTransaction(ExpressionUpgrader.java:107) [dhis-service-core-2.35.0.jar:?]
	at org.hisp.dhis.system.startup.TransactionContextStartupRoutine$1.doInTransaction(TransactionContextStartupRoutine.java:58) [dhis-support-system-2.35.0.jar:?]
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) [spring-tx-5.2.8.RELEASE.jar:5.2.8.RELEASE]
	at org.hisp.dhis.system.startup.TransactionContextStartupRoutine.execute(TransactionContextStartupRoutine.java:53) [dhis-support-system-2.35.0.jar:?]
	at org.hisp.dhis.system.startup.DefaultStartupRoutineExecutor.execute(DefaultStartupRoutineExecutor.java:114) [dhis-support-system-2.35.0.jar:?]
	at org.hisp.dhis.system.startup.DefaultStartupRoutineExecutor.execute(DefaultStartupRoutineExecutor.java:71) [dhis-support-system-2.35.0.jar:?]
	at org.hisp.dhis.system.startup.StartupListener.contextInitialized(StartupListener.java:70) [dhis-support-system-2.35.0.jar:?]
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4697) [webapp-runner.jar:?]
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5165) [webapp-runner.jar:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [webapp-runner.jar:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1412) [webapp-runner.jar:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1402) [webapp-runner.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_272]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_272]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_272]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272]

so I assume some custom data entry won’t work as expected

the htmlcode associated
"<p><strong><span style=\"font-size:16px;\">Milestone Initiation</span></strong></p>\n\n<p><strong>Date of Initiation&nbsp;</strong></p>\n\n<p><input id=\"ElNWXB4odHy-PstoNhHeToG-val\" name=\"entryfield\" title=\"S KPI - Initiation - Date\" value=\"[ S KPI - Initiation - Date ]\"/></p>\n\n<p><strong>Work Achieved</strong></p>\n\n<p><input id=\"ElNWXB4odHy-t5auUVbOk1T-val\" name=\"entryfield\" title=\"S KPI - Initiation - Achievement\" value=\"[ S KPI - Initiation - Achievement ]\"/></p>\n\n<p><strong>Evidence File</strong></p>\n\n<p><input id=\"ElNWXB4odHy-lEj4nqBz9mR-val\" name=\"entryfield\" title=\"S KPI - Initiation - Evidence\" value=\"[ S KPI - Initiation - Evidence ]\"/></p>\n\n<p>&nbsp;</p>\n"

not sure If I should be concerned… the expression update seem look for expression with “id” and turn them in “uuid”

reported in jira : https://jira.dhis2.org/browse/DHIS2-9950

@Stephan_Mestach colortranslations and colorsettranslations were removed in version 2.31. I am surprised how it still exists in your 2.34 db. https://github.com/dhis2/dhis2-core/blob/de6f851903b5535d5f0518b31c7307ac960b4dc5/dhis-2/dhis-support/dhis-support-db-migration/src/main/resources/org/hisp/dhis/db/migration/2.31/V2_31_1__Migrations_for_release_v31.sql#L348 This should have been executed and thereby removing it from the db.

For the second part of the issue (expressionUpgrader), that might be a potential candidate for a jira issue so we can look on it further.

Sorry for this particular db; don’t know much how the migration was performed before 2.34.

Flyway seem to have been added in 2.31, I found this record in the flyway_schema_history

installed_rank | 3
version        | 2.31.1
description    | Migrations for release v31
type           | SQL
script         | 2.31/V2_31_1__Migrations_for_release_v31.sql
checksum       | -271885416
installed_by   | dhis2
installed_on   | 2020-03-11 10:21:42.020018
execution_time | 2820
success        | True

so indeed can’t explain the translation tables are still there.

Hope it’s not too late but the solution to this problem as Oslo fix this issue, delete all version 2.5xx in the flyway_history table and drop every column flyway has added to the tables. This worked.