Upgrade 2.34 to 2.35 : Detected failed migration to version 2.35.4 (Remove color set)

Implementation - Implémentation
#1

the tomcat server starts but the migration fails.

Caused by: org.flywaydb.core.api.FlywayException: Validate failed: Detected failed migration to version 2.35.4 (Remove color set)

this is what I suppose the script running

Since some tables were dropped, I deduced it’s the color table causing the issue

I ran manually in psql the following command : drop table if exists color;

The command outputs this :

ERROR:  cannot drop table color because other objects depend on it
DETAIL:  constraint fk_objecttranslation_colorid on table colortranslations depends on table color

The thing I don’t understand all these tables are empty

dhis2=> select * from colortranslations;
 colorid | objecttranslationid 
---------+---------------------
(0 rows)

dhis2=> select * from color;
 colorid | uid | code | created | lastupdated | lastupdatedby | color | name | translations 
---------+-----+------+---------+-------------+---------------+-------+------+--------------
(0 rows)

I assume the migration that drops the table “forgot” about colortranslations

#2

I manually dropped the colortranslations table

Restarted the server but the migration still fails

drop table if exists colorset;
ERROR:  cannot drop table colorset because other objects depend on it
DETAIL:  constraint fk_objecttranslation_colorsetid on table colorsettranslations depends on table colorset
HINT:  Use DROP ... CASCADE to drop the dependent objects too.

so I dropped also colorsettranslations

Restarted the server but the migration still fails…

Is there something to tell to flyway to relaunch the migration ?

#3

so I delete a flyway record

delete from flyway_schema_history where flyway_schema_history.success = false

now the server starts

* INFO  2020-11-10T13:01:40,066 Executing startup routine [6 of 10, runlevel 11]: ExpressionUpgrader (DefaultStartupRoutineExecutor.java [localhost-startStop-1])
 * ERROR 2020-11-10T13:01:40,184 Upgrading custom data entry form failed: S KPI - Initiation (ExpressionUpgrader.java [localhost-startStop-1])
 java.lang.NumberFormatException: For input string: "ElNWXB4odHy"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:1.8.0_272]
	at java.lang.Integer.parseInt(Integer.java:580) ~[?:1.8.0_272]
	at java.lang.Integer.parseInt(Integer.java:615) ~[?:1.8.0_272]
	at org.hisp.dhis.startup.ExpressionUpgrader.upgradeDataEntryForms(ExpressionUpgrader.java:232) [dhis-service-core-2.35.0.jar:?]
	at org.hisp.dhis.startup.ExpressionUpgrader.executeInTransaction(ExpressionUpgrader.java:107) [dhis-service-core-2.35.0.jar:?]
	at org.hisp.dhis.system.startup.TransactionContextStartupRoutine$1.doInTransaction(TransactionContextStartupRoutine.java:58) [dhis-support-system-2.35.0.jar:?]
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) [spring-tx-5.2.8.RELEASE.jar:5.2.8.RELEASE]
	at org.hisp.dhis.system.startup.TransactionContextStartupRoutine.execute(TransactionContextStartupRoutine.java:53) [dhis-support-system-2.35.0.jar:?]
	at org.hisp.dhis.system.startup.DefaultStartupRoutineExecutor.execute(DefaultStartupRoutineExecutor.java:114) [dhis-support-system-2.35.0.jar:?]
	at org.hisp.dhis.system.startup.DefaultStartupRoutineExecutor.execute(DefaultStartupRoutineExecutor.java:71) [dhis-support-system-2.35.0.jar:?]
	at org.hisp.dhis.system.startup.StartupListener.contextInitialized(StartupListener.java:70) [dhis-support-system-2.35.0.jar:?]
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4697) [webapp-runner.jar:?]
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5165) [webapp-runner.jar:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [webapp-runner.jar:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1412) [webapp-runner.jar:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1402) [webapp-runner.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_272]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_272]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_272]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272]

so I assume some custom data entry won’t work as expected

#4

the htmlcode associated
"<p><strong><span style=\"font-size:16px;\">Milestone Initiation</span></strong></p>\n\n<p><strong>Date of Initiation&nbsp;</strong></p>\n\n<p><input id=\"ElNWXB4odHy-PstoNhHeToG-val\" name=\"entryfield\" title=\"S KPI - Initiation - Date\" value=\"[ S KPI - Initiation - Date ]\"/></p>\n\n<p><strong>Work Achieved</strong></p>\n\n<p><input id=\"ElNWXB4odHy-t5auUVbOk1T-val\" name=\"entryfield\" title=\"S KPI - Initiation - Achievement\" value=\"[ S KPI - Initiation - Achievement ]\"/></p>\n\n<p><strong>Evidence File</strong></p>\n\n<p><input id=\"ElNWXB4odHy-lEj4nqBz9mR-val\" name=\"entryfield\" title=\"S KPI - Initiation - Evidence\" value=\"[ S KPI - Initiation - Evidence ]\"/></p>\n\n<p>&nbsp;</p>\n"

#5

not sure If I should be concerned… the expression update seem look for expression with “id” and turn them in “uuid”

#6

reported in jira : https://jira.dhis2.org/browse/DHIS2-9950

#7

@Stephan_Mestach colortranslations and colorsettranslations were removed in version 2.31. I am surprised how it still exists in your 2.34 db. https://github.com/dhis2/dhis2-core/blob/de6f851903b5535d5f0518b31c7307ac960b4dc5/dhis-2/dhis-support/dhis-support-db-migration/src/main/resources/org/hisp/dhis/db/migration/2.31/V2_31_1__Migrations_for_release_v31.sql#L348 This should have been executed and thereby removing it from the db.

For the second part of the issue (expressionUpgrader), that might be a potential candidate for a jira issue so we can look on it further.

#8

Sorry for this particular db; don’t know much how the migration was performed before 2.34.

Flyway seem to have been added in 2.31, I found this record in the flyway_schema_history

installed_rank | 3
version        | 2.31.1
description    | Migrations for release v31
type           | SQL
script         | 2.31/V2_31_1__Migrations_for_release_v31.sql
checksum       | -271885416
installed_by   | dhis2
installed_on   | 2020-03-11 10:21:42.020018
execution_time | 2820
success        | True

so indeed can’t explain the translation tables are still there.

#9

Hope it’s not too late but the solution to this problem as Oslo fix this issue, delete all version 2.5xx in the flyway_history table and drop every column flyway has added to the tables. This worked.