Can't upgrade from 2.36 to 2.39

Hi developers, community

I am trying to upgrade from 2.36.12 to 2.39.2.1 and facing number of problems, although I never ever had before any problems with upgrading process starting from 2.32 till 2.36.

First problem was connected to the job scheduler - I have resolved it by removing all custom jobs scheduled and rerunning upgrade process.
But then I have received number of problems which can be found in the attached catalina_out.docx file below
catalina_out.docx (34.1 KB)

Any suggestions on the errors and how to do upgrade process more smooth?

Regards,
Ulanbek

cc: @phil

Hi @Ulanbek

Looking at the log, it shows about 50 times the error where it says ‘already exists’ and most of these issues appear to be related to the resource tables.

Would you try to use Maintenance (I’d select all the choices) in the Data Administration app and then generate resource tables again, then restart before running an upgrade? Hopefully this clears away these issues.

Additionally, in one of the posts in the community, it’s mentioned that improper RAM configuration could cause these issues to arise so it might be worth checking that before running the upgrade process.

Thank you! I hope this helps.

@Ulanbek literally the problem is not upgrading from 2.36.12 to 2.39.2.1 but its from 2.36.12 to 2.37… If you pass this stage you can easily jump to 2.39. There is a need for some scripts that need to be executed in the database otherwise upgrading by just changing war file you might face such challenge .

Thant is my thoughts and past experience

1 Like

Hi @Gassim

Thank you for your reply.
I have analysed as well the error source and I believe this is something connected to the code bug.
I got the error message “ERROR: constraint “fk_organisationunit_fileresourceid” for relation “organisationunit” already exists” and removed manually it from the table.
But during the upgrading process it recreated again - the script name V2_37_14__Add_image_column_into_organisationunit_table.sql
Moreover why the script cannot just check first if the relationship constraint already exists and go to the next instruction?

Most probably the source of error is here

ERROR 2023-10-04T17:46:09,879 Migration of schema “public” to version “2.37.14 - Add image column into organisationunit table” failed! Changes successfully rolled back. (Log4j2Log.java [main])

  • WARN 2023-10-04T17:46:09,882 Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘webMvcConfig’: Unsatisfied dependency expressed through field ‘userSettingService’; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘org.hisp.dhis.user.UserSettingService’ defined in URL [jar:file:/usr/local/tomcat9.39/webapps/ROOT/WEB-INF/lib/dhis-service-core-2.39.2.1.jar!/org/hisp/dhis/user/DefaultUserSettingService.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘org.hisp.dhis.user.UserSettingStore’ defined in URL [jar:file:/usr/local/tomcat9.39/webapps/ROOT/WEB-INF/lib/dhis-service-core-2.39.2.1.jar!/org/hisp/dhis/user/hibernate/HibernateUserSettingStore.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘flyway’ defined in class path resource [org/hisp/dhis/db/migration/config/FlywayConfig.class]: Invocation of init method failed; nested exception is org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException: Migration V2_37_14__Add_image_column_into_organisationunit_table.sql failed

and it has connection to the Users table due to upgrade error from 2.36 to 2.39

Regards,

@Shapr0019 I do agree with you! I just pointed this issue in my previous post.

Do you think I should try to first upgrade it to 2.37 and then 2.39? Hopefully developers will fix this issue as I’m not only the person who is trying to upgrade.

Thank you @Shapr0019

@phil @Gassim

I have tried to upgrade from 2.36 to 2.37
The same errors can be found there. Please check out the attached 2.36 to 2.37 upgrading catalina.out file below

Catalina_out_2.37_upgrade.docx (23.9 KB)

Thank you
Regards,

Hi @Ulanbek,

It seems that the Flyway migration script you pointed out has not been created in an idempotent way. Unfortunately it is not easy to change the script without breaking migrations that have already gone above that version.
A workaround would be to remove the conflicting constraint from the table just before upgrading, with something like:

ALTER TABLE organisationunit
DROP CONSTRAINT fk_organisationUnit_fileresourceid;

Then it should be possible to upgrade directly to 2.39.

Kind regards,
Phil

1 Like

Hi @phil,

I have followed you suggestions. Removed directly in the DB contraint you have mentioned.
Now I have following errors. Could you check the catalina_out pls?

catalina_OUT.pdf (99.1 KB)

Thank you

Hi @Ulanbek

There are other instances in the log with the ‘already exists’ error, so it appears the conflicting constraints exist in several tables and not just the organisationunit table.

@Ulanbek, would it be possible to test upgrading from 2.36 directly to version 40.1.0?

Hi @Gassim

Thank you for your suggestion. I have followed and tried to upgrade directly from 2.36 to 2.40. As a result all errors what I had before appeared here as well. Please check out the catalina.out file below

catalina_out.pdf (99.1 KB)
And now have no idea what should be solved first…

Thank you
Regards,

1 Like

Hi there,

Finally I have managed to upgrade from 2.36 to 2.39, even 2.40.

I have spent many different ways to upgrade, and sharing my steps to upgrade incase if anyone need to use in the future.
Before to start everything just backup all your DB

  1. Remove all your custom jobs scheduled - ver2.36. Just in case create a backup of current state
  2. Remove all your analytics - ver.2.37 - Just in case create a backup of current state. I used it to revert DB when upgrade failed
  3. Upgrade to 2.37
  4. Update your analytics - Just in case create a backup of current state
  5. Upgrade to 2.39 - Just in case create a backup of current state
  6. Upgrade to 2.40

So hopefully it will be useful for someone

Regards,

1 Like

Thank you for sharing your steps @Ulanbek !

Does this mean you managed to upgrade by following those steps and making no other changes to the database? That would be a bit mysterious to me, as my expectation is that analytics generation and data model migrations should never affect the same tables - i.e I would expect them to be mutually exclusive.

Also, given your steps, I would expect you to be able to skip steps 3,4 and 5. Is that something you tried without success?

Thanks again, and in any case I’m glad to hear that you managed to complete the upgrade, and hope you can now make use of some of the features of 2.40!

Kind regards,
Phil

1 Like

Hi @phil

Yes, I have tried many other steps.
And these steps finally were fruitful.
For the analytical tables I had a problem that periods table couldn’t be found.

So anyway I spent maybe around 2 weeks to find a right way for upgrade and finally got it.

I believe all problems are connected to the users script, when upgrading from 2.36 to 2.37.
The rest versions assumes that the user migration has been done therefore they don’t have such script and as a consequence they have all the problems connected to the users permissions etc. Please check the versions 2.38 and higher so they must have user migration script included.

Regards,
Ulanbek