Questions About Updating DHIS2 Backup Scripts and Backup Frequency

Hi DHIS2 Community,

I recently took over administration of our DHIS2 infrastructure running on Ubuntu LXD containers. I am reviewing our automated PostgreSQL backup scripts and cron schedules, and I would appreciate feedback on script update.

Our Environment & LXD Architecture

We run LXD containers on an Ubuntu host with separate DHIS2 instances and a centralized PostgreSQL container:

  • postgres(Container IP: xxxx) — PostgreSQL server hosting 3 databases:
    • app— DHIS2 Aggregate data entry instance
    • vida — DHIS2 Tracker data entry instance
    • dhis— Static read-only historical Tracker instance
  • app(Container IP: xxxx) — Aggregate DHIS2 instance
  • vida(Container IP: xxxx) — Tracker DHIS2 instance
  • dhis(Container IP: xxxx) — Read-only DHIS2 instance
  • proxy(Container IP: xxxx) &monitor(Container IP: xxxx)

LXD Host

├── app (DHIS2 Aggregate)

├── vida (DHIS2 Tracker)

├── dhis (DHIS2 Read-Only)

├── postgres (Centralized PostgreSQL Engine)

│ ├── database: app

│ ├── database: vida

│ └── database: dhis

├── proxy (NGINX Reverse Proxy)

└── monitor (Munin Monitoring)

Current Backup Setup & Cron Configuration

Our backup job runs daily via root crontab at 03:00 AM UTC:

0 3 * * * /usr/local/bin/dhis2-backup

The script performs the following operations via LXC execution commands:

  1. Dumps and truncates theaudittable separately into compressed SQL files.
  2. Runs pg_dump -O -Fp $DBNAME $EXCLUDED | gzip across active databases (vida, app, dhis).
  3. Excludes temporary/analytics tables (-T aggregated_* -T analytics_* -T completeness_*).
  4. Manages multi-tier rotation (Daily: 7 days retention, Weekly: 4 weeks retention, Monthly: 2 months retention).

Questions for the Community:

  1. Editing the Backup Script: We only want to correct a few typos and make minor changes to our backup script. Does editing or replacing the backup script require stopping PostgreSQL, DHIS2/Tomcat, or any other service? Or is it sufficient to update the script and let the next scheduled cron job use the new version?
  2. Static Read-Only Database: Our dhis database is a historical read-only instance that is never modified. Is it common practice to remove such a database from the daily backup job and back it up only when changes are made, or do most DHIS2 administrators continue including it in their regular automated backups?
    Thank you for your guidance and insights!

Best regards,

Hi Fernando

Thanks for the thorough description. Taking your questions one at a time:

  1. Editing the backup scripts doesn’t require starting or stopping any services. One caution I would have is that you might want to call it something else (eg dhis2-moz-backup) in case your changes get over written the next time you run ansible playbook. And edit your crontab accordingly.
  2. The default backup script operates off an environment file (/usr/local/etc/dhis/dhis2-env). Check if there is a variable in there called PLAIN_BACKUPS. By default this will list all the databases. Just remove the ones you dont need to back up and it will exclude them.

There have been some very recent changes which I need to verify, but if you have the PLAIN_BACKUPS variable then you can just edit as above.

Regards
Bob

Hi Bob,

Thank you for the clear feedback and guidance!

Regarding the environment file, checking/usr/local/etc/dhis/dhis2-envand updatingPLAIN_BACKUPSto exclude the static read-only database (dhis) is very straightforward.

For updating both/usr/local/etc/dhis/dhis2-envand/usr/local/bin/dhis2-backup, I plan to prepare the edits in advance. Between direct editing in place (e.g., usingsudo nanodirectly on the active file) versus atomic replacement (editing a.newfile, checking syntax/permissions, and usingsudo mvto replace the live file), is atomic replacement considered best practice for this environment, or is direct editing typically safe enough since these files are only read once daily when the cron job triggers?

Approach 1: Atomic Replacement
-rwx-----x 1 root root 3659 Jan 21 2025 /usr/local/bin/dhis2-backup

# Prepare workspace copy, edit, check syntax/permissions, and move into place:

sudo cp -a /usr/local/bin/dhis2-backup /usr/local/bin/dhis2-backup.new

sudo nano /usr/local/bin/dhis2-backup.new

sudo bash -n /usr/local/bin/dhis2-backup.new

sudo chmod 701 /usr/local/bin/dhis2-backup.new

sudo chown root:root /usr/local/bin/dhis2-backup.new

sudo cp -a /usr/local/bin/dhis2-backup /usr/local/bin/dhis2-backup.$(date +%Y%m%d-%H%M%S).bak

sudo mv /usr/local/bin/dhis2-backup.new /usr/local/bin/dhis2-backup

Approach 2: Direct Edit

# Backup current version and edit in-place:

sudo cp -a /usr/local/bin/dhis2-backup /usr/local/bin/dhis2-backup.$(date +%Y%m%d-%H%M%S).bak

sudo nano /usr/local/bin/dhis2-backup

sudo bash -n /usr/local/bin/dhis2-backup

Also, if you don’t mind, may I paste the current backup script here? I’d appreciate your feedback on a few minor changes and corrections before I deploy them.

Thanks again for your guidance.

Regards,

Fernando

Please do share your modified version.

There is no major concern around editing in place. But see my advice above about copying the file. If your edited version is called dhis2-backup then it might get overriden with an update. Better make a copy and edit that. Then change crontab to use your version

Hi Bob,

Thank you for clarifying! That makes total sense regarding why creating a custom script copy (e.g.,dhis2-moz-backup) is safer—to prevent an Ansible playbook run or upstream package update from overwriting local changes.

Just to confirm, your recommended workflow is:

  1. Copy and rename script:

Bash

sudo cp -a /usr/local/bin/dhis2-backup /usr/local/bin/dhis2-moz-backup

sudo chmod 700 /usr/local/bin/dhis2-moz-backup

  1. Update Root Crontab (sudo crontab -e):

0 3 * * * /usr/local/bin/dhis2-moz-backup

If dhis2-env is managed by Ansible/upstream, would you recommend creating a local/custom environment file in the same way as the custom backup script, e.g. dhis2-moz-env, and having dhis2-moz-backup source that file instead?

***grep -Rni --exclude-dir=.git “dhis2-backup” ~/dhis2-server-tools/ ***

roles/backups/tasks/lxd.yml:

*** src: dhis2-env.j2***

*** dest: /usr/local/etc/dhis/dhis2-env***

*** job: /usr/local/bin/dhis2-backup***

As requested, here is the current/usr/local/bin/dhis2-backupscript written by Tito Kipkurgat currently running on our server, along with our/usr/local/etc/dhis/dhis2-envconfiguration.

1. Script (/usr/local/bin/dhis2-backup)

#!/usr/bin/env bash

#########################################################

# postgres backup script v3.1

# author: Tito Kipkurgat

# licence: public domain

# using some ideas from

# Automated Backup on Linux - PostgreSQL wiki

########################################################

# load variables from a file

source /usr/local/etc/dhis/dhis2-env

function perform_backups()

{

SUFFIX=$1

DB_BUCKUP_DIR=$BACKUP_DIR/db-backup

AUDIT_BACKUP_DIR=$BACKUP_DIR/audit-backup

FINAL_BACKUP_DIR=$DB_BUCKUP_DIR/“$(date +%Y%m%d-%H%M%S)$SUFFIX”

FINAL_AUDIT_DIR=$AUDIT_BACKUP_DIR/“$(date +%Y%m%d-%H%M%S)$SUFFIX”

echo “Backup Directory = $FINAL_BACKUP_DIR”

echo “Audit Directory = $FINAL_AUDIT_DIR”

if ! mkdir -p $FINAL_BACKUP_DIR; then

echo “`date` Cannot create backup directory in $FINAL_BACKUP_DIR. Go and fix it!”

exit 1;

fi;

if ! mkdir -p $FINAL_AUDIT_DIR; then

echo “`date` Cannot create backup directory in $FINAL_AUDIT_DIR. Go and fix it!”

exit 1;

fi;

for DBNAME in $PLAIN_BACKUPS

do

set -o pipefail

# First backup and truncate audit table

if ! lxc exec postgres – pg_dump -a -t audit $DBNAME | gzip - > “$FINAL_AUDIT_DIR/“audit_”$DBNAME”.sql.gz.in_progress; then

echo “`date` [!!ERROR!!] Failed to backup audit table of database $DBNAME”

else

mv “$FINAL_AUDIT_DIR”/audit_“$DBNAME”.sql.gz.in_progress “$FINAL_AUDIT_DIR”/audit_“$DBNAME”.sql.gz

echo “TRUNCATE audit” | lxc exec postgres psql $DBNAME

fi

# Perform backup of the main database

if ! lxc exec postgres – pg_dump -O -Fp $DBNAME $EXCLUDED | gzip > $FINAL_BACKUP_DIR/“$DBNAME”.sql.gz.in_progress; then

echo “`date` [!!ERROR!!] Failed to produce plain backup of database $DBNAME”

else

mv $FINAL_BACKUP_DIR/“$DBNAME”.sql.gz.in_progress $FINAL_BACKUP_DIR/“$DBNAME”.sql.gz

fi

done

for DBNAME in $ENCRYPTED_BACKUPS

do

set -o pipefail

if ! lxc exec postgres – pg_dump -O -Fp $DBNAME $EXCLUDED | gzip | openssl $CIPHER -e -pass file:$PASSWORD_FILE -salt > $FINAL_BACKUP_DIR"$DBNAME".sql.gz.enc.in_progress; then

echo “`date` [!!ERROR!!] Failed to produce plain backup of database $DBNAME”

else

mv $FINAL_BACKUP_DIR"$DBNAME".sql.gz.enc.in_progress $FINAL_BACKUP_DIR"$DBNAME".sql.gz.enc

fi

done

# if a remote backup machine is defined, rsync to it

if [ -n “$REMOTE” ]; then

rsync -avq $BACKUP_DIR/* $REMOTE

fi

}

# MONTHLY BACKUPS

DAY_OF_MONTH=`date +%d`

if [ $DAY_OF_MONTH = “01” ]; then

# Delete all expired monthly directories, keeping only two

find $BACKUP_DIR -maxdepth 1 -name “*-monthly” -mtime +60 -exec rm -rf ‘{}’ ‘;’

find $DB_BUCKUP_DIR -maxdepth 1 -name “*-monthly” -mtime +60 -exec rm -rf ‘{}’ ‘;’

perform_backups “-monthly”

fi

# WEEKLY BACKUPS

DAY_OF_WEEK=$(date +%u) #1-7 (Monday-Sunday)

EXPIRED_DAYS=$(expr $((($WEEKS_TO_KEEP * 7) + 1)))

if [ $DAY_OF_WEEK = $DAY_OF_WEEK_TO_KEEP ]; then

# Delete all expired weekly directories

find $DB_BUCKUP_DIR -maxdepth 1 -mtime +$EXPIRED_DAYS -name “*-weekly” -exec rm -rf ‘{}’ ‘;’

find $BUCKUP_DIR -maxdepth 1 -mtime +$EXPIRED_DAYS -name “*-weekly” -exec rm -rf ‘{}’ ‘;’

perform_backups “-weekly”

fi

# Delete daily backups 7 days old or more

find $DB_BUCKUP_DIR -maxdepth 1 -mtime +$DAYS_TO_KEEP -name “*-daily” -exec rm -rf ‘{}’ ‘;’

find $BUCKUP_DIR -maxdepth 1 -mtime +$DAYS_TO_KEEP -name “*-daily” -exec rm -rf ‘{}’ ‘;’

perform_backups “-daily”

2. Environment File (/usr/local/etc/dhis/dhis2-env)

BACKUP_DIR=/var/pgbackups

PLAIN_BACKUPS=“dhis viva app treino”

EXCLUDED="-T aggregated_* -T analytics_* -T completeness_* "

CIPHER=“aes-256-cbc”

DAY_OF_WEEK_TO_KEEP=7

WEEKS_TO_KEEP=4

DAYS_TO_KEEP=7

Noted Issues & Areas Where We Seek Guidance:

  1. Typo Bugs in Variables:
    • $DB_BUCKUP_DIRvs$BUCKUP_DIR(e.g.,find $BUCKUP_DIR …). Because$BUCKUP_DIRis undefined,findattempts to search from root or fails silently.
    • Path missing a trailing slash in encrypted backup target:$FINAL_BACKUP_DIR"$DBNAME"…
  2. Hard-coded Retention Logic (Monthly vs Daily):
    • The monthly cleanup uses a fixed-mtime +60rather than leveraging dynamic retention values.
    • On the 1st day of the month, the script runs monthly cleanup/backup, then continues sequentially to perform daily cleanup/backup on the same run, creating duplicate directory creations or overlaps.

We would love your suggestions on the cleanest way to refactor this script to fix these typos and else.

Thanks again for your support!

Best regards

Fernando