Backup & Disaster Recovery

What survives a lost disk, and how to bring the stack back from a backup.

What is backed up

The backup tree holds secrets (the keystore is password-protected, the .env is plain text). Keep OTAVI_BACKUP_VOLUMES private and point OTAVI_BACKUP_REMOTE at a private bucket or, better, an rclone crypt remote.

Redis is intentionally not backed up. The BullMQ queue rebuilds, and unsent chunks re-drain from the satellite outboxes, so there is nothing worth saving. Nor are two caches: storage/worker-models (the worker downloads its models again from the birdnet-models bucket) and storage/perch (a copy of the Perch model the bucket also holds).

The backup sidecar (in docker-compose.yml, image built from config/backup/) writes everything to a local directory first (fast restore), then pushes the whole tree off-site with rclone (real disaster recovery).

Configuration

All via .env (see .env.example):

The in-stack MinIO is reached over S3 via RCLONE_CONFIG_MINIO_* env wired in the compose service; do not configure it in rclone.conf. That file is only for the off-site offsite remote (templated with examples for S3 / B2 / SFTP).

Off-site setup

1. Set OTAVI_BACKUP_ENABLED=true in .env (scheduled backups are off by default).
2. Edit config/backup/rclone.conf (or point OTAVI_BACKUP_RCLONE_CONFIG elsewhere): add an [offsite] section for your provider.
3. Set OTAVI_BACKUP_REMOTE=offsite:<bucket-or-path> in .env.
4. docker compose up -d backup.

Without these, backups are local-only, which is not disaster recovery (a lost disk takes the backups with it).

Operating

The sidecar runs one backup at boot (surfacing config errors immediately) then on OTAVI_BACKUP_SCHEDULE. Run one on demand:

docker compose run --rm backup backup.sh

Inspect the latest run without unpacking anything:

cat "$OTAVI_BACKUP_VOLUMES/MANIFEST.txt"

pnpm ops:check reads that manifest and fails when the last backup is older than 26 hours. Until hub 0.67.5 the sidecar wrote its schedule with a user column that busybox crond does not take, so only the run at container start ever happened; a running sidecar did not mean scheduled backups.

Restore

Restoring is destructive: it overwrites the live database and MinIO buckets. Do it deliberately.

1. Stop the writers

The hub must not be running during a Postgres restore (it holds connections and would race migrations):

docker compose stop api dispatcher worker web

2. Restore Postgres + MinIO

# Restores the most recent local dump; pass a path to pick another.
docker compose run --rm backup restore.sh
# or:  docker compose run --rm backup restore.sh /backups/postgres/birdnet-YYYYMMDD-HHMMSS.dump

This pg_restore --cleans the database and rclone syncs each bucket mirror back into MinIO.

3. Restore the Mosquitto credentials (if lost)

With the broker stopped, copy the saved file back into its data volume, then start it:

docker compose stop mosquitto
cp "$OTAVI_BACKUP_VOLUMES/mosquitto/dynamic-security.json" "$OTAVI_APP_VOLUMES/mosquitto/data/"
docker compose up -d mosquitto

3b. Restore the secrets (if lost)

Never written automatically: copy them by hand and check the permissions.

cp "$OTAVI_BACKUP_VOLUMES/secrets/otavi.env" .env && chmod 600 .env
mkdir -p "$OTAVI_APP_VOLUMES/mobile" && cp "$OTAVI_BACKUP_VOLUMES/secrets/release.keystore" "$OTAVI_APP_VOLUMES/mobile/"

4. Bring the stack back

docker compose up -d

The hub runs migrations on startup; restoring a current dump leaves nothing to apply, but the step is idempotent and safe.

Restoring from off-site

If the local backup dir is gone too, pull it back first, then restore as above:

rclone sync "$OTAVI_BACKUP_REMOTE" "$OTAVI_BACKUP_VOLUMES"

Recovering onto a fresh host

1. Install Docker, clone the repo, restore .env from secrets/otavi.env in the backup (keep it somewhere safe; it holds OTAVI_AUTH_JWT_SECRET, OTAVI_AUTH_MFA_ENCRYPTION_KEY, and DB/MinIO credentials, all of which must match the backup for sessions and encrypted secrets to keep working).
2. rclone sync "$OTAVI_BACKUP_REMOTE" "$OTAVI_BACKUP_VOLUMES" to pull the backup local.
3. docker compose up -d postgres minio mosquitto (infrastructure only).
4. Run the restore steps above.
5. docker compose up -d.

The .env and the keystore are in secrets/ since hub 0.57.0. Earlier backups do not have them: without the original OTAVI_AUTH_MFA_ENCRYPTION_KEY every enrolled TOTP and every sealed webhook secret is unreadable, and without the keystore every installed phone must be reinstalled.

Verifying a backup

A backup nobody has restored is a guess. To check a dump restores cleanly without touching production, restore it into a scratch database and compare counts:

docker exec <postgres> psql -U birdnet -d postgres -c "CREATE DATABASE restore_check;"
PW=$(grep ^OTAVI_DB_PASSWORD= .env | cut -d= -f2-)
docker compose run --rm -e PGPASSWORD="$PW" backup \
  pg_restore --clean --if-exists --no-owner --no-privileges \
  -h postgres -U birdnet -d restore_check /backups/postgres/<dump>
docker exec <postgres> psql -U birdnet -d restore_check -c "SELECT count(*) FROM detections;"
docker exec <postgres> psql -U birdnet -d postgres -c "DROP DATABASE restore_check;"