Configuration backup and rollback

For Super Admin in-product config version history, whole-platform backups, and JSON export/import of TelWeb-managed platform domains, see Config versioning & backups.

Delphi does not store "the complete platform configuration" in one file. A working deployment is reconstructed from several sources at boot time:

• Bootstrap env on each host: /opt/deployment/.env.
• Generated deployment bundle from S3: Compose files, vars.yaml, service scripts, collector config, Caddyfiles.
• AWS SSM Parameter Store for non-sensitive runtime values.
• AWS Secrets Manager for sensitive runtime values and TLS material.
• Postgres for TelWeb-managed platform, team, app, flow, SIP trunk, number, provider, cron, notification, and user configuration.
• Redis for runtime state only. Redis should rebuild from Postgres and service heartbeats; do not treat it as the source of truth for rollback.

Use this recipe before upgrades, certificate rotations, service-wide config changes, or any change where you need a known-good rollback point.

STOP - know the rollback scope

A config rollback can undo tenant-visible changes. If users changed flows, numbers, providers, or platform settings after your backup point, a Postgres restore will lose those writes. Prefer rolling back only the changed config source when possible.

Choose the recovery scope

What changed or was lost	Capture before changing it	Recovery action	What this does not restore
Container images / generated service files	Running image identity, bootstrap file, and complete configuration bundle reference/version.	Restore the compatible image/config set and restart affected consumers.	Database writes or external objects.
SSM runtime values	Parameter names, versions, and recoverable previous values in the approved configuration store.	Restore the intended values, then refresh all consumers.	Database-backed UI settings.
Secrets or TLS material	Complete secret version IDs and required external certificate/key material.	Restore the correct complete version, coordinating presenting and trusting services.	Expired/revoked certificate validity or provider-side credentials that were independently revoked.
TelWeb-managed configuration	Supported configuration version/export for the affected domain.	Use Config versioning when it covers the change.	Full database history, user sessions, deployment files, or object-store contents.
Database/schema	A verified database backup or managed snapshot and a compatible release set.	Quiesce writers and follow Restore Postgres.	Recordings/media stored outside Postgres or secret values stored in AWS.
Recording/media objects	Object-store versioning/backup or protected volume snapshot appropriate to the deployment.	Recover objects using the storage system's procedure and verify retrieval through Delphi.	Database metadata lost separately.

Metadata inventories identify what must be recovered. They are not substitutes for a recoverable copy: for example, listing a secret name without retaining the needed version does not establish a rollback point.

Preconditions

• Identify the environment and host roles before running commands. Have operator access to those hosts, the applicable AWS account/buckets, and the database backup mechanism.
• Confirm the backup destination exists, has enough space, and is restricted to authorized operators. /backup in these examples is a host path; create/provision an approved location before redirecting output there.
• Record the expected release/schema/config combination with the backup. Verify the old images and bundle will remain available throughout the rollback window.
• Agree the restore point and resulting lost-write interval with the service owner. Pause scheduled automation where it could replace hosts or reapply a changed configuration during recovery.

Step 1 - Record the running release

On every service host you plan to touch:

cd /opt/services/<service>
docker compose ps
docker compose images
grep -E '^(ECR_TAG|CONFIG_REF|CONFIG_BUCKET|NAMESPACE|ENVIRONMENT)=' /opt/deployment/.env

Record the current ECR_TAG, CONFIG_REF, NAMESPACE, and service image tags. update.sh automatically snapshots /opt/deployment/.env before it changes it, but taking a ticket-attached note before the work makes the rollback target unambiguous.

Step 2 - Capture generated config

Generated service config is versioned by CONFIG_REF. Before changing it, confirm which ref each host resolves:

grep -E '^(CONFIG_REF|CONFIG_BUCKET|NAMESPACE)=' /opt/deployment/.env

If your S3 bucket has versioning enabled, keep the object version IDs for the current bundle. If it does not, copy the current bundle to a dated prefix before uploading a replacement.

Step 3 - Snapshot SSM and Secrets Manager

For SSM, export the parameter names, versions, and last-modified timestamps. Store values only in your approved secure evidence store.

aws ssm get-parameters-by-path \
  --path "/<namespace>/<service>/" \
  --recursive \
  --query 'Parameters[].{Name:Name,Version:Version,LastModifiedDate:LastModifiedDate}'

For Secrets Manager, preserve version metadata and rely on native secret version history for rollback whenever possible:

aws secretsmanager list-secrets \
  --filters Key=name,Values="<namespace>/<service>/" \
  --query 'SecretList[].{Name:Name,ARN:ARN,LastChangedDate:LastChangedDate}'

If you must export secret values for a regulated change window, put the export in the customer's approved encrypted vault and delete any local working files immediately after validation.

Step 4 - Snapshot Postgres-backed config

Before a release or risky config migration, take a database snapshot using your deployment's backup mechanism. The supported options are deployment-specific: RDS/Aurora snapshots, pg_dump, pg_basebackup plus WAL, or volume snapshots.

For a small logical safety snapshot on a self-managed Postgres host:

cd /opt/services/database
docker compose exec -T voiceai-postgres \
  pg_dump -Fc -U <user> <db> > /backup/pre-change-$(date -u +%Y%m%dT%H%M%SZ).dump

The Ops Tasker also runs scheduled Postgres backups when S3_BUCKET / S3_PREFIX are configured. Verify the latest backup exists before relying on it.

Verify scheduled backup and rehearse recovery

Tasker's BACKUP_DATABASE job streams a PostgreSQL custom-format .dump archive to S3. Its default schedule is every six hours; the job payload can override the destination and retention. The default prefix is database-dumps and retention is 30 days unless overridden. Check the actual cron schedule, job result, and Ops configuration before relying on these defaults.

1. Check the latest job is successful and record its completion time, s3Path, and retention. A failed dump/upload can leave an object, so object existence by itself is insufficient evidence.
2. Check the recorded object's metadata and download the selected archive to an approved protected backup location when validating recovery:

aws s3api head-object --bucket <backup-bucket> --key <backup-key>
aws s3 cp s3://<backup-bucket>/<backup-key> /backup/selected.dump
pg_restore --list /backup/selected.dump > /backup/selected.contents.txt

Use a PostgreSQL client compatible with the archive. Listing verifies readability, not full recoverability: restore into an isolated test database, verify expected schema/data and application access, and record elapsed recovery time before treating the backup as tested.

3. Record acceptable RPO/RTO and compare them with actual successful-backup age and rehearsal time. A six-hour schedule is not a guarantee of at most six hours' data loss if jobs fail.
4. Retain configuration/secret versions and required external object data separately. A database dump does not include deployment bundles, recording/media objects, or the contents of Secrets Manager. TelWeb configuration export also does not replace a database backup.

Restore rehearsal acceptance

Use an isolated environment with its own database and runtime credentials. Before starting restored application services, disable or redirect outbound email, calls, scheduled jobs and provider integrations so restored data cannot trigger production actions.

Rehearsal stage	Required evidence
Retrieve	Exact backup object/snapshot identity, timestamp, size, and successful retrieval using the recovery operator's access.
Restore	Successful restore exit/status, destination identity, archive/client compatibility, and any warnings investigated.
Check data	Expected schema and selected known teams, users, apps, flows and conversation records from the backup point.
Check application	The intended recovery images start against the restored database and an authorized account can inspect expected configuration.
Check external data	Referenced recordings/media and secret/config versions needed for the scenario are still recoverable.
Measure	Time to retrieve, restore, reconfigure, start and verify; compare the total with RTO.

Example: an incident begins at 12:00 UTC and the newest usable backup completed at 06:00 UTC. Restoring it may lose writes between the backup's recovery point and the incident, even if a 12:00 backup job was scheduled. Record the actual recovery point provided by the backup mechanism; job start/completion times do not by themselves prove the exact database snapshot time.

If backup verification fails

Symptom	Next action
Job remains PENDING	Check Tasker availability and queue processing. Do not count the queued job as a backup.
Job is FAILED or DEAD	Read Last Error and attempts; resolve the database, storage, credentials, or connectivity failure before retrying through the supported job procedure.
S3 object exists but job failed	Treat the object as unverified; investigate dump/upload failure and select another validated recovery point.
pg_restore --list cannot read the archive	Check retrieval, archive format and client compatibility. A managed snapshot or plain SQL dump is not a custom-format archive.
Restore completes with errors	Keep application consumers stopped; resolve ownership/extensions/schema errors and repeat in an empty isolated destination before accepting the backup.
Previous secret/config version is unavailable	The rollback plan is incomplete. Recover the required version from its protected store or obtain a compatible alternative before proceeding.

Step 5 - Apply the change

Use the narrowest operation:

# New image
./update.sh --ecr-tag <tag>

# New generated config bundle
./update.sh --config-ref <ref>

# SSM / Secrets Manager value changed
./update.sh --restart-only

Use Restart matrix to pick the affected services and restart order.

Rollback options

Choose a schema-compatible target

Before reverting images, check whether the change applied a database migration or changed shared configuration/job contracts. An image rollback does not undo a migration. Use the release-specific rollback instructions and component dependency matrix. If the old images cannot use the new schema, plan a coordinated database restore and the resulting lost-write interval before restarting them.

Roll back a bootstrap or image/config ref change

Restore the previous host bootstrap file, then run init.sh:

cp /opt/deployment/backups/env/.env.<timestamp> /opt/deployment/.env
cd /opt/services/<service>
./init.sh

The image tag and config ref from that .env are fetched and started again.

Roll back an SSM value

Put the previous value back, then restart consumers:

aws ssm put-parameter \
  --name "/<namespace>/<service>/<VAR_NAME>" \
  --value "<previous-value>" \
  --type String \
  --overwrite

cd /opt/services/<service>
./update.sh --restart-only

Roll back a Secrets Manager value

Identify the current and intended previous version IDs with aws secretsmanager list-secret-version-ids --secret-id "<namespace>/<service>/secrets". Promote the selected previous complete secret version, then restart consumers. This rolls back every field in that version; do not replace a multi-key secret with a one-key JSON object:

aws secretsmanager update-secret-version-stage \
  --secret-id "<namespace>/<service>/secrets" \
  --version-stage AWSCURRENT \
  --move-to-version-id <previous-version-id> \
  --remove-from-version-id <current-version-id>

cd /opt/services/<service>
./update.sh --restart-only

Keep shared secrets coordinated. For example, DATABASE_URL, Redis passwords, JWT/session secrets, and TLS trust bundles may be consumed by multiple services.

Roll back Postgres-backed configuration

Only restore Postgres when the bad change lives in database-backed configuration and cannot be corrected from TelWeb or a targeted SQL fix.

If you need a full database restore, follow Restore Postgres. It is disruptive and can lose writes after the restore point.

Verification

• docker compose ps is healthy on each touched host.
• fetch-env.sh --format export shows the expected value count and no validation errors.
• TelWeb loads and a SUPERUSER can inspect the changed settings.
• A test call confirms SIP routing, provider config, and flow behavior still work.
• SigNoz shows fresh telemetry for every restarted service.

See also

• Restart matrix
• init.sh and update.sh
• Environment resolution
• Restore Postgres