Skip to main content
Version: 0.9.14

Configuration backup and rollback

Delphi does not store "the complete platform configuration" in one file. A working deployment is reconstructed from several sources at boot time:

  • Bootstrap env on each host: /opt/deployment/.env.
  • Generated deployment bundle from S3: Compose files, vars.yaml, service scripts, collector config, Caddyfiles.
  • AWS SSM Parameter Store for non-sensitive runtime values.
  • AWS Secrets Manager for sensitive runtime values and TLS material.
  • Postgres for TelWeb-managed platform, team, app, flow, SIP trunk, number, provider, cron, notification, and user configuration.
  • Redis for runtime state only. Redis should rebuild from Postgres and service heartbeats; do not treat it as the source of truth for rollback.

Use this recipe before upgrades, certificate rotations, service-wide config changes, or any change where you need a known-good rollback point.

STOP - know the rollback scope

A config rollback can undo tenant-visible changes. If users changed flows, numbers, providers, or platform settings after your backup point, a Postgres restore will lose those writes. Prefer rolling back only the changed config source when possible.

Step 1 - Record the running release​

On every service host you plan to touch:

cd /opt/services/<service>
docker compose ps
docker compose images
grep -E '^(ECR_TAG|CONFIG_REF|CONFIG_BUCKET|NAMESPACE|ENVIRONMENT)=' /opt/deployment/.env

Record the current ECR_TAG, CONFIG_REF, NAMESPACE, and service image tags. update.sh automatically snapshots /opt/deployment/.env before it changes it, but taking a ticket-attached note before the work makes the rollback target unambiguous.

Step 2 - Capture generated config​

Generated service config is versioned by CONFIG_REF. Before changing it, confirm which ref each host resolves:

grep -E '^(CONFIG_REF|CONFIG_BUCKET|NAMESPACE)=' /opt/deployment/.env

If your S3 bucket has versioning enabled, keep the object version IDs for the current bundle. If it does not, copy the current bundle to a dated prefix before uploading a replacement.

Step 3 - Snapshot SSM and Secrets Manager​

For SSM, export the parameter names, versions, and last-modified timestamps. Store values only in your approved secure evidence store.

aws ssm get-parameters-by-path \
  --path "/<namespace>/<service>/" \
  --recursive \
  --query 'Parameters[].{Name:Name,Version:Version,LastModifiedDate:LastModifiedDate}'

For Secrets Manager, preserve version metadata and rely on native secret version history for rollback whenever possible:

aws secretsmanager list-secrets \
  --filters Key=name,Values="<namespace>/<service>/" \
  --query 'SecretList[].{Name:Name,ARN:ARN,LastChangedDate:LastChangedDate}'

If you must export secret values for a regulated change window, put the export in the customer's approved encrypted vault and delete any local working files immediately after validation.

Step 4 - Snapshot Postgres-backed config​

Before a release or risky config migration, take a database snapshot using your deployment's backup mechanism. The supported options are deployment-specific: RDS/Aurora snapshots, pg_dump, pg_basebackup plus WAL, or volume snapshots.

For a small logical safety snapshot on a self-managed Postgres host:

cd /opt/services/database
docker compose exec voiceai-postgres \
  pg_dump -Fc -U <user> <db> > /backup/pre-change-$(date -u +%Y%m%dT%H%M%SZ).dump

The Ops Tasker also runs scheduled Postgres backups when S3_BUCKET / S3_PREFIX are configured. Verify the latest backup exists before relying on it.

Step 5 - Apply the change​

Use the narrowest operation:

# New image
./update.sh --ecr-tag <tag>

# New generated config bundle
./update.sh --config-ref <ref>

# SSM / Secrets Manager value changed
./update.sh --restart-only

Use Restart matrix to pick the affected services and restart order.

Rollback options​

Roll back a bootstrap or image/config ref change​

Restore the previous host bootstrap file, then run init.sh:

cp /opt/deployment/backups/env/.env.<timestamp> /opt/deployment/.env
cd /opt/services/<service>
./init.sh

The image tag and config ref from that .env are fetched and started again.

Roll back an SSM value​

Put the previous value back, then restart consumers:

aws ssm put-parameter \
  --name "/<namespace>/<service>/<VAR_NAME>" \
  --value "<previous-value>" \
  --type String \
  --overwrite

cd /opt/services/<service>
./update.sh --restart-only

Roll back a Secrets Manager value​

Promote the previous secret version or write the previous JSON payload back, then restart consumers:

aws secretsmanager put-secret-value \
  --secret-id "<namespace>/<service>/secrets" \
  --secret-string '{"KEY":"previous-value"}'

cd /opt/services/<service>
./update.sh --restart-only

Keep shared secrets coordinated. For example, DATABASE_URL, Redis passwords, JWT/session secrets, and TLS trust bundles may be consumed by multiple services.

Roll back Postgres-backed configuration​

Only restore Postgres when the bad change lives in database-backed configuration and cannot be corrected from TelWeb or a targeted SQL fix.

If you need a full database restore, follow Restore Postgres. It is disruptive and can lose writes after the restore point.

Verification​

  • docker compose ps is healthy on each touched host.
  • fetch-env.sh --format export shows the expected value count and no validation errors.
  • TelWeb loads and a SUPERUSER can inspect the changed settings.
  • A test call confirms SIP routing, provider config, and flow behavior still work.
  • SigNoz shows fresh telemetry for every restarted service.

See also​