Skip to main content
Version: Next — legacy draft snapshot

Restore Postgres

A Postgres restore is the most disruptive operation an operator can perform on a Delphi deployment. It stops every service that depends on the database and replaces the schema and data with whatever the chosen backup contains. Read this page top to bottom before running anything.

When this is the right answer​

  • Data corruption confirmed at the SQL level, not just a misbehaving feature.
  • A migration left the schema in an unrecoverable state and a backup is fresher than the damage.
  • A destructive admin action (mass delete) needs to be reverted and no application-level rollback exists.

When this is the wrong answer​

  • Application-level issue with a workaround — fix it in the app.
  • Single tenant or single record problem — extract and re-insert from a logical backup, don't roll the whole instance back.
  • You aren't 100% sure when the bad event happened — figure that out first; a restore with the wrong target time makes things worse.
STOP — pick the backup mechanism

Delphi does not prescribe a backup tool. Your deployment uses one of: AWS RDS snapshots, managed Postgres (Aurora, etc.), self-managed pg_basebackup + WAL, pg_dump archives, or volume snapshots. Confirm which one is yours before continuing. The rest of this page is structured around that.

Step 1 — Snapshot the current (bad) state​

Even when "everything is broken", grab a snapshot of the current database first. You may need it to investigate later, or as a fallback if the restore itself fails.

# Self-managed Postgres example
docker compose exec -T voiceai-postgres \
  pg_dump -Fc -U <user> <db> > /backup/pre-restore-$(date -u +%Y%m%dT%H%M%SZ).dump

For RDS / Aurora: take a manual snapshot from the console / CLI before continuing.

Step 2 — Quiesce the platform​

Stop every service that writes to the database:

# On each consumer host (Web, API, Voice, Ops, Media)
cd /opt/services/<service>
docker compose stop

Stop admitting new calls and drain existing calls before stopping Voice. Voice writes conversation state and must not remain a writer during restore. Pause automated restarts/scaling for the maintenance window and confirm all database consumers are quiesced.

Step 3 — Restore​

Use your deployment's mechanism. Two common shapes:

RDS / Aurora point-in-time​

Use the AWS console or CLI's "Restore to point in time" against the appropriate cluster. This creates a new instance by default — you must then either:

  • update DATABASE_URL in Secrets Manager to point at the new endpoint, or
  • promote / rename / DNS-swap so the existing endpoint resolves to the new instance.

Either way, the URL the Delphi services use must end up pointing at the restored database.

Self-managed pg_restore​

# Run on the Database host; the archive path is on this host.
cd /opt/services/database
docker compose exec -T voiceai-postgres \
  pg_restore --exit-on-error --clean --if-exists --no-owner \
  -U <user> -d <db> < /backup/<snapshot>.dump

--clean --if-exists drops conflicting objects before recreating them. It does not guarantee removal of objects absent from the archive; for an exact restore, use an empty replacement database provisioned with the correct owner/permissions. The input redirection streams the host archive into the container; no /backup container mount is assumed. Check the exit status before starting consumers.

Step 4 — Reconcile setup state​

Select the target application and migration images before running migrations. For rollback, use the schema-compatible release recorded with the backup; running the newer release's migrations could reapply the change you are reversing. The setup registry is restored with the database.

If the selected recovery release requires pending migrations, run its migration image on the Ops host:

cd /opt/services/ops
docker compose --profile migration run --rm --no-deps db-migrate

Then reconcile registry steps if needed — see First use (delphi-setup) and delphi-setup CLI.

Step 5 — Bring services back up​

# In each /opt/services/<service>
docker compose up -d

Step 6 — Verify​

  • TelWeb loads, an admin can log in.
  • Place a test call against a known endpoint.
  • Check SigNoz monitoring for fresh telemetry from every service.
  • Spot-check critical tenant data (a known conversation, a known tool, a known billing record).

Step 7 — Communicate​

A database restore loses every write that happened after the restore point. Inform tenant admins of the restore window and what was lost (new conversations, settings changes, etc.).

See also​