Choose and verify a configuration recovery

Use this procedure when a configuration change must be reversed or configuration has been lost. Choose the smallest recovery scope that contains the problem. Restoring a database to recover one setting can discard unrelated writes made since that backup.

Availability: Configuration snapshot, backup creation, restore and import apply require Super Admin. Super Viewer accounts can use the read/export/import-preview API operations; read access does not authorize restore. A Team Admin role alone is insufficient. Host, database, storage and AWS recovery require their separate operator permissions.

Choose the correct recovery asset

What you need to recover	Appropriate asset	What it does not recover
A supported platform configuration row/domain	Configuration version or configuration backup/export	All application data, cloud secrets, deployment files or runtime cache state
An earlier application flow	Flow Version History and reviewed publication	Other platform settings or host configuration; see the application journey
Image/configuration deployment	Compatible previous image set, bootstrap and service bundle references	A reversed database migration or recovered application writes
Runtime environment/TLS credentials	Recoverable SSM/Secrets Manager versions and required certificate/key material	Provider-side validity of a revoked credential or expired certificate
Database/schema	Verified database backup or managed snapshot	External secret values and recording/media objects
Recording/media objects	Storage versioning, backup or snapshot for those objects	Separately lost database metadata

A configuration backup stored in Delphi is a database row. If that database is lost, that row is not an independent recovery copy. Keep required exports and database backups in approved protected storage and verify that the recovery operator can retrieve them.

Snapshot or backup: which asset fits

A version snapshot covers one row. It is the unit of that row's history: you can read its numbered list, unfold any version to compare it, and restore one. It records only the fields of that single row.

A backup covers the whole configuration: one capture of every supported row of all eight domains, stored as a database row and restorable per domain or as a whole. It is not a version list you can browse; the row's History action does not show backups.

Neither replaces a database backup. A configuration backup lives in the same database it describes.

If you need to …	Use	Keep in mind
Reverse one row's change	That row's History	Restoring records the state it overwrites, so the rollback is itself reversible.
Record the state of many rows before a change	A backup, then restore the domain	The backup is platform-wide; you choose the domain when you restore.
Keep configuration outside the database	Download the backup or export	The file contains authentication values such as authPass; treat it as sensitive.
Move configuration to another environment	Export here, import there	Referenced teams, secrets and TLS material must exist there.
Recover a whole domain, including rows that were deleted	A stored backup	Restoring a domain deletes live rows the backup does not contain.

Restoring a version or applying a backup also writes a version for the state it overwrites, so a restore shows up in the affected row's history and can be undone from there. A restore that would overwrite a row with the state already recorded as its latest version records nothing new.

What a platform configuration bundle contains

The backup and current-configuration export use the same bundle structure. The export contains these eight domains:

Domain	Stored in the bundle	Nested or related records	Described in
SIP trunks	Stored trunk fields, including authentication values and TLS-material references	No nested records. TLS material and secrets are references only; the referenced records are not in the bundle.	Configure an inbound SIP trunk
Egress trunks	Trunk fields and the base-number IDs that use the trunk	The base-number association is restored with the trunk; route membership is not part of the bundle.	Manage outbound trunks and pools
Base numbers	Number configuration with its team, header-manipulation and egress-trunk associations	Extension and endpoint records are separate; they are not part of the bundle.	Manage base numbers
Platform settings	Every setting key known to the selected release; the stored value or its effective default	The row's description and last editor are not part of the bundle.	Platform Settings catalogue
Cron schedules	Stored scheduled-job configuration	Next run, last run and last result are live state, not stored configuration.	Change a recurring schedule deliberately
Header manipulation	Configuration fields with their nested rules and conditions	Rules and conditions are replaced wholesale; assignments are held by the base number, not by the configuration.	Author header rules
Server groups	Stored group configuration: role, capacity, load-balancing and scaling settings	Server membership, team assignment and separate legacy load-balancing rows are not part of the bundle.	Configure a server group
Notification types	Stored notification-type configuration: severity, recipients, templates and interval	No nested records.	Notifications

The bundle does not include the platform-secret registry, AWS secret values, teams/users, application flows, conversations, full database history, deployment bundles or object-store contents. An ID referencing one of those resources does not recreate the resource. For example, restoring a trunk's TLS-material reference does not restore the referenced platform-secret registry entry or its certificate/key value.

Treat the export as sensitive. Excluding AWS secrets does not make it secret-free: SIP trunk records include the stored authPass field. Keep bundles and snapshots restricted; do not attach an unreviewed export to an ordinary support ticket or paste it into a public example.

Rules and conditions in a header configuration

A header configuration owns an ordered list of rules, and each rule owns the match conditions that decide when it applies. Rule fields and their substitution values are described under the rule fields; the matching restrictions are described under conditions and order.

A restore replaces that list wholesale: the configuration's existing rule and condition rows are removed and recreated from the stored copy. Two consequences follow.

• The internal record IDs of the rules and conditions change. The rule ID itself (ruleId) is preserved, and that is the identity the synchronized and exported rule set uses, so runtime rule identity survives a restore. A comparison of two snapshots therefore ignores the regenerated IDs and reports a change only when a rule field or a condition value actually differs.
• Deleting a live-only configuration during exact replacement removes its rules, its conditions and its base-number assignments, because those records belong to the configuration. The list's ordinary Delete action behaves differently: it refuses to delete a configuration while a base number still uses it. Review the expanded category list before a restore that can delete a configuration.

Associations that belong to the other record

Some relationships are stored on the opposite side, and this decides which part of a restore re-establishes them:

Relationship	Stored with	Restored by
Base number ↔ header configuration	The base number	Restoring the base number; a configuration restore does not assign it to numbers.
Base number ↔ team	The base number	Restoring the base number.
Base number ↔ egress trunk	Both sides	Restoring either side rewrites its own copy of the association.
Header configuration ↔ rules and conditions	The configuration	Restoring the configuration.
Trunk → TLS material or platform secret	A reference on the trunk	Nothing: the referenced record must exist in the target.
Server group → voice/API servers and teams	The servers and teams	Nothing: reassign them separately.

References that can block an exact replacement

Deleting a live-only row is part of exact replacement. Where another record requires the row, the database refuses the deletion and the whole category transaction rolls back rather than committing a partial replacement. Where the reference is optional, the deletion succeeds and the reference is cleared to nothing, which no error reports. Known cases:

• A route that still references a base number the backup does not contain blocks that base-number restore. Review the routes with outbound selection before restoring base numbers.
• A server group that still owns API server records cannot be deleted; review the group and its servers as described under Remove a group safely.
• A referenced record that is missing entirely (team, platform secret, TLS material, extension) is not a deletion problem: it is a missing dependency that the restore cannot create.

Server-group recovery boundary

Server-group snapshots contain the group's configuration: enabled state, role, capacity/load-balancing choices and scaling settings. They do not restore API/voice server membership, team-to-group assignments or separate legacy load-balancing configuration rows. Recreating a missing group does not recreate its fleet or reassign teams. Verify those relationships separately after recovery; restoring scalar settings is not a server provisioning operation.

Two further details matter when a restore is used to re-create a group:

• A group whose stored role is the SIP edge role (telpro: Kamailio and RTPEngine) is never deleted by exact replacement, even when the backup does not contain it. The exemption covers deletion only: a telpro group that the backup does contain still has its stored configuration applied, and one the backup does not contain keeps the configuration the environment has.
• Restoring a group is a database write to that group's row. It does not start, stop or resize a server, and it does not re-point traffic. Follow Deploy, verify and update Delphi for the activation step and confirm the resulting runtime group.

Exact category replacement may need to delete a live group absent from the backup. If that group still owns API server records, the database's Restrict relationship rejects deletion and the bulk restore/import transaction rolls back. Do not delete server records just to bypass the error. Review the intended group and server assignments with the infrastructure operator and choose an appropriate recovery scope.

Backup capture is not a frozen system state

Backup creation and live export read categories through multiple database queries, including later reads of nested associations. They are not wrapped in one consistent read transaction. Concurrent edits can therefore yield a bundle whose categories or relationships reflect different moments. Its exportedAt timestamp labels the export; it is not proof that every row was captured atomically at that instant.

For a recovery point that must preserve related configuration coherently, coordinate a pause in configuration editing, capture the bundle, and review its dependency references and intended contents. A transaction around applying a restore does not retroactively make the source backup a consistent snapshot. Use a verified database backup when the recovery requirement actually concerns the whole database.

Record and restore one kind of entity

There is no separate asset for one kind of entity: you use the same backup, version or bundle and select a smaller scope. Choose by what you are actually replacing.

Scope you want	Where to do it
One row	The row's History action: restore a stored version of that row only.
One domain	Config Backups → Restore on the backup row, then choose that domain in the confirmation. Only that domain is replaced.
One domain from a file	Import, with only that category selected.
All eight domains	Config Backups → Restore all on the backup row, or an import with every category selected.

Before applying a domain-level operation, inspect its scope:

1. Select the domain count on the backup row to expand it into its individual rows. Rows marked Changed show the live and the restored value per field, and No change rows would not be written. Rows marked Missing live exist in the backup but not in the live configuration, and Only in live rows would be deleted by the exact replacement. Restoring a domain is therefore not “apply the differences to the rows that exist”.
2. For an import, read the preview counts as well; the dialog shows unchanged, added and removed counts but not the affected IDs, so download and compare the bundle when the decision depends on which rows are involved.
3. Confirm that the dependencies of the rows you are about to write exist in the target environment: teams, header configurations, egress trunks, server groups, platform secrets and TLS material.

There is no per-domain export. A backup or export file always contains all eight domains; the choice of category happens when you apply it, not when you capture it. If you must keep only one domain outside the database, export the bundle and store the file as it is, or extract the category you need from the JSON with the same care you would apply to the whole file.

Understand what Restore and Import change

Operation	Database effect
Restore one configuration version	Replaces the chosen row's supported fields, or recreates a missing row. Captures an existing live row before overwriting it.
Restore one backup domain	Restores that category to the bundle state, including deletion of live rows absent from the bundle.
Restore all backup domains	Restores all eight supported categories, including deletion of live-only rows.
Import selected categories	Replaces only the selected categories with their bundle state, including deletion of live-only rows. Unselected categories are not selected for replacement.

Deletion can remove relationships as well as the row itself. A category-only restore can fail if its referenced resources are missing or incompatible. Selecting more categories is not automatically the right repair: first establish the intended dependency set.

An absent category is not a request to leave it alone. Missing category fields in an imported bundle are parsed as empty lists or an empty settings map. Applying such a selected category can delete all its live rows. This is particularly important for older or manually edited bundles. Inspect the actual category contents before choosing Restore all or selecting a category for import.

Bulk restore/import database changes run in a transaction; a failed row operation causes the transaction to fail rather than committing the other rows as a partial restore. This does not make database, cloud stores and runtime services one atomic recovery operation. Audit writes after a bulk commit can fail separately and be logged; missing audit evidence alone does not prove no configuration changed.

No preserve-extra-rows import switch

The supported import input consists of the bundle and selected categories. An older or hand-written caller sending deleteMissing: false does not disable exact replacement: that field is not in the accepted schema, and the apply path still deletes selected-category live rows absent from the bundle. Do not use it as a safety switch. Use import preview and choose only the intended categories; if preserving live-only rows is required, stop and prepare a reviewed complete desired state rather than assuming merge semantics.

Caveats that change the outcome

• A stored backup is not an independent copy. It is a database row; a lost database takes it with it.
• Capture is not atomic across domains. One bundle can span several moments of concurrent editing.
• Absent categories in a file mean empty. Importing an older or hand-edited bundle can delete a whole selected domain.
• Exact replacement deletes. Within the selected domains, live rows the backup does not contain are removed, together with the records that belong to them: header rules and conditions, base-number assignments and route memberships. SIP edge (telpro) server groups are the one exception.
• A reference can block the replacement, or be cleared silently. A route pointing at a base number, or a group whose API server records require it, makes the category transaction roll back; resolve the referencing row first. Optional references, such as a base number's group or an endpoint's base number, are cleared to nothing when the row is deleted, without an error.
• References are not restored. Teams, users, platform secrets, TLS material and extension or endpoint records must exist in the target; an ID does not recreate a record.
• Operational state stays as the environment has it. Trunk maintenance mode, number suspension, and the provider-derived fields of a base number (status, country, external ID, provisioning block) are captured in the snapshot but a restore does not write them, and the comparison ignores them.
• Values are written as stored. An empty string in the live row becomes NULL when that is what the backup holds, and the reverse.
• A platform setting keeps its description and last editor only when the row already exists. They are not part of the bundle, so an existing row keeps whatever the target environment has. A setting key with no stored row in the target is created by the restore with a generated description (Platform <key> settings), not the description the source had.
• A restore is a database write only. Runtime caches, the synchronized rule and routing state and running calls are separate; see the activation section below.
• “No new version” is not proof that nothing happened. A version is recorded only when the row changed, so an unchanged-looking history can also mean the change matched the latest stored state.

Find a row's history

The list-row History action is available on SIP Trunks, Egress Trunks, Base Numbers, Cron schedules and Header Manipulation configurations, and those editors offer Save snapshot once the row exists. It opens that row's version history in a popup over the list. Other categories are included in the configuration bundle/API, but this does not mean each has an identical History button or snapshot control in its editor.

The version list is ordered by descending version number. Snapshot and backup notes accept at most 500 characters. A snapshot is recorded only when it would say something new: capturing the same unchanged state again records no version and the popup names the version that already holds it, while the attempt is still audited. Restoring a version first records the existing live row unless that state is already the latest version, then updates the row, or recreates it when it was missing.

Select a version to unfold its comparison in place, or switch that comparison to the version recorded just before it. The comparison lists changed fields, with nested header rules matched by their rule ID. It reports only values a restore writes: platform-managed fields (row timestamps, scheduled-job run state, trunk maintenance, number suspension and provider-derived state) and regenerated rule or condition IDs are not shown as differences. Comparing against the live row requires that row to exist even when a stored baseline is supplied; a missing-row comparison failure does not imply the snapshot cannot be restored.

1. Find the correct row and verify its identity before opening History.
2. Inspect the stored versions and compare a candidate with the current state before restoring it. Review changed references as well as scalar values.
3. If taking a new checkpoint from one of these editors, save the intended row first, reopen the editor if necessary, then choose Save snapshot. This captures the saved database row, not unsaved form edits, and records nothing when that row already matches the latest stored version.
4. Restore only the intended version after reviewing the confirmation and the permission requirements above. Verify the saved result and follow the activation steps below; the history operation does not establish runtime readiness.

The availability of a History icon is not write authorization. Restore and snapshot capture still require platform-administrator permission, while read-only users may inspect the history where their screen access permits it.

1. Establish a recovery point and expected result

Record the environment, release, affected category/row and the symptom you intend to reverse. Identify the exact snapshot/backup and its source environment. Preserve the current state before overwriting it, including a protected export or relevant configuration snapshot and any required database/cloud/storage recovery point.

Confirm that referenced teams, base numbers, server groups and TLS materials needed by the selected state exist in the target environment. Review related categories together. A bundle being valid JSON does not establish that these dependencies can be restored or that its values are appropriate for another environment.

Agree on a maintenance or traffic-handling plan for routing, capacity or scheduled-job changes. A restore can change live database configuration; it is not a draft workflow. If the requested change is only a flow definition, use the application's draft-and-publish recovery instead.

2. Review the scope before applying it

Open the platform Config Backups area under /admin-settings, or the affected configuration row's History action where available.

• For a single version, compare it with the current row and confirm the exact row and version to restore.
• For an import, load the intended JSON and use Preview. The UI shows Unchanged, Added and Removed counts; it does not list the affected IDs. Compare the downloaded bundle with a protected current configuration export to identify the live-only rows that would be removed. The separately authorized preview API returns ID arrays, but the dialog does not display them.
• Inspect changed field values as well. The import summary compares IDs/counts; matching IDs do not mean matching settings.
• For a stored backup, select a category count on its row to expand that domain into its individual rows and what a restore would change. Each row shows as Changed with the live and the restored value per field, No change, Missing live (the backup holds a row the live configuration no longer has) or Only in live (a row added after the backup was taken). Only values a restore writes are compared, so operational state such as trunk maintenance or number suspension is not listed. Inspect the downloaded payload as well, and do not assume a backup contains categories introduced after it was created.

Select only the intended categories, but include required dependencies when restoring them is part of the approved recovery. Stop if the preview contains unexpected live-only rows, empty selected categories, another environment's IDs or missing secret references.

3. Apply once and establish the database outcome

Review the confirmation and apply the chosen operation. Wait for the result; avoid concurrent configuration editing during the recovery.

On success, re-open the affected rows and verify both values and associations. Check deletion results as well as applied rows. Record the snapshot/backup identity, actor and completion time.

If the operation fails, inspect the reported validation, dependency or transaction error and the relevant audit/runtime evidence. Do not repeatedly apply the same payload hoping that missing dependencies resolve themselves. If the request times out or the result is uncertain, inspect the current database-backed configuration before retrying; a client error alone does not establish whether a commit happened.

Inspect configuration recovery records

Ask an authorized operator to correlate the actor, entity, time and selected backup/version in the audit store using these operation-specific records. Access Logs is a filtered feed and does not include every configuration category. An absent snapshot or backup row there is not evidence that no audit record exists. Follow the access-feed limits. The action and category are separate fields:

Operation	Category / action	Useful details
Capture a row snapshot	CONFIG_VERSION / UPDATE	captured is true only when a new version was recorded; versionId, version, configCategory and the optional note name the recorded version or the latest one that already held the state.
Create a stored backup	CONFIG_BACKUP / CREATE	backup.id and the optional backup note.
Restore a row version	The row's domain / UPDATE or CREATE	restoredFrom, restoredFromVersion, configCategory, old, new and recreated, plus versionRecorded / rollbackVersionId telling whether a rollback point was recorded. A missing live row produces CREATE.
Apply a backup/import row	The row's domain / UPDATE or CREATE	restoredFromBackup, configCategory, before/after values where applicable, and recreated for a recreated row.
Delete a live-only row during exact replacement	The row's domain / DELETE	old, restoredFromBackup, configCategory and exactState: true.
Bulk restore/import transaction failure	CONFIG_RESTORE_FAILURE / UPDATE	error, backupId, rolledBack: true and a category when the failure is category-scoped.

restoredFromBackup is an operation correlation field, not always the ID of a stored backup: imports use the import operation's marker. Do not interpret a failure record as “one row skipped while the others committed.” The bulk operation rolls back on that failure. Conversely, deferred audit writes run after a successful bulk commit and can fail separately; use the saved configuration and runtime verification to establish the outcome. Review audit record structure before exporting details, which can contain sensitive configuration.

Bulk backup restore/import transactions take exclusive locks on their selected configuration tables and use serializable isolation. Full backup restore acquires the supported tables in a fixed order. This does not describe the separate single-row version restore transaction. Concurrent edits can block or fail; no fixed completion time or universal deadlock-free guarantee follows. Coordinate the recovery window and inspect the result instead of repeatedly retrying a timeout.

4. Activate and verify the restored behavior

A successful database restore is not proof that routing, runtime caches, capacity or already-running services have adopted the restored state. The configuration restore path does not perform a general synchronization/restart of every consumer.

Identify the affected consumers and follow the applicable activation procedure from Deploy, verify and update Delphi. Do not restart the entire deployment merely because one row changed.

Verify the result that motivated recovery:

• Routing: the controlled call reaches the intended app/entry point and uses the expected transfer or header behavior.
• Capacity/server-group configuration: the observed runtime group and routing state match the restored policy.
• Scheduling/notifications: the intended schedule/type configuration is visible and its next approved execution behaves as expected.
• TLS/authentication: the consumer can use the referenced material and establish the expected connection; a registry ID alone is insufficient.

Use fresh conversation and log evidence where relevant. Keep traffic restrictions in place until the affected acceptance checks pass. Record any remaining runtime mismatch rather than reporting the configuration as recovered from a database success message alone.

Capture a recoverable deployment change

Before changing a host's image/configuration references, runtime parameters or secrets, identify the service hosts, the affected consumers and the acceptable lost-write interval. Keep the previous release and its database/schema requirements available throughout the recovery window. A list of names or versions is an inventory, not a recoverable backup.

On each affected service host, inspect the running state and the selected bootstrap references. Replace <service> with the service directory from that host's deployment inventory:

cd /opt/services/<service>
docker compose ps
docker compose images
grep -E '^(ECR_TAG|CONFIG_REF|CONFIG_BUCKET|NAMESPACE|ENVIRONMENT)=' /opt/deployment/.env

Record the output in the protected change record, including the actual running image identities. The selected tag/ref and running containers can differ; do not infer that a saved bootstrap value has already been applied. Do not attach the complete .env, resolved environment or secret values to a support ticket.

Capture each recovery asset deliberately:

Asset	What to retain and verify
Host bootstrap	A protected copy of the intended host's bootstrap configuration, its timestamp and host/environment identity. The copy can contain sensitive values.
Generated configuration	The exact bucket/ref and complete bundle needed by the selected recovery release, including any immutable generation/manifest identity provided by that deployment. Verify it remains retrievable; an S3 object name alone does not prove the same bytes will remain there.
Runtime parameters	Exact names, current versions and the recoverable previous values in the approved configuration store. Confirm every affected service consumes the parameter being changed.
Runtime secrets/TLS	Exact secret identity and complete previous version, plus required external certificate/key material. Confirm that the version remains available and its provider/certificate validity has not been revoked.
Database and objects	The independently retrievable database recovery point and required recording/media backups. Use the database rehearsal below; a configuration export does not include these assets.

update.sh creates /opt/deployment/backups/env/.env.<timestamp> when it changes bootstrap variables. A restart-only invocation without a bootstrap-variable change does not establish a new bootstrap snapshot. That local copy neither backs up SSM/Secrets Manager contents nor preserves a remotely replaced bundle; it is also lost with the host unless protected separately. Inspect the candidate backup before relying on it rather than choosing a file solely by its timestamp.

Restore only the changed layer

1. Confirm the recovery target, service owner and affected consumers. Pause automation that could replace hosts or reapply a newer configuration during the operation. Arrange the applicable maintenance/drain procedure before a restart; neither of the commands below promises uninterrupted calls.
2. For a wrong image or generated bundle, select a compatible pair of existing references and verify the corresponding images, bundle and schema compatibility before applying it. On the affected host:

cd /opt/services/<service>
./update.sh --ecr-tag <approved-image-tag> --config-ref <approved-config-ref>

Review the script's displayed changes and confirmation. This updates the selected bootstrap references and delegates to that service's init.sh; it is not an atomic cross-service rollback or a database downgrade. Do not use a floating tag/ref as proof that the earlier bytes were restored. If the problem is another bootstrap field, have the operator restore the reviewed host-specific configuration; do not copy an entire backup from another host or environment.

3. For a runtime parameter, restore the intended value through its owning configuration store and preserve the correct parameter type, account, region and path. Replacing one SSM parameter does not restore a database-backed setting or update an already-running process.
4. For a secret, identify the current and intended previous version IDs before promoting the previous complete version. Check every field/consumer affected by that version, particularly shared database, session and TLS material. Do not overwrite a multi-key JSON secret with a one-key document or assume an old provider credential is still accepted.
5. After the store change, use the affected service's deployment procedure to reread its configuration. Where the host's reviewed service script supports this mode:

cd /opt/services/<service>
./update.sh --restart-only

This uses cached images and delegates restart behavior to the service initializer; it does not select an earlier image version. Keep presenting and trusting TLS services coordinated and verify the resolved non-secret settings, connection outcome and actual running image identity. A successful script exit alone is not end-to-end acceptance.

6. Repeat the controlled interaction that failed, inspect service health and record the recovery result. Restore normal traffic and automation only after the agreed checks pass. If the previous bundle/secret version is missing or the schema is incompatible, stop that rollback path and use a separately reviewed recovery target; repeated restarts cannot recreate a missing asset.

Recovering cloud secrets, the database or stored objects

For cloud-secret recovery, retain the needed complete secret version, not only its name or a list of keys. A JSON secret may contain several required values; restoring one field with a replacement document that omits the others can break other consumers. Coordinate consumers and presenting/trusting TLS services, then verify the recovered behavior. A credential revoked at its provider cannot be made valid merely by restoring its previous stored bytes.

For database recovery, establish the usable recovery point and lost-write interval with the service owner. Verify the backup job's success and archive/snapshot usability, then rehearse recovery in an isolated environment. Disable or redirect outbound calls, notifications, jobs and provider integrations before starting restored application services in that rehearsal. A successful archive listing is not a completed restore test.

Recover external media/recording objects separately and verify retrieval through Delphi with the corresponding database metadata. Follow the database backup and restore procedure below for the database mechanism in your deployment. Do not downgrade images to reverse a schema migration without confirming compatibility.

Stop and involve the responsible operator when the required secret version is unavailable, a selected category is unexpectedly empty, referenced resources cannot be recovered, the schema/release combination is uncertain, or runtime state still differs after activation. Preserve evidence and the current recovery point before attempting a broader rollback.

Database backup and restore

A database dump is distinct from a Delphi configuration export, runtime Redis state, external secret values and recording/media objects. Before recovery, identify which asset was lost and whether the deployment uses the bundled PostgreSQL service, a managed database snapshot/PITR mechanism, or Tasker's logical dumps. The following describes the bundled dump handler and a bounded restore procedure, not a universal managed-database command.

Verify what the backup job produced

The default database-backup schedule is enabled with UTC cron 0 */6 * * * (every six hours), retentionDays: 30 and three maximum attempts. Check the saved schedule and payload in your deployment, because operators can change both.

BACKUP_DATABASE runs pg_dump in custom format (-F c) and streams its output to S3-compatible storage. The Tasker image includes the PostgreSQL 17 client. The handler parses Tasker's startup DATABASE_URL into host/port/user/password/database; if absent it uses DB_* configuration. This is separate from Prisma's runtime pool-refresh path. A later application credential refresh does not establish that this external backup process uses a rotated password.

Backup setting or result	Meaning
s3Bucket / S3_BUCKET	Payload override, otherwise configured bucket; missing bucket fails the job. Confirm the intended deployment's storage before starting.
s3Prefix / S3_PREFIX	Payload override, otherwise configured prefix; application default database-dumps. Retention cleanup scans this prefix, so do not share it casually across unrelated backup policies.
retentionDays	Payload override; handler default 30 days. This is not an enforced platform-wide 14-day policy or proof the cron runs every 12 hours. Check actual cron schedules.
s3Path, filename, duration	Returned after upload completes and pg_dump reports successful exit. The filename ends .dump; the timestamp/extension alone does not prove a usable archive.
deletedExpired	Cleanup attempts old .dump objects by LastModified only after a successful new backup. List/deletion failures warn and may still leave the job successful. Bucket lifecycle, versioning and retained noncurrent versions are separate.

The handler passes -U, -h, -p, database name and PGPASSWORD to pg_dump; it does not translate the Node helper's DATABASE_SSL_MODE or connection-URL TLS parameters into libpq settings. Verify the actual backup connection's TLS/identity and permissions separately. A healthy application connection is not evidence that the backup client has equivalent trust or complete data access.

An upload can finish before the dump's nonzero exit is detected; the handler does not remove that already uploaded object. Treat failed-job objects as unverified. The backup is not validated by a restore test and does not include external cloud secrets, roles supplied outside the dumped database, Redis routing state or object storage. The optional backup-bucket Terraform module enables versioning/default KMS encryption and lifecycle rules, but the handler accepts other destinations: do not promise these properties for every configured bucket or infer immutability/Object Lock.

Recover a selected database without confusing it with config restore

1. Establish the required recovery point, affected data and schema-compatible release. Preserve the current damaged state when possible and record how much later data will be lost. Prefer a reviewed targeted recovery when the problem is limited to one record/team; do not roll back the whole database to repair an unrelated runtime error.
2. Verify that the selected backup can be retrieved and that its format, database identity and PostgreSQL/client versions are suitable. Perform the restoration drill on an isolated replacement first where possible. Object existence, a successful cron status and HTTP 200 from configuration restore are not database recoverability tests.
3. Quiesce all writers, including application services, workers, scheduled jobs, migrations and automatic scaling/restarts. Stop new admission and handle active calls through the deployment's approved procedure. Delphi has no standalone drain-without-restart operation, and stopping only Web is not sufficient.
4. Restore through the deployment's chosen mechanism. Managed snapshot/PITR endpoint replacement must follow the database operator's procedure. For the bundled service, voiceai-postgres is the actual Compose service name; restore only after confirming the host, database and archive path.

The following is a destructive example, not an instruction to run during ordinary troubleshooting. It streams an already verified host-side custom-format archive into the bundled container; no /backup container mount is assumed:

cd /opt/services/database
docker compose exec -T voiceai-postgres \
  pg_restore --exit-on-error --clean --if-exists --no-owner \
  -U <restore-role> -d <target-database> < /approved/path/verified.dump

--clean --if-exists removes conflicting archive objects; it does not guarantee removal of objects absent from that archive. For an exact replacement, use a separately prepared empty target with reviewed ownership/grants. --no-owner does not recreate the intended role model automatically. Check the restore exit status and remaining errors before permitting writers; this example is not an all-or-nothing database rollback. See the PostgreSQL 17 pg_restore reference for these flags and the separate single-transaction option.

5. Reconcile schema and setup state with the selected recovery release using bootstrap and migration credentials. A restore also restores migration/setup registry rows; blindly running the newest migration image can reapply the change being reversed.

6. Reconcile the runtime endpoint/credential source, external secrets, storage objects and Redis prepared flows/routing separately. Confirm that every restarted consumer connects to the intended restored database identity and endpoint, using the intended credential source; a healthy connection to the old database is not a successful recovery. A changed endpoint or restored old secret reference needs the corresponding runtime/bootstrap update; the database cannot restore revoked credentials or absent media files. Use the other recovery sections for configuration-specific restore effects.

7. Start the intended services through the deployment procedure and verify migrations, application reads/writes, team access, background jobs and a controlled application/call scenario. Inspect service health and logs for database, permission and Redis mismatches. Keep maintenance in effect if restored data/schema or dependent assets remain uncertain.

8. Notify affected tenant administrators of the selected recovery point, the recovery window and the known or possible lost-write interval. Describe which later conversations, settings changes or other writes may be absent, and record any uncertainty that still needs reconciliation.

Related

• Deploy, verify and update Delphi: to choose which services must adopt a restored configuration.
• Configure and rotate database credentials: to run the matching migration release after a database restore.
• Diagnose background jobs: to check the database backup job and restored schedules.
• Author header rules: to understand the rules, conditions and assignments a header-configuration restore rewrites.
• Manage outbound trunks and pools: to resolve the route references that can block a base-number restore.
• Configure a server group: to reassign membership after restoring group settings.
• Check service health: to confirm that every consumer is healthy before you return traffic.