Skip to main content
Version: Next — unreleased

Configure and rotate database credentials

Configure database credentials separately for application traffic and schema/setup work. This guide explains the optional runtime refresh used by Delphi's shared TypeScript database client and how to verify a rotation without assuming every database consumer refreshes automatically. For migration and startup resolution, use Bootstrap and migration credentials; for encrypted transport, use database TLS configuration.

Availability: Operators with shell access to the Ops and Database hosts and the right to change deployment configuration. Nothing on this page is configured in the web interface.

Start with the section that matches your task:

Bootstrap and migration credentials​

Use the deployment bootstrap workflow to supply the initial runtime connection and the separate credential used for schema/setup work. A configured runtime secret ARN is not a replacement for a usable initial DATABASE_URL; runtime refresh and host environment generation are different paths.

The Ops bundle gives MIGRATION_DATABASE_URL only to the profile-gated, one-shot db-migrate service. Scaler and Tasker use DATABASE_URL. The migration entrypoint sets its own DATABASE_URL from MIGRATION_DATABASE_URL when present; it can otherwise use an already supplied DATABASE_URL, but the Ops manifest requires the dedicated migration variable. Keep a migration-capable direct database connection separate from restricted application credentials. Variable names and container scoping do not create database roles or grants; verify those privileges independently.

Resolve bootstrap configuration without exposing values​

fetch-env.sh reads the deployment bootstrap and the selected service manifest. For a declared, non-composed variable, the last collected value wins in this order:

  1. Variables explicitly declared source: local from the bootstrap environment.
  2. Namespace-level SSM parameters, then service-level SSM parameters.
  3. Discovered service Secrets Manager JSON keys, then explicitly configured external-secret fields.
  4. A manifest default only when the resolved value is empty.

External secret.arn_from looks up the named variable in the script's environment; make that ARN available through bootstrap. The field supports plaintext or a named top-level JSON key. Secret discovery can encounter multiple objects with the same key; do not rely on duplicate secret keys as a safe ownership/override scheme. Only declared variables enter the final output.

compose.template then builds declared values from the resolved final fields. Only names in compose.encode are percent-encoded; missing referenced fields become empty strings. Required-variable checks establish presence, not a correct URI, permissions, TLS negotiation or database reachability. Use the manifest's declared source/composition deliberately; do not assume every service already has a managed-database composition template simply because the resolver supports it.

Secret-handling boundary: while it runs, fetch-env.sh writes resolved values, including secrets, to temporary files on the host and removes them when it exits. The Ops wrappers also write CA files to disk. Treat the host as sensitive during this step. Do not paste fetched exports or rendered Compose environment into tickets or chats; share variable names, source locations and sanitized errors instead.

Compose a startup URL from a managed secret​

Use this pattern when the database owner supplies a Secrets Manager JSON object containing the runtime user's username and password. Keep migration credentials separate. The variable names beginning APP_DATABASE_ below are illustrative declarations for a reviewed service manifest; they are not additional built-in application settings.

  1. Put the secret ARN in that host's bootstrap environment as APP_DATABASE_SECRET_ARN. Grant the service's AWS identity permission to read that specific secret. The ARN itself does not grant permission.
  2. Merge the following entries into the existing service manifest's variables list; do not replace the full manifest. Supply DATABASE_HOST, DATABASE_PORT and DATABASE_NAME through the selected service's configuration. A missing or empty resolved component must be fixed before activation.
variables:
  - name: APP_DATABASE_SECRET_ARN
    source: local
    required: true
  - name: APP_DATABASE_USERNAME
    required: true
    sensitive: true
    secret:
      arn_from: APP_DATABASE_SECRET_ARN
      type: json
      key: username
  - name: APP_DATABASE_PASSWORD
    required: true
    sensitive: true
    secret:
      arn_from: APP_DATABASE_SECRET_ARN
      type: json
      key: password
  - name: DATABASE_HOST
    required: true
  - name: DATABASE_PORT
    default: '5432'
  - name: DATABASE_NAME
    required: true
  - name: DATABASE_URL
    required: true
    sensitive: true
    containers: [telapi]
    compose:
      template: 'postgresql://${APP_DATABASE_USERNAME}:${APP_DATABASE_PASSWORD}@${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_NAME}'
      encode: [APP_DATABASE_USERNAME, APP_DATABASE_PASSWORD]
  1. Ensure the intended Compose service actually receives DATABASE_URL from the resolved environment. The illustrative containers: [telapi] metadata does not edit Compose automatically or make this suitable for every service. Retain the deployment's explicit database TLS settings; URL composition does not install trust material.
  2. Apply the approved manifest and activate the affected service through the deployment update procedure. Configuration download alone does not recreate a running container. Verify a new database connection and an authorized application operation without printing the URL or fetched exports.

The startup resolver runs when that deployment workflow invokes it; it does not subscribe running processes to AWS secret rotation. For automatic authentication-triggered runtime refresh, configure the separate DATABASE_SECRET_ARN path below and verify its payload, endpoint and TLS behavior. A provider-managed secret may belong to a privileged master user: do not reuse it as the application credential solely because an older example calls it a master secret.

Run the intended migration/setup release​

  1. Identify the target database, selected application and migration image versions, intended schema change, backup/recovery point and permitted maintenance window. Check release compatibility and TLS configuration. Do not use a newer migration image while trying to recover an older schema.
  2. Verify that the provisioned Ops bootstrap, selected manifest, ECR access, network/proxy path, CA material and migration credential are ready. Use the normal deployment path to load them; a fresh SSH shell does not inherit the environment established by init.sh.
  3. Normal Ops initialization invokes docker compose --profile migration run --rm --no-deps db-migrate after loading configuration and preparing TLS. An initial TCP-port wait (default 30 attempts, 10-second retry delay) establishes reachability only. The default entrypoint generates Prisma, runs prisma migrate deploy, then optionally runs baseline seed and active-flow Redis backfill.
  4. Review each stage's result. Ops initialization refuses to start its services if migration/seed fails; this does not undo committed migrations, seed writes or work already performed on other hosts. Restart-only or skip-migrations modes skip migration; they are not recovery or validation actions.
  5. For approved setup-registry work on a provisioned Ops host, use ./delphi-setup.sh from its service directory. It loads bootstrap/proxy and service environment, prepares CA files and launches the one-shot setup CLI. --apply-missing applies pending steps; it is a mutation, not a preview. --no-pull deliberately uses local image availability. Even an ordinary pull failure can fall back to a cached image: confirm its identity before continuing.

The standalone migration image defaults RUN_BASELINE_SEED true and RUN_ACTIVE_FLOW_REDIS_SYNC false; the Ops Compose file sets both to true unless overridden. Active-flow backfill calls delphi-redis-sync --active-flows-only; it is not a full platform Redis rebuild. Baseline setup can change configuration as well as create missing structure—review the legal-catalogue boundary, for example. Passing a custom prisma or delphi-setup command does not run the entrypoint's complete default sequence automatically.

On failure, preserve the migration identifier, selected image, stage and sanitized error. Inspect database migration/setup state before retrying. Do not mark a failed migration resolved, rerun baseline seed indiscriminately or restore an old credential merely to make the runner green. Coordinate recovery with the database owner; these scripts do not roll back the schema automatically.

Select the runtime credential source​

The normal application source is DATABASE_URL. The shared client creates its first pool from this URL on first use. DATABASE_SECRET_ARN does not replace the startup URL: it adds a source that can be refetched after a database authentication failure. A missing startup URL fails before this recovery path can help.

TelWeb, TelAPI, Telphi/Voice, Tasker and Scaler use the shared @delphi/db client. The reference Web, API, Voice and Ops service definitions declare and pass the refresh settings below; Ops passes them to Tasker and Scaler. Verify the actual deployed container environment as well as its declaration. A raw PostgreSQL client, migration command, PgBouncer or unrelated service does not acquire this behavior merely because these variables exist elsewhere.

The database/runtime identity and the AWS identity are separate. The runtime user needs the application's database permissions; the service's AWS identity must be able to read the chosen Secrets Manager secret. Use the runtime secret, not a privileged migration/master credential just because a legacy example calls its bootstrap variable DB_MASTER_SECRET_ARN. A platform UI secret registry row is not the same thing as this database-secret source.

VariableRange and defaultEffect
DATABASE_URLRequired.Initial PostgreSQL URL, including runtime user, host and database. In composed JSON mode it also supplies omitted connection components.
DATABASE_SECRET_ARNOptional.Secrets Manager identifier. Without it, automatic refresh cannot obtain replacement credentials.
DATABASE_SECRET_REGIONWhen absent, the helper falls back to AWS_REGION, then AWS_DEFAULT_REGION. An empty string does not use that fallback.Secret region. The reference Compose files can pass an empty value, so set the region explicitly when enabling refresh.
DATABASE_SECRET_MODEauto, plaintext, json or rds-json; default auto.Secret payload format; formats below.
DATABASE_SECRET_JSON_KEYOptional. Not a dotted-path expression.Top-level JSON key to select before the normal URL-field search.
DATABASE_SECRET_URL_FIELDDefault DATABASE_URL.Preferred URL-field name. Standard alternatives are still searched afterward.
DATABASE_SECRET_CACHE_TTL_MSMilliseconds; default 60000.Cache lifetime for ordinary resolver calls. This is not a polling interval; authentication-triggered refresh bypasses this cache.
DATABASE_REFRESH_FOREGROUND_WAIT_MSMilliseconds; default 2000.How long a failed operation waits for the shared refresh before returning its original error. A timeout does not cancel the background refresh.
DATABASE_REFRESH_MAX_RETRY_MSMilliseconds; default 120000.Retry budget checked after a failed refresh attempt. It is not a hard end-to-end deadline or an AWS request timeout.
DATABASE_POOL_SIZEShared client fallback 50; deployed configuration may supply another value.Connections per pool/process. During replacement, old and new pools briefly coexist.

Use positive integers for the millisecond settings; zero, negative or unparseable values fall back to the defaults rather than disabling refresh. Configure database TLS explicitly using its owning deployment guidance: refreshing credentials reuses the process's current TLS configuration and does not correct a trust/hostname mismatch.

Choose a supported secret payload​

The resolver reads SecretString. A missing secret or a secret with only binary content cannot supply credentials. Keep values out of terminal output, tickets and screenshots; the shapes below describe the contract without providing a credential.

Mode/selectorParsing behavior
plaintextReturns the trimmed string as the URL. This does not validate that it is a usable PostgreSQL URL.
autoAccepts a postgres:// or postgresql:// string directly; otherwise attempts JSON. If JSON parsing fails, returns the raw string for the connection layer to handle. A malformed string is not thereby validated.
jsonRequires JSON. With DATABASE_SECRET_JSON_KEY, a non-empty string at that exact key is returned. Without it, URL-field search is followed by component composition.
rds-jsonRequires JSON, but still checks an explicit selector and recognized URL fields before composing individual credentials; this is not strict validation against an AWS-managed-secret schema.

Without an explicit JSON key, URL fields are tried in this order: configured DATABASE_SECRET_URL_FIELD, then DATABASE_URL, databaseUrl, url, connectionString, connection_string. The first non-empty string wins. With an explicit key, a missing or empty selected field is an error. In auto/rds-json, a selected value that is not a PostgreSQL URL proceeds to component composition; do not configure that selector merely to name a password field.

Component composition requires non-empty string username (or user) and password. Optional string fields are host/hostname, port, and dbname/database/db/name. Numeric fields are not read as strings: a JSON number such as a port can be ignored. The URL builder encodes username/password and replaces supplied components while keeping unspecified components and query parameters from the current URL. Without a base URL it starts from postgresql://localhost/postgres; do not rely on that helper fallback for a deployed service.

An RDS-shaped secret can contain the database server's direct host, while the application URL points at PgBouncer. Supplying host in the secret changes the endpoint during refresh; supplying a database name can likewise change the target. Review the entire resulting connection contract, not only the password. Use a complete intended URL or a deliberately scoped component payload when the endpoint must stay fixed.

Understand automatic refresh​

The proxy recognizes PostgreSQL authentication codes 28P01/28000, Prisma P1000, and specific authentication-failure messages, including a nested cause. It does not trigger this recovery for every database error. Invalid TLS, DNS/network failures, permissions for an application table, or an invalid AWS secret format need their own repair.

Only one refresh runs at a time within a process. Each attempt fetches the secret afresh, builds a replacement client and runs SELECT 1. Only a successful validation replaces the shared client. Failed validation disconnects the candidate and retains the previous client; a successful swap disconnects the previous client asynchronously. This is not a fleet-wide atomic cutover or a guaranteed drain of existing work.

Authentication failures during refresh retry with delays of approximately 1, 2, 5, 10, 20 and then 30 seconds, with ±20% jitter, while the budget permits. Non-authentication failures stop that refresh. The foreground operation can return its original error after its shorter wait even if the background replacement later succeeds. Further operations can attempt recovery again; there is no periodic refresh timer.

Transaction boundary: the wrapper can retry the failed top-level Prisma method once, including $transaction. It does not inspect whether callback code sent mail, called an external API or performed another non-database side effect before the error. A callback retry can repeat those effects; a transaction's inner client is not independently wrapped by this proxy. Do not interpret successful pool replacement as permission to blindly repeat a user action or as a guarantee that an interrupted transaction resumes where it stopped. Reconcile the operation outcome and use the application's idempotency/recovery procedure.

You cannot trigger a refresh manually. Delphi offers no command or administration endpoint for it, and no schedule runs it; only a database authentication failure starts a refresh.

Plan and verify a runtime rotation​

  1. Inventory the actual consumers and distinguish runtime credentials from migration credentials, startup-only tools and pooler authentication. Identify service instances as well as service names; one healthy process does not prove another has refreshed.
  2. Verify the non-secret configuration: initial URL destination/user, secret identifier/region, payload shape, TLS trust and service AWS access. Ensure the secret's optional host/database fields will not unexpectedly redirect the replacement pool.
  3. Establish the database/provider's supported rotation and rollback procedure. Updating the secret alone does not change PostgreSQL's password, and updating PostgreSQL alone does not update the secret. Keep any supported overlap until consumer verification succeeds.
  4. Apply the authorized rotation through that owning procedure. The Delphi refresh path only reads the resulting secret; it does not perform the database password change. If a consumer uses startup-only values, follow its deployment refresh/recreation procedure instead.
  5. Verify a controlled application operation and the refresh outcome on each affected runtime. Existing authenticated connections may continue working, so a successful request on an old connection alone does not prove the new credential can establish a connection. Pool replacement's SELECT 1 proves a candidate connected, not full application permissions or transaction correctness.
  6. Reconcile failed or timed-out work before replay. Complete the upstream old-credential revocation only when the relevant consumers and rollback obligations have been checked.

A platform health response alone is insufficient. TelAPI /health checks Redis and the recorded database refresh state rather than issuing a database query; it can be green before a database failure has been observed. Its local/debug /metrics response includes database manager state. Telphi's dependency check performs SELECT 1 with a two-second outer timeout and reports refresh state. These are different checks, not interchangeable acceptance signals.

Diagnose and recover​

ObservationCheck and next action
DATABASE_URL is requiredSupply the correct runtime startup URL through deployment configuration. The secret ARN does not bootstrap the initial pool.
Secret region missing although AWS region existsCheck for an explicitly empty DATABASE_SECRET_REGION; set the intended region rather than relying on absent-variable fallback.
“no SecretString”, JSON/key errors or AWS access errorVerify secret type/identifier, mode/field names and service AWS permissions privately. These failures do not receive the database-authentication retry loop.
Authentication failure persistsConfirm secret and database credentials agree and the payload has not changed host/database/user unexpectedly. Check direct database versus pooler authentication separately.
Foreground failure followed by “Database client refreshed”Background recovery can outlast the operation's wait. Verify effective access, then reconcile the failed operation before retrying it.
degraded / lastRefreshErrorReplacement failed or the ARN is missing. The previous client may still exist; that does not establish that new connections work. Read redacted diagnostics and repair the cause.
idle / hasRuntimeSecret: trueidle is the initial state too; hasRuntimeSecret only indicates an ARN is configured. Neither proves a successful refresh. Check lastRefreshAt, service behavior and the relevant connection test.
Application still uses an old valueThere is no polling refresh. Verify whether a new authentication attempt failed, whether the process uses the shared client, and whether its deployment actually passes the ARN/settings.

Rollback must restore agreement between the database/pooler and the secret source, not just an old string in one store. It may require an approved deployment recreation or a further authentication-triggered refresh; restoring an old secret does not broadcast invalidation. Preserve affected-operation evidence and use Configuration recovery for the separate database/backup recovery boundary. Redact URLs and credentials from errors before sharing them.

Confirm the rotation is complete​

  1. On each affected service instance, run one controlled operation that writes to the database and confirm that it succeeds.
  2. Check that the refresh state reports a recent lastRefreshAt and no lastRefreshError.
  3. Revoke the old credential only after both checks pass on every consumer.