Skip to main content
Version: 0.9.12

Rotate secrets

platform v0.9.11verified 2026-05-14

Delphi services read secrets either at container start (most cases) or at process start. A rotation in your secret store does not propagate until the consumer restarts. This recipe is the safe, minimum-disruption order.

Before you start​

  • Know which services consume the secret. The Environment variable index lists source: sm rows. If unsure, search there first.
  • Know whether the secret is shared (e.g. DATABASE_URL) or service-local (e.g. VONAGE_SMS_API_KEY). Shared secrets need a coordinated restart; local secrets can be restarted on a single host.
  • Have the previous value to hand for rollback. Many secret stores keep a version history — make sure yours does.
STOP

For database secrets, see Managed database secrets first. Rotation there is multi-step (dual-credential window) and must be planned, not improvised.

Step 1 — Stage the new value​

Write the new secret in your secret store, but do not restart anything yet.

# AWS example, redact values
aws secretsmanager put-secret-value \
  --secret-id "<your-secret-id>" \
  --secret-string '{"VONAGE_SMS_API_KEY":"<new-value>"}'

Verify the staged value with get-secret-value from the same host (sanity-check IAM access).

Step 2 — Identify the consumers​

Run, on each candidate host:

cd /opt/services/<service>
grep -R "<SECRET_NAME>" .

The variable should appear in docker-compose.yaml, possibly in a generated env file. The set of services that grep matches is your restart set.

Step 3 — Re-pull and restart, one host at a time​

For each host in the restart set:

cd /opt/services/<service>
./init.sh
docker compose up -d --no-deps <containers-that-consume-the-secret>

init.sh re-fetches the secret from your store; up -d recreates the container with the new env. Tail logs to verify the consumer logged in / connected with the new secret before moving to the next host.

For services with multiple instances (Voice scaling), restart one instance first, verify it's healthy, then continue with the rest. This keeps the platform serving while you roll forward.

Step 4 — Verify​

  • The secret consumer logs show a successful auth/connect at start.
  • The feature that uses the secret works end to end (place a call, send an SMS, send a webhook).
  • Error rate in SigNoz hasn't ticked up on the restarted services.

Step 5 — Revoke the old value at the provider​

After everything is verified, revoke or rotate the old credential at the upstream provider (e.g. delete the old API key in the provider's console). Do this only after verification so a rollback is still possible.

Rolling back​

If something downstream breaks after the rotation:

  1. Put the previous value back in your secret store (most stores keep a version history — restore to the prior version).
  2. Repeat Step 3 on the affected hosts.
  3. Then debug the failure without time pressure.

See also​