Redeploy a service

This recipe gets a service back to a clean, fully-running state after a failed or interrupted update. It applies to any service under /opt/services/<service> (web, api, voice, telpro, database, media, ops, squid).

When to use this

• A previous update.sh exited non-zero and the containers are now mixed (some new, some old).
• You changed vars.yaml and want the running containers to pick the changes up.
• You need to roll back to a previous ECR_TAG.

When not to use this

• The host itself is unreachable or the disk is full — fix the host first.
• A Postgres migration partly succeeded and the schema is in an unknown state. See the migration sub-recipe below; it is not safe to just re-run update.sh blindly in that case.

Step 1 — Confirm the current state and recovery target

On the affected host:

cd /opt/services/<service>
docker compose ps --all
docker compose images
grep -E '^(ECR_TAG|CONFIG_REF|ENVIRONMENT|NAMESPACE)=' /opt/deployment/.env
docker compose logs --since 15m --tail 200 <compose-service>

Record the state, image identity and recent errors for each affected container. Check the environment before changing anything. Select the intended image/config combination from the deployment record or release instructions, including its database compatibility; see Component dependencies.

A failed update can leave the bootstrap tag pointing at a target that never started successfully. Compare it with actual running images and the previous change record. Do not infer the rollback target from the tag in .env alone.

Step 2 — Prepare the affected service

1. Confirm the chosen images/configuration are retrievable and preserve the applicable backup/rollback point.
2. For Voice or API, remove/drain the affected replica using the deployment's supported procedure; confirm remaining replicas can handle traffic. For TelPro or shared stateful services, use the planned maintenance/cutover procedure.
3. Confirm active work has drained before proceeding. A direct shell update.sh or init.sh invocation is not a promise of graceful draining; use Service Refresh where appropriate.
4. If migrations failed, follow Partial migration recovery before continuing with application restarts.

Step 3 — Apply the desired image and configuration

Choose one operation matching the intended change. Replace placeholders with the recorded target; the commands below are alternatives.

# Preview an image/config change without applying it.
./update.sh --ecr-tag <target-tag> --config-ref <target-config-ref> --dry-run

# Apply the approved image and configuration combination.
./update.sh --ecr-tag <target-tag> --config-ref <target-config-ref>

# Refresh runtime configuration and restart using cached images only.
./update.sh --restart-only

A dry run describes the planned changes; it does not prove registry access, schema compatibility or application health. --restart-only requires the necessary images to be available locally and does not fetch a newly published image tag.

update.sh delegates to the service's init.sh, which refreshes configuration/environment and starts/verifies containers. init.sh can restart or recreate containers; do not use it as a read-only configuration preview. Its behavior, including migrations on Ops, is described in Init and update.

Keep the command output and exit status with the change record. A zero exit status is the starting point for functional verification, not the end of the procedure.

Step 4 — Verify before returning traffic

docker compose ps --all
docker compose images
docker compose logs --since 5m --tail 100 <compose-service>

Check	Expected result
Images/config	Match the intended target on the affected host; no unintended old/new combination remains.
Containers	Expected long-running services remain running, configured health checks pass, and no restart loop appears. A successful one-shot can be exited with code 0.
Web/API	Login and representative authorized requests succeed, including a negative permission/authentication check when that area changed.
Voice/TelPro	A fresh call completes signaling, required bot/audio turns, and hangup; transfer works if affected.
Ops	Affected jobs complete, including notification or backup jobs where relevant.
Telemetry	Fresh component logs/metrics arrive and the changed path is observable.

Return drained capacity to service through the deployment's routing/scaling procedure only after these checks. Observe it for the agreed window and record test call/request/job IDs and final versions.

If verification fails

Failure	Action
Image or configuration cannot be fetched	Check the exact target, registry/bucket access, proxy and credentials; avoid repeatedly restarting still-working replicas.
Containers restart repeatedly	Inspect the first startup error and missing configuration/dependencies; keep the replica out of traffic.
Health passes but a call or request fails	Investigate the full dependency path; a process health check does not validate credentials, media or provider integration.
Schema compatibility is unknown	Stop the rollout and resolve migration state before selecting another image.
Rollback criteria are reached	Restore the recorded compatible image/config set using the same procedure. If schema/data must be reverted, use the coordinated database recovery procedure.

If rollback is unsuccessful, retain logs and both attempted image/config identities for support; do not cycle through unverified release combinations.

Partial migration recovery

If update.sh failed during the v0.9.13+ database migration step, the schema may be partially advanced. Do not just retry the full service rollout.

STOP

Confirm with whoever owns the release whether the new tag's migrations are designed to be re-runnable after partial failure. If you don't know, treat the database as degraded and consult support before proceeding.

If you have confirmed the migrations are safe to re-attempt:

1. Keep long-running app services stopped unless the release owner confirms the current schema remains compatible with their existing images. Do not restart Web/API/Voice on the new tag yet.

2. Capture a Postgres snapshot before anything else (see Restore Postgres for the snapshot mechanism your deployment uses).

3. Re-run the migration step explicitly from the Ops host:

   cd /opt/services/ops
   docker compose --profile migration run --rm --no-deps db-migrate

4. If the migration fails again with the same error, stop here. Do not loop. File a ticket per Getting help with the failing migration name and the error.

Once the schema converges, continue the standard service rollout and finish Step 4 — Verify before returning traffic.

See also

• Init and update — the canonical reference for init.sh / update.sh.
• Ops operations — one-shot migration container.
• Web operations — Web service startup and dashboard runtime.
• delphi-setup CLI — re-run the setup registry after a tag bump.