Restart matrix
Most Delphi configuration is resolved when a service starts. Changing an AWS SSM parameter, Secrets Manager value, bootstrap value, or generated config bundle does not change running containers until the affected service re-runs init.sh through update.sh.
Use this page before deployments, secret rotations, and certificate changes to decide the smallest safe restart set.
Rules of thumb
| Change type | What to run | Why |
|---|---|---|
| New image tag | ./update.sh --ecr-tag <tag> on each changed service | Pulls the image, refreshes env, recreates containers. |
New generated config bundle (CONFIG_REF, Compose, Caddyfile, collector config, vars.yaml) | ./update.sh --config-ref <ref> on each affected service | Re-syncs the bundle from S3, then recreates containers. |
| SSM or Secrets Manager value | ./update.sh --restart-only on each consumer | Re-runs fetch-env.sh and restarts containers with the new env. |
Bootstrap /opt/deployment/.env (ECR_TAG, CONFIG_REF, NAMESPACE, AWS credentials) | Prefer ./update.sh ...; if edited manually, run ./init.sh on that host | Bootstrap is read before fetch-env.sh; update.sh also backs up the previous .env. |
| TelWeb database setting | Usually no host restart; verify the setting-specific doc | Many platform/admin settings are DB-backed and reload in the app path. |
| TelAPI rate-limit settings | No restart | Saved from TelWeb and reloaded through Redis; see API operations. |
TLS material decoded by init.sh | ./update.sh --restart-only on the service presenting or trusting the cert | Cert files are materialized during service startup. |
| Database schema migration | Run the one-shot migration on Ops before app restarts | Long-running Web/API/Voice/Ops containers should match the migrated schema. |
Service restart matrix
| Service | Restart when these values change | Restart notes |
|---|---|---|
| Web | DOMAIN_TELWEB, DOMAIN_SIGNOZ, auth/session/password policy, NEXTAUTH_*, DATABASE_*, REDIS_*, feature flags, legal banner env | Caddy can renew Let's Encrypt certificates itself, but Caddyfile/domain changes require Web restart. Password-expiry policy must match Ops. |
| API | DOMAIN_API, JWT_SECRET, DATABASE_*, REDIS_*, WebRTC/API-access feature flags, media-cache TLS/token values, provider webhook secrets | Multiple API instances sit behind a load balancer; restart one instance at a time when possible. |
| Voice | TelSys SIP settings, TelPhi provider/runtime settings, DATABASE_*, REDIS_*, recording, media-cache TLS/token values, PII logging, SIP_100REL_MODE | Drain active calls before restarting a Voice instance. The Scaler can remove an instance from routing before replacement. |
| TelPro | SIP/TLS ports and domains, Kamailio/RTPEngine log levels, PUBLIC_IP, PRIVATE_IP, TLS_*, Redis settings, Janus/TURN settings | TelPro is the public SIP edge. Plan a maintenance window or an active/passive cutover for cert and routing changes. |
| Database | Postgres/Redis/PgBouncer passwords, TLS server material, TLS mode flags, storage mount changes | Restarting Database interrupts all consumers. Coordinate app restarts for password or TLS changes. |
| Media | Media TLS material, upload token, media data path, SAN/DNS values | After rotating Media TLS, restart Media first, then API and Voice consumers with the new trust bundle. |
| Ops | Scaler settings, Tasker settings, SMTP/SES, backup S3 settings, password-expiry reminder values, migration connection | Restart Ops after changing scheduled-job, backup, scaling, or reminder policy. Password-expiry values must match Web. |
| SigNoz | SigNoz backend config, ClickHouse/ZooKeeper settings, retention/storage settings | Restart affects observability, not call processing, but telemetry may buffer or drop while down. |
| Squid | Proxy ACLs, upstream proxy, listen port | Private services depend on Squid for outbound ECR, AWS, and provider traffic; update Squid before services that need the new proxy behavior. |
Graceful in-place refresh (Service Refresh)
For env/TLS re-reads on TelPro, TelAPI, and TelPhi you usually do not need SSH at all: a Super Admin can trigger a graceful refresh from TelWeb (Super Admin → Service Refresh), and deployments can schedule the same job via Tasker cron. The job drains active work before restarting, so calls in progress are not cut.
Strategies
| Strategy | What it does | Use for |
|---|---|---|
| Drain-restart | Stops accepting new work, waits for active calls/connections to finish (configurable timeout, default 60 min), then restarts and re-runs env fetch | Releases, secret/TLS rotation on live systems (recommended default) |
| Restart-only | Restarts immediately without waiting for a drain — the "just restart" option | Maintenance windows, hosts already out of traffic, urgent config fixes |
| Reload-only | Reloads TelPro trunk TLS material without restarting containers | Trunk certificate rotation on the SIP edge |
Per-service behavior
| Service | Mechanism | Preconditions | Active calls |
|---|---|---|---|
| TelPro | Host reconciler on each edge picks the command up from Redis; with multiple registered edges, hosts are refreshed strictly one at a time | Drain-restart requires at least 2 ready edges so one edge always carries traffic (the job refuses to start otherwise) | The draining edge stops taking new dialogs; established calls continue until they end or the drain timeout expires |
| TelAPI | Scaler rolling replace of instances behind the load balancer | API server group with scaling enabled | WebSocket/HTTP connections drain per instance before replacement |
| TelPhi (Voice) | Scaler rolling refresh per server group | At least one voice server group with scaling enabled | Instance is removed from call routing, drains its calls, then restarts |
Singletons (Web, Ops, Database, SigNoz, Squid) are not covered by Service Refresh — use the host-level update.sh flows from the tables above.
Interrupt/Stop
A running refresh can be stopped from the same panel (Stop refresh on the job card). Cancel semantics are deliberately conservative:
- The host or instance currently draining/restarting always finishes — no mid-host abort, which could leave an edge half-updated.
- Hosts and services that have not started yet are skipped and marked accordingly in the job result (
cancelled: true, instancesskippedwith "Cancelled by operator"). - The refresh timeline in the panel records
cancel_requestedandcancelledevents with who requested the stop.
Force
For drain-restart, the force option skips the drain wait and restarts as soon as active work is detected — equivalent to restart-only but still going through the drain bookkeeping. Use it only when dropping the remaining calls is acceptable.
Suggested rollout order
For a normal release:
- Read the release notes and release compatibility to identify changed services.
- On Ops, run release migrations if the release requires them.
- Restart shared infrastructure only when it changed: Squid, Database, SigNoz, Media.
- Restart application services: API, Web, Ops, Voice, then TelPro when telephony-edge config or image changed.
- For scaled API and Voice groups, roll instances one at a time and keep at least one healthy instance in service.
For a configuration-only change:
- Find the consumer in Environment variable index or with
show-env.sh. - Restart only those services with
./update.sh --restart-only. - Verify the resolved value from service logs, health checks, or the relevant UI.
Verification checklist
After the restart set completes:
docker compose psis healthy on every touched host.- Web and API health endpoints respond.
- A short test call reaches Voice and creates a conversation.
- SigNoz receives fresh logs/spans from each restarted service.
- For TLS/certificate changes, verify the caller/client sees the new certificate chain.
See also
- init.sh and update.sh — command reference and rollback of bootstrap
.env. - Environment resolution — how SSM and Secrets Manager values become process env.
- Rotate secrets — safe secret rotation order.
- Redeploy a service — recovering from a failed update.
TelPro host prerequisites from v0.9.17-patch2
Run the normal TelPro service initialization when installing this patch. It installs the native redis-tools package if missing, so the host needs access to its configured package repositories. Copying only the reconciler script onto a host without redis-cli is insufficient.
The idle reconciliation tick reuses the deployed container's Redis connection settings instead of fetching the complete cloud configuration and starting temporary Redis-client containers. Refresh and credential-rotation operations still fetch fresh configuration. The 30-second timer and 120-second health/heartbeat TTLs are unchanged.
After deployment, check that heartbeats continue, idle ticks no longer fetch cloud configuration or create client containers, and real dialog/media session counts are reported. A failed probe is unknown, not a successful empty drain. Validate multi-edge drain/Stop behavior separately when using high availability.