Check service health and intervene safely
Use this procedure when a service is unhealthy, traffic fails or a deployment change needs acceptance. Running, healthy, reachable and working for the user are different results. Record the deployment, release, service, time and symptom before changing state.
Availability: Host inspection and intervention require authorized operator access. Application tests require a designated test account/app/destination. Start with the actual service topology, including enabled features; an intentionally absent optional component is not an unhealthy one.
1. Inspect state without replacing it
On the relevant host, inspect a known container without dumping its environment:
docker ps --format 'table {{.Names}} {{.Status}} {{.Image}}'
docker inspect --format '{{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{else}}no-healthcheck{{end}}' voiceai-telapi
The second command is an API-host example. Substitute an actual container from that host's inventory. no-healthcheck means no Docker probe is configured; it is not a health verdict. Record restart loops, unhealthy status and the running image identity. Use the log evidence map to inspect a bounded time interval around the failure.
2. Understand the probe before trusting it
The following checks are the probes in the reference service Compose files. Your generated deployment may differ.
| Component | What the configured probe establishes | What still needs verification |
|---|---|---|
| TelAPI | Its local /health responds successfully; handler rejects Redis failure, degraded database state or a drain flag | Real ingress, authentication, provider/callback behavior and a representative session. The local TLS probe disables certificate verification, so it does not certify client trust. |
| TelWeb | The local web root responds | Actual sign-in, selected-team access and the intended write/read task |
| Web Caddy | Local Caddy configuration endpoint responds | Public DNS, certificate chain and route to the correct application |
| PostgreSQL/PgBouncer | pg_isready reports readiness on the configured listener | Correct runtime and migration credentials, expected schema, writable storage and application queries |
| Redis | Authenticated local ping on the configured plain/TLS connection | Required routing/queue state, consumer reachability and correct deployed TLS trust |
| TelSys/Asterisk | ARI HTTP responds with 200 or 401 | A 401 demonstrates a responding service, not correct ARI credentials or a functioning call/media path |
| Kamailio | Control command core.uptime responds | Expected SIP routing, trunk/TLS configuration and transaction outcomes |
| RTPEngine | Its configured UDP control listener and control query respond | Actual negotiated media, packets, codecs and audio in both directions |
| Janus, when present | PID 1 identifies itself as janus through /proc/1/comm | Signaling access, ICE/media connectivity and an actual browser voice session |
| coturn, when present | Local TCP listener accepts a connection | Authenticated relay allocation and the required external UDP/TCP/TLS relay path |
| AudioProc, when present | Local processing-service health URL responds | Actual audio format compatibility, entitlement and use in the selected flow |
| Media upload | Local upload-service /health responds | Authorized upload, persistent storage, HTTPS serving and consumer retrieval/trust |
| Squid | Squid process exists | Authorized egress to each required destination through the configured proxy |
TelPhi's own readiness check is stricter than the TelSys container probe above. It requests /ari/asterisk/info with its configured ARI username/password, requires a successful HTTP response, refuses redirects and times out after two seconds. A 401 response therefore makes TelPhi's Asterisk dependency unhealthy; it does not pass readiness just because Asterisk answered. Check the effective ARI address and credential source without exposing the secret. A successful authenticated probe still does not establish SIP routing or two-way media.
Do not invent a universal /health endpoint for every process. Tasker and Scaler need workload/leadership evidence. For SigNoz and collectors, inspect the deployed backend checks and verify fresh telemetry for a known test; a working UI can still show only old data.
3. Verify background work and the complete interaction
Tasker: verify scheduler leadership and due-job creation separately from worker processing. The scheduler runs on the leader; workers run on all Tasker instances. Inspect job state, attempts and last error. A queued backup, recording job or notification is not a completed result. Check its actual storage/delivery outcome before retrying; replay can repeat external effects.
Scaler: check leadership, recent decisions, pending operations and expected instance registration/heartbeats. A running process is not proof that cloud credentials work or that a new instance joined routing. Investigate stale observations before forcing replacement.
Call or browser path: use a new controlled interaction through the normal ingress. Verify the expected published flow, input/output, termination and relevant conversation evidence. Browser success does not certify SIP routing; SIP setup does not certify two-way audio. Use SIP diagnostics, browser integration and recording diagnosis for the specific failure.
4. Choose the smallest justified intervention
| Change or failure | Intervention boundary |
|---|---|
| Environment/secret/TLS change | Identify every presenting, trusting or credential-consuming service; apply through its supported initialization/refresh path. Changing the store is not evidence that a process adopted it. |
| API or Voice instance problem | Use applicable drain/rolling procedures and check healthy remaining capacity. A direct restart can drop sessions/calls. |
| Telephony-edge change | Confirm the actual edge redundancy, routing and media state. Use the deployment's edge refresh/cutover procedure; do not infer HA from two containers on one host. |
| Database/Redis/storage problem | Preserve recovery evidence and coordinate consumers. Restarting a shared dependency affects multiple roles; image rollback does not undo schema or data changes. |
| Ops problem | Inspect leader, worker and in-flight operations before restart/replay. Repeatedly restarting workers does not resolve a malformed or unauthorized job. |
| Observability problem | Diagnose collection/export/storage separately from call processing. Do not discard data or shorten retention merely to clear an alert without an approved recovery choice. |
Use Deploy, verify and update Delphi for actual update commands and configuration recovery for rollback scope. --restart-only is not an image upgrade; an Ops restart-only operation also skips migrations. A drain can wait, time out or be forced, so confirm the actual result before claiming no interruption.
After intervention, repeat the failed probe and the affected functional test. Record what changed and the new evidence. If health is green but the user task still fails, continue at the failing dependency or protocol stage; do not mark the incident resolved from the container badge alone.
Capacity and high availability need measurements
No calls-per-instance or zero-downtime guarantee follows from this reference. Measure the actual provider, codec, preprocessing, recording, API/WebSocket, database and telemetry workload. Include quotas, concurrency peaks, drain time and the capacity left after losing an instance or edge. A configured scale threshold is not measured capacity, and a second instance is not a tested failover procedure.
Keep workload, image/configuration identities, test duration, observed bottleneck and recovery timing with the result. Use capacity planning and maintenance for that separate exercise; do not extrapolate a light test call into a production sizing promise.
If the affected operation is asynchronous, follow background jobs and recurring schedules to distinguish scheduler leadership, pending retries and downstream effects before restarting a worker.
For database authentication failures or a healthy process that cannot use a new connection, follow database rotation diagnosis. Inspect the credential source and each consumer rather than treating every health endpoint as a database query.
Related
- Service topology: to identify which role owns the failing request path.
- Logs and diagnostics: to collect bounded evidence before you intervene.
- Deploy, verify and update Delphi: to refresh a service through the supported procedure.
- Choose and verify a configuration recovery: to roll back the smallest affected layer.