Operational guide

Use this guide as the entry point for operating an installed Delphi voice gateway. Select the documentation version matching the deployed release. Record the deployment's host inventory, image tags/digests, configuration reference, enabled features, database backup location, and support owner in its handover record.

Find the right procedure

Operational need	Procedure and reference
Provision machines for a call volume	Capacity planning, instance specifications, and load-test conditions.
Inspect services, dependencies, and ports	Service operations and architecture.
Deploy or roll back one component	Component dependency matrix, restart matrix, and image/config rollback.
Back up configuration and data	Configuration backup and rollback; Restore Postgres for database recovery.
Restore a TelWeb-managed configuration version	Config versioning and backups. This has a narrower scope than infrastructure or database disaster recovery.
Check supported SIP methods	SIP methods and support scope.
Interpret SIP errors	SIP response definitions, including how to identify the response source.
Interpret API errors	API status and error-code definitions; use the API reference for endpoint-specific responses.
Change component log levels	Per-service log levels and service log inventory.
Provision or audit privileged users	Users and teams, Access logs, and Super Admin recovery.

Routine checks

1. Check service/container health and expected instance counts against the deployment inventory. Investigate missing heartbeats or unhealthy replicas using Instance debugging.
2. Confirm fresh logs and metrics in SigNoz, including trunk health, call failures, media quality, resource saturation, and queue failures.
3. Confirm the most recent scheduled database backup completed successfully and its object exists. Record backup age, retention, and the latest successful restore rehearsal; object existence alone is insufficient.
4. Review Job history for failed/dead jobs, including backups and role-change email delivery, and Access logs for privileged changes.
5. Check disk space, database growth, telemetry retention, provider quotas, and call-capacity headroom before the next expected peak.

Record the result of each routine check

Check	Expected result	If the result differs
Container state	Expected long-running services are running; configured health checks pass. Completed one-shot migration containers may be exited with code 0.	Identify the component and exit/restart state; inspect its recent logs before restarting it.
Call processing	A representative call connects, completes the expected bot/audio turns, and ends cleanly.	Correlate the SIP ladder, conversation and provider events. SIP 200 OK alone does not demonstrate working audio.
Telemetry	Logs/metrics have recent timestamps for the expected active services.	Check collector health and export connectivity; distinguish a telemetry outage from a call-processing outage.
Backup	A recent completed backup job has a matching retained object and a known restore procedure.	Determine the last usable restore point and investigate failed/pending jobs; report backup age against the agreed RPO.
Email jobs	Required notifications reach the configured transport and intended mailbox.	Correlate job status and recipient/time with SMTP/SES delivery; avoid repeatedly triggering role transitions to test email.
Capacity	Remaining capacity covers the expected peak and maintenance/failure allowance.	Adjust admission/fleet planning and verify shared dependencies before increasing traffic.

Keep a timestamped result and owner for unresolved deviations. Use the deployment's agreed thresholds rather than treating a single healthy probe as proof that the whole platform is healthy.

Collect a first diagnostic snapshot

On the affected host, substitute the service directory from the deployment inventory:

cd /opt/services/<service>
date -u
docker compose ps --all
docker compose images
docker compose logs --since 15m --tail 200 <compose-service>
df -h

<compose-service> is a key in the generated Compose file; it can differ from the container name. Record the UTC incident window, image/config reference, affected call or request IDs, and any change made immediately beforehand. Review logs before attaching them to a ticket; preserve correlation IDs while following the deployment's handling rules for customer data and credentials.

Example: a call receives SIP 503

1. Open the failed conversation's Debug → SIP ladder and locate the first final failure response. If no conversation was created, search SigNoz by SIP Call-ID and the UTC incident window.
2. Identify who generated the response using its capture point. A TelPro-local 503 directs investigation toward gateway routing, drain state, or available Voice targets. A carrier-originated 503 directs investigation toward the trunk/carrier path.
3. Inspect the emitting service's logs at the same time, including the reason phrase and relevant routing/health events. Check whether the failure began during a deployment or drain.
4. Correct the identified cause and place a fresh representative call. Verify both signaling and completed audio/bot turns, then record the successful call ID.

Use SIP response definitions for other codes. Increasing retries before identifying the emitting hop can increase load on an already unavailable dependency.

Before a change window

1. Record the change, affected hosts, current and target release/config references, dependencies, and expected user/call impact.
2. Capture the applicable backup sources, including the database before migrations. Agree the maximum acceptable lost-write interval (RPO) and restore duration (RTO) for this deployment.
3. Establish rollback triggers such as failed health checks, login failure, increased call rejection, missing audio, or unsuccessful notification delivery. Name the rollback image/config and database restore point where needed.
4. Follow the release-specific deployment order and restart matrix. Drain affected call-serving capacity before restarting it; preserve enough remaining capacity for live traffic.
5. Validate health, login, a representative call with completed bot/audio turns, telemetry, and affected background jobs. For role-management changes, also verify actor code delivery, target notification, the audit event, and forced target re-login.
6. Record the result. If rollback is required, restore only the affected source where possible; database restore requires stopping writers and accepting loss of later writes.

Change record to complete

Field	What to record
Change reference / owner	Ticket or change ID, executing operator and recovery contact.
Scope	Environment, affected service hosts/replicas, user impact and maintenance window.
Before / after	Exact image tag/digest and configuration reference for each affected component.
Dependencies	Required database schema, shared secrets, job/protocol compatibility, and release instructions.
Recovery point	Backup job/object or snapshot identity, UTC time, configuration/secret version identifiers, and restore rehearsal result. Do not paste secret values.
Decision limits	Agreed health/error/latency criteria, observation window, and the latest time to start rollback within the maintenance window.
Verification	Health results, test call/request IDs, telemetry, and affected job/email outcomes.
Outcome	Completed or rolled back, final running versions, remaining issues and follow-up owner.

A rollback decision should be made against the criteria recorded before the change. If recovery exceeds the planned window or the rollback target is incompatible with the resulting schema, keep the incident open and involve the release/database owner before trying another version.

Keep the deployment record current

Update the inventory and runbook after each release, capacity test, dependency change, backup/restore rehearsal, or privileged-access change. Retain measured workload conditions with capacity results and exact tested image/config combinations with any component-only deployment approval. A list of changed components alone does not establish mixed-version compatibility.