Version: 0.9.16 — archived documentation

SIP trunk health monitoring

Delphi provides continuous monitoring of inbound SIP trunks when the relevant monitoring and probe path is configured. Status transitions can trigger alarms; verify the effective configuration and notification delivery rather than assuming the default deployment sends probes. This page explains how the monitor works, what each status means, how to query current health, and how to configure alarming.

For the individual log lines the monitor emits (when each fires, what it means, payload fields), see SIP trunk log reference. For trunk configuration, see SIP trunks (admin).

How it works

The health monitor runs on the Voice host inside TelPhi. It continuously polls Asterisk Manager Interface (AMI) for the status of each inbound trunk endpoint (pjsip show endpoint <name>), tracks state transitions, writes a Redis heartbeat, and triggers notifications on state change.

TelPhi ──AMI poll──▶ Asterisk (TelSys)
   │  status: available | degraded | unavailable
   │
   ├──▶ Redis heartbeat (voiceai:services:telsys:*:sip_trunks)
   │
   ├──▶ structured logs → SigNoz (logType: sip_trunk_health)
   │
   ├──▶ OTel alarm span (alarm.sip_trunk_health)
   │
   └──▶ sendNotification('SIP_TRUNK', …) → TRUNK_DOWN / TRUNK_DEGRADED / TRUNK_RECOVERED

AMI limitation in this release: the lookup needs an actual Asterisk PJSIP endpoint. A trunk identified only by Kamailio rules does not automatically create that endpoint; using its Delphi trunk ID as the AMI endpoint name can return a missing-object result. That is not evidence that the carrier is unreachable. Check the configured monitor endpoint and Asterisk state before acting on an alarm. This historical implementation does not provide the later edge-originated OPTIONS measurement.

TelAPI aggregates the per-server Redis heartbeats (with a database fallback) and serves them over the /health/sip-trunks REST endpoint, which TelWeb and operators consume.

Scope

Inbound trunks only (the SipTrunk model). Egress/outbound trunks are not health-monitored.
AMI endpoint status only — reachability and RTT of the trunk endpoint.
Alarming only — the monitor never changes routing; it only observes and reports.

Status semantics

Each trunk is reported as one of three statuses:

Status	Meaning
`available`	The endpoint responded to the AMI probe within the degraded RTT threshold.
`degraded`	The endpoint is reachable but the probe RTT exceeded `SIP_TRUNK_HEALTH_DEGRADED_RTT_MS` (default 500 ms).
`unavailable`	The endpoint failed `SIP_TRUNK_HEALTH_FAILURE_THRESHOLD` consecutive monitor evaluations (default 2).

Newly discovered trunks initially have available state before a successful measurement. Hysteresis also retains the previous state until its threshold is met; a degraded result changes state immediately. Inspect lastCheckedAt, reason and RTT together: an available response alone does not prove a fresh successful probe.

A trunk returns to available after the configured failure-threshold count of consecutive successful monitor evaluations. Every transition is recorded as a status-transition event and, for available → degraded/unavailable and back, triggers a notification.

`/health/sip-trunks` endpoint

TelAPI serves aggregated per-trunk health for every voice server. It combines live Redis heartbeat data with the most recent database snapshots.

Route: GET /health/sip-trunks
Auth: internal API token with the read:trunk_health permission
Response: 200 with generatedAt, count, and servers[]; 503 Redis unavailable if TelAPI cannot reach Redis

Each servers[] entry carries serverId, group, ip, lastHeartbeat, source (redis | db | heartbeat_only), and a trunks[] array. Each trunk has trunkId, trunkName, status, reason, rttMs, and lastCheckedAt.

curl -H "Authorization: Bearer $TOKEN" \
  https://<telapi>/health/sip-trunks

{
  "generatedAt": "2026-09-03T10:00:00.000Z",
  "count": 1,
  "servers": [
    {
      "serverId": "voice-01",
      "group": "default",
      "ip": "10.0.0.5",
      "lastHeartbeat": 1756886400000,
      "source": "redis",
      "trunks": [
        {
          "trunkId": "trk_01",
          "trunkName": "example-inbound",
          "status": "available",
          "reason": null,
          "rttMs": 42,
          "lastCheckedAt": "2026-09-03T09:59:30.000Z"
        }
      ]
    }
  ]
}

Alarming

On a status transition the monitor calls sendNotification('SIP_TRUNK', …). The notification types are seeded in the database:

Code	Severity	Fires when
`TRUNK_DOWN`	`error`	An inbound trunk endpoint is reported unavailable
`TRUNK_DEGRADED`	`warn`	An inbound trunk endpoint reports high latency
`TRUNK_RECOVERED`	`info`	A trunk returns to `available`

These lowercase values are notification-type severities, not normalized telemetry log levels such as INFO, WARN and ERROR.

Each notification carries serverId, trunkId, status, previousStatus, reason, rttMs, and timestamp. Notification types default to minIntervalMs of 300 s. The notification service keeps this limiter in process-local memory, keyed by category/code rather than trunk or recipient; it is not durable fleet-wide deduplication. Configure recipients through the notification subsystem (see Webhooks and the notification admin UI).

The monitor also emits an OTel span named alarm.sip_trunk_health with alarm.* semantic attributes (alarm.name, alarm.severity, alarm.state, alarm.previous_state, alarm.resource.type, alarm.resource.id, alarm.server.id, alarm.reason, alarm.rtt_ms) and an alarm.raised / alarm.recovered event.

Configuration

The monitor is tuned with environment variables (see Environment variable index):

Variable	Default	Purpose
`AMI_HOST`	`127.0.0.1`	AMI host (TelSys)
`AMI_PORT`	`8038`	AMI port (matches TelSys `MYAMIPORT`)
`AMI_USER` / `AMI_SECRET`	—	AMI credentials (must match TelSys)
`SIP_TRUNK_HEALTH_INTERVAL_MS`	`30000`	Poll interval in milliseconds
`SIP_TRUNK_HEALTH_FAILURE_THRESHOLD`	`2`	Consecutive failures before a trunk is `unavailable`
`SIP_TRUNK_HEALTH_DEGRADED_RTT_MS`	`500`	RTT above which a reachable trunk is `degraded`

Observability

Trunk health is exposed as logs, events, and a REST endpoint — there is no Prometheus metric series for trunk health. To monitor in SigNoz, query the sip_trunk_health log type or the alarm.sip_trunk_health span. See SIP trunk log reference for the log catalog and Monitoring in SigNoz for query patterns.

How it works​

Scope​

Status semantics​

/health/sip-trunks endpoint​

Alarming​

Configuration​

Observability​

See also​