SIP trunk health monitoring

Delphi provides continuous monitoring of inbound SIP trunks when the relevant monitoring and probe path is configured. Status transitions can trigger alarms; verify the effective configuration and notification delivery rather than assuming the default deployment sends probes. This page explains how the monitor works, what each status means, how to query current health, and how to configure alarming.

For the individual log lines the monitor emits (when each fires, what it means, payload fields), see SIP trunk log reference. For trunk configuration, see SIP trunks (admin).

How it works

The health monitor runs on the Voice host inside TelPhi. It continuously polls Asterisk Manager Interface (AMI) for the status of each inbound trunk endpoint (pjsip show endpoint <name>), tracks state transitions, writes a Redis heartbeat, and triggers notifications on state change.

TelPhi ──AMI poll──▶ Asterisk (TelSys)
   │  status: available | degraded | unavailable
   │
   ├──▶ Redis heartbeat (voiceai:services:telsys:*:sip_trunks)
   │
   ├──▶ structured logs → SigNoz (logType: sip_trunk_health)
   │
   ├──▶ OTel alarm span (alarm.sip_trunk_health)
   │
   └──▶ sendNotification('SIP_TRUNK', …) → TRUNK_DOWN / TRUNK_DEGRADED / TRUNK_RECOVERED

AMI limitation in this release: the lookup needs an actual Asterisk PJSIP endpoint. A trunk identified only by Kamailio rules does not automatically create that endpoint; using its Delphi trunk ID as the AMI endpoint name can return a missing-object result. That is not evidence that the carrier is unreachable. Check the configured monitor endpoint and Asterisk state before acting on an alarm. This historical implementation does not provide the later edge-originated OPTIONS measurement.

TelAPI aggregates the per-server Redis heartbeats (with a database fallback) and serves them over the /health/sip-trunks REST endpoint, which TelWeb and operators consume.

Scope

• Inbound trunks only (the SipTrunk model). Egress/outbound trunks are not health-monitored.
• AMI endpoint status only — reachability and RTT of the trunk endpoint.
• Alarming only — the monitor never changes routing; it only observes and reports.

Status semantics

Each trunk is reported as one of three statuses:

Status	Meaning
available	The endpoint responded to the AMI probe within the degraded RTT threshold.
degraded	The endpoint is reachable but the probe RTT exceeded SIP_TRUNK_HEALTH_DEGRADED_RTT_MS (default 500 ms).
unavailable	The endpoint failed SIP_TRUNK_HEALTH_FAILURE_THRESHOLD consecutive monitor evaluations (default 2).

Newly discovered trunks initially have available state before a successful measurement. Hysteresis also retains the previous state until its threshold is met; a degraded result changes state immediately. Inspect lastCheckedAt, reason and RTT together: an available response alone does not prove a fresh successful probe.

A trunk returns to available after the configured failure-threshold count of consecutive successful monitor evaluations. Every transition is recorded as a status-transition event and, for available → degraded/unavailable and back, triggers a notification.

/health/sip-trunks endpoint

TelAPI serves aggregated per-trunk health for every voice server. It combines live Redis heartbeat data with the most recent database snapshots.

• Route: GET /health/sip-trunks
• Auth: internal API token with the read:trunk_health permission
• Response: 200 with generatedAt, count, and servers[]; 503 Redis unavailable if TelAPI cannot reach Redis

Each servers[] entry carries serverId, group, ip, lastHeartbeat, source (redis | db | heartbeat_only), and a trunks[] array. Each trunk has trunkId, trunkName, status, reason, rttMs, and lastCheckedAt.

curl -H "Authorization: Bearer $TOKEN" \
  https://<telapi>/health/sip-trunks

{
  "generatedAt": "2026-09-03T10:00:00.000Z",
  "count": 1,
  "servers": [
    {
      "serverId": "voice-01",
      "group": "default",
      "ip": "10.0.0.5",
      "lastHeartbeat": 1756886400000,
      "source": "redis",
      "trunks": [
        {
          "trunkId": "trk_01",
          "trunkName": "example-inbound",
          "status": "available",
          "reason": null,
          "rttMs": 42,
          "lastCheckedAt": "2026-09-03T09:59:30.000Z"
        }
      ]
    }
  ]
}

Alarming

On a status transition the monitor calls sendNotification('SIP_TRUNK', …). The notification types are seeded in the database:

Code	Severity	Fires when
TRUNK_DOWN	error	An inbound trunk endpoint is reported unavailable
TRUNK_DEGRADED	warn	An inbound trunk endpoint reports high latency
TRUNK_RECOVERED	info	A trunk returns to available

These lowercase values are notification-type severities, not normalized telemetry log levels such as INFO, WARN and ERROR.

Each notification carries serverId, trunkId, status, previousStatus, reason, rttMs, and timestamp. Notification types default to minIntervalMs of 300 s. The notification service keeps this limiter in process-local memory, keyed by category/code rather than trunk or recipient; it is not durable fleet-wide deduplication. Configure recipients through the notification subsystem (see Webhooks and the notification admin UI).

The monitor also emits an OTel span named alarm.sip_trunk_health with alarm.* semantic attributes (alarm.name, alarm.severity, alarm.state, alarm.previous_state, alarm.resource.type, alarm.resource.id, alarm.server.id, alarm.reason, alarm.rtt_ms) and an alarm.raised / alarm.recovered event. See Service logs → SIP trunk health alarms.

Configuration

The monitor is tuned with environment variables (see Environment variable index):

Variable	Default	Purpose
AMI_HOST	127.0.0.1	AMI host (TelSys)
AMI_PORT	8038	AMI port (matches TelSys MYAMIPORT)
AMI_USER / AMI_SECRET	—	AMI credentials (must match TelSys)
SIP_TRUNK_HEALTH_INTERVAL_MS	30000	Poll interval in milliseconds
SIP_TRUNK_HEALTH_FAILURE_THRESHOLD	2	Consecutive failures before a trunk is unavailable
SIP_TRUNK_HEALTH_DEGRADED_RTT_MS	500	RTT above which a reachable trunk is degraded

Observability

Trunk health is exposed as logs, events, and a REST endpoint — there is no Prometheus metric series for trunk health. To monitor in SigNoz, query the sip_trunk_health log type or the alarm.sip_trunk_health span. See SIP trunk log reference for the log catalog and Monitoring in SigNoz for query patterns.

See also

• SIP trunk log reference — every log line the monitor emits.
• SIP trunks (admin) — inbound trunk configuration.
• Egress trunks — outbound trunks (not health-monitored).
• Service logs — alarm.sip_trunk_health span details.