SIP trunk health monitoring

Delphi provides continuous monitoring of inbound SIP trunks when the relevant monitoring and probe path is configured. Status transitions can trigger alarms; verify the effective configuration and notification delivery rather than assuming the default deployment sends probes. This page explains how the monitor works, what each status means, how to query current health, and how to configure alarming.

For the individual log lines the monitor emits (when each fires, what it means, payload fields), see SIP trunk log reference. For trunk configuration, see SIP trunks (admin).

How it works

When edge probing is enabled, the SIP edge (Kamailio / telpro) sends a SIP OPTIONS reachability probe to each trunk's carrier source IP (derived only from SipTrunk.allowedIps[0], port HEALTH_PROBE_TARGET_PORT), measures the response RTT, and writes the raw result to Redis under voiceai:trunk-probe:<trunkId> (TTL 60s). The TelPhi monitor reads those keys, maps the raw { ok, rttMs, error } result to a status (so the degradedRttMs threshold stays in the monitor), tracks state transitions, writes a Redis heartbeat, and triggers notifications on state change.

Kamailio (telpro, edge) ──SIP OPTIONS──▶ trunk carrier source IP (allowedIps[0], :5060)
   │  reply + RTT
   │
   └──▶ Redis voiceai:trunk-probe:<id>  { ok, rttMs, error, checkedAt }  (TTL 60s)
            │
            ▼
        TelPhi monitor ──▶ status: available | degraded | unavailable
            │
            ├──▶ Redis heartbeat (voiceai:services:telsys:*:sip_trunks)
            │
            ├──▶ structured logs → SigNoz (logType: sip_trunk_health)
            │
            ├──▶ OTel alarm span (alarm.sip_trunk_health)
            │
            └──▶ sendNotification('SIP_TRUNK', …) → TRUNK_DOWN / TRUNK_DEGRADED / TRUNK_RECOVERED

Trunks in this system are identified by Kamailio (source IP / user-agent / digest), not as Asterisk PJSIP endpoints, so the reachability probe targets the carrier's source IP directly rather than an endpoint in Asterisk. The probe originates from the SIP edge's real edge IPs — the source IP the carrier expects traffic from — but carrier ACLs, supported OPTIONS behavior and the actual network path still require verification.

TelAPI aggregates the per-server Redis heartbeats (with a database fallback) and serves them over the /health/sip-trunks REST endpoint, which TelWeb and operators consume.

The target resolver uses only the first allowed-IP entry; it does not scan for another reachable address. A bare value passes through unchanged. For a value containing /, it drops the prefix and constructs the first three dot-separated components plus .1. This is not general CIDR host selection and does not establish IPv6 support. Verify the resulting destination with the carrier; do not assume it is the correct host for an arbitrary subnet.

Scope

• Inbound trunks only (the SipTrunk model). Egress/outbound trunks are not health-monitored.
• Reachability and RTT only — whether the carrier answers a SIP OPTIONS within the probe's transaction timeout, and how fast.
• Alarming only — the monitor never changes routing; it only observes and reports.

The probe is edge-originated, so it measures carrier reachability from the SIP edge, not anything inside Asterisk/Kamailio. A trunk that is enabled in Prisma but absent from the edge's Redis trunk store (store drift) has no probe result and is reported unavailable with reason no probe result from edge — an operator signal, not a routing change.

Status semantics

Each trunk is reported as one of three statuses:

Status	Meaning
available	The carrier answered the SIP OPTIONS within the degraded RTT threshold.
degraded	The carrier is reachable but the probe RTT exceeded SIP_TRUNK_HEALTH_DEGRADED_RTT_MS (default 500 ms).
unavailable	The edge reported the probe failed (no reply / non-2xx), or no fresh probe result exists (reason no probe result from edge), for SIP_TRUNK_HEALTH_FAILURE_THRESHOLD consecutive monitor evaluations (default 2).

Newly discovered trunks initially have available state before a successful measurement. Hysteresis also retains the previous state until its threshold is met; a degraded result changes state immediately. Inspect lastCheckedAt, reason and RTT together: an available response alone does not prove a fresh successful probe.

A trunk returns to available after the configured failure-threshold count of consecutive successful monitor evaluations. Every transition is recorded as a status-transition event and, for available → degraded/unavailable and back, triggers a notification.

/health/sip-trunks endpoint

TelAPI serves aggregated per-trunk health for every voice server. It combines live Redis heartbeat data with the most recent database snapshots.

• Route: GET /health/sip-trunks
• Auth: internal API token with the read:trunk_health permission
• Response: 200 with generatedAt, count, and servers[]; 503 Redis unavailable if TelAPI cannot reach Redis

Each servers[] entry carries serverId, group, ip, lastHeartbeat, source (redis | db | heartbeat_only), and a trunks[] array. Each trunk has trunkId, trunkName, status, reason, rttMs, and lastCheckedAt.

curl -H "Authorization: Bearer $TOKEN" \
  https://<telapi>/health/sip-trunks

{
  "generatedAt": "2026-09-03T10:00:00.000Z",
  "count": 1,
  "servers": [
    {
      "serverId": "voice-01",
      "group": "default",
      "ip": "10.0.0.5",
      "lastHeartbeat": 1756886400000,
      "source": "redis",
      "trunks": [
        {
          "trunkId": "trk_01",
          "trunkName": "example-inbound",
          "status": "available",
          "reason": null,
          "rttMs": 42,
          "lastCheckedAt": "2026-09-03T09:59:30.000Z"
        }
      ]
    }
  ]
}

Alarming

On a status transition the monitor calls sendNotification('SIP_TRUNK', …). The notification types are seeded in the database:

Code	Severity	Fires when
TRUNK_DOWN	error	An inbound trunk endpoint is reported unavailable
TRUNK_DEGRADED	warn	An inbound trunk endpoint reports high latency
TRUNK_RECOVERED	info	A trunk returns to available

These lowercase values are notification-type severities, not normalized telemetry log levels such as INFO, WARN and ERROR.

Each notification carries serverId, trunkId, status, previousStatus, reason, rttMs, and timestamp. Notification types default to minIntervalMs of 300 s. The notification service keeps this limiter in process-local memory, keyed by category/code rather than trunk or recipient; it is not durable fleet-wide deduplication. Configure recipients through the notification subsystem (see Webhooks and the notification admin UI).

The monitor also emits an OTel span named alarm.sip_trunk_health with alarm.* semantic attributes (alarm.name, alarm.severity, alarm.state, alarm.previous_state, alarm.resource.type, alarm.resource.id, alarm.server.id, alarm.reason, alarm.rtt_ms) and an alarm.raised / alarm.recovered event. See Service logs → SIP trunk health alarms.

Configuration

The monitor is tuned with environment variables (see Environment variable index):

Variable	Default	Purpose
SIP_TRUNK_HEALTH_INTERVAL_MS	30000	Monitor poll interval in milliseconds
SIP_TRUNK_HEALTH_FAILURE_THRESHOLD	2	Consecutive failures before a trunk is unavailable
SIP_TRUNK_HEALTH_DEGRADED_RTT_MS	500	RTT above which a reachable trunk is degraded
SIP_TRUNK_HEALTH_REDIS_TTL_MS	60000	Freshness window for an edge probe result; older results are treated as unavailable (no probe result from edge). Keep in sync with the edge's voiceai:trunk-probe:<id> TTL.

The probe itself is configured on the SIP edge (Kamailio / telpro), not TelPhi:

Variable	Default	Purpose
HEALTH_PROBE_ENABLED	false	Master switch for the edge's periodic per-trunk OPTIONS probe
HEALTH_PROBE_INTERVAL	60	Probe cycle interval in seconds
HEALTH_PROBE_TARGET_PORT	5060	Remote SIP port the OPTIONS probe is sent to

The reference deployment manifest and Compose supply HEALTH_PROBE_ENABLED=false. The image entrypoint alone falls back to true when no value is supplied; do not mistake that image fallback for the deployment default. Enable probing deliberately through the deployment configuration and verify that the running edge adopted it. The TelPhi monitor can run while edge probing is disabled.

Monitor evaluations and edge probes have independent clocks: with their defaults of 30 and 60 seconds, the same retained probe result can be evaluated more than once. Hysteresis counts monitor evaluations, not distinct OPTIONS exchanges; do not infer a fixed outage-detection time from the failure threshold.

How long a probe waits for a reply is not a health-probe setting: the probe uses Kamailio's SIP transaction timeout (tm.fr_timer, currently 45 s). The legacy HEALTH_PROBE_TIMEOUT example from earlier releases does not control it.

The edge writes one raw result per trunk to voiceai:trunk-probe:<trunkId> (TTL 60s). SIP_TRUNK_HEALTH_REDIS_TTL_MS should be set to match that TTL so a result is not treated as stale before the edge refreshes it.

Enable probing and verify a dedicated trunk

There is no independent per-trunk probe switch in this version's SIP trunk editor or schema. HEALTH_PROBE_ENABLED controls the edge-wide probe loop. In the trunk editor, Enabled and Allowed IPs affect probe eligibility; Internal trunk remains a routing classification: the edge reads enabled trunks from its Redis trunk store and skips trunks without the first allowed IP. This release does not use the internal classification to exclude probes or monitor entries. These are routing/identification settings, not monitoring-only controls. Do not disable a production trunk, mark it internal, or remove its allowed IPs merely to suppress probes; those changes can affect call routing or trust boundaries.

1. Identify the intended external inbound trunk in SIP trunks. Check its enabled/internal classification and allowed IPs against the intended carrier configuration. A dedicated test trunk must use an authorized test peer; do not modify a live carrier record to simulate failure.
2. Have the deployment operator explicitly enable HEALTH_PROBE_ENABLED=true on the relevant edge and apply that deployment's configuration activation procedure. Verify the running edge's effective value and that the intended trunk reached its Redis trunk store; saving a UI record alone is not proof of activation. There is no dedicated “enable health check” UI action.
3. Observe trunk_probe_success / trunk_probe_failure and the corresponding voiceai:trunk-probe:<trunkId> result. Check its checkedAt, ok, rttMs and error; then compare the TelPhi heartbeat and /health/sip-trunks response. Allow for independent probe and monitor intervals plus hysteresis; a single initial available response is insufficient.
4. For an authorized test peer, verify a successful 2xx reply and a failure separately. 408 is a failure, not a second successful status. A callback with no reply code uses raw error no reply; a nonzero failing status uses status N (for example status 408). Structured failure codes are trunk_probe_no_reply and trunk_probe_status_N. Missing or stale Redis results instead produce the monitor reason no probe result from edge.
5. Verify the intended notification recipient and downstream logs separately. If no result appears, check the global switch, Redis trunk eligibility/store synchronization, carrier OPTIONS policy and network path before changing alarm thresholds. Restoring the previous edge-wide setting affects all eligible trunks; it is not a per-trunk mute.

The callback code classifies 2xx as success, but this does not establish reliable successful-result delivery in patch1: it uses uac_req(evroute)=2, whereas patch2 changes it to 1 to correct successful-reply receipt. Patch2 also corrects the probe From construction, compact Redis JSON serialization and internal-trunk exclusion. Patch1 uses space-bearing raw error labels and the earlier serialization path. Do not apply patch2 mock-probe acceptance to this release; verify actual callback receipt and Redis persistence for the deployed image before relying on its status.

Observability

Trunk health is exposed as logs, events, and a REST endpoint — there is no Prometheus metric series for trunk health. To monitor in SigNoz, query the sip_trunk_health log type or the alarm.sip_trunk_health span. See SIP trunk log reference for the log catalog and Monitoring in SigNoz for query patterns.

See also

• SIP trunk log reference — every log line the monitor emits.
• SIP trunks (admin) — inbound trunk configuration.
• Egress trunks — outbound trunks (not health-monitored).
• Service logs — alarm.sip_trunk_health span details.