SIP trunk log reference
Use this page when you see a SIP trunk log line in SigNoz or container stdout and need to know what triggered it and which fields to inspect.
Log entry titles are colored by severity: ERROR red, WARN yellow, INFO green, DEBUG blue.
For the SIP signaling ladder and response codes, see SIP signaling reference. For general log levels and SigNoz queries, see Monitoring in SigNoz.
TelPhi
Source: apps/telphi/src/services/sip-trunk-health-monitor.ts (runs on the Voice host, polls Asterisk AMI for inbound SIP trunk endpoint status). For Asterisk/PJSIP docs see docs.asterisk.org.
Lifecycle
Starting SIP trunk health monitorINFOsip_trunk_health
When it fires
At the very beginning of SipTrunkHealthMonitor.start().
What it means
The monitor is booting and will immediately try to connect/login to AMI, then run the first poll.
Payload fields
| Field | Value |
|---|---|
serverId | Voice server / container identifier. |
amiHost | AMI host being connected to. |
amiPort | AMI port (usually 5038). |
intervalMs | Poll interval in milliseconds. |
SIP trunk health monitor startedINFOsip_trunk_health
When it fires
After the first AMI connection attempt and the recurring poll interval have been scheduled.
What it means
Startup is complete. Subsequent health checks run on intervalMs. The first poll is fire-and-forget, so a slow first probe does not block this message.
Payload fields
| Field | Value |
|---|---|
serverId | Voice server / container identifier. |
Stopping SIP trunk health monitorINFOsip_trunk_health
When it fires
At the start of SipTrunkHealthMonitor.stop(), for example during graceful shutdown.
What it means
The monitor is closing its AMI socket and clearing pending requests. No further polls or state transitions will be emitted until the monitor restarts.
Payload fields
| Field | Value |
|---|---|
serverId | Voice server / container identifier. |
Poll and AMI failures
Initial poll failedERRORsip_trunk_health
When it fires
The fire-and-forget first poll in start() rejects.
What it means
The monitor could not finish the first endpoint sweep. This is usually a symptom of an AMI connection/login failure (see AMI connection/login failed) or an Asterisk pjsip show endpoint command failure. The next poll is already scheduled and will retry.
Payload fields
| Field | Value |
|---|---|
error | Error object with message and stack. |
Poll cycle failedERRORsip_trunk_health
When it fires
A recurring interval poll rejects.
What it means
The whole poll cycle aborted before evaluating all endpoints. Individual endpoint evaluation errors are swallowed per-endpoint, so this message points to a broader problem such as AMI disconnection or an unexpected exception in the poll loop.
Payload fields
| Field | Value |
|---|---|
error | Error object with message and stack. |
AMI connection/login failed; will retry on next pollWARNsip_trunk_health
When it fires
ensureConnected() fails to open the TCP socket to AMI, or the AMI Login action returns a non-success response.
What it means
The monitor cannot reach Asterisk. All known trunks eventually become unavailable once consecutive-failure thresholds are hit. Check that amiHost/amiPort are reachable and that the AMI manager credentials are correct.
Payload fields
| Field | Value |
|---|---|
serverId | Voice server / container identifier. |
amiHost | Target AMI host. |
amiPort | Target AMI port. |
error | Failure message, for example ECONNREFUSED or AMI login failed: Authentication failed. |
AMI socket errorERRORsip_trunk_health
When it fires
The open AMI socket emits an error event after login succeeded.
What it means
A low-level TCP/AMI error occurred on an already authenticated socket. The monitor disconnects and reconnects on the next poll. Correlate with AMI connection lost and subsequent polls.
Payload fields
| Field | Value |
|---|---|
error | Node.js Error from the socket. |
AMI connection lost; reconnecting on next pollWARNsip_trunk_health
When it fires
The AMI socket closes after the monitor was logged in.
What it means
AMI became unreachable mid-session. The monitor clears the logged-in flag and will reconnect on the next poll. If this repeats, inspect Asterisk logs and network path between TelPhi and AMI.
Payload fields
| Field | Value |
|---|---|
serverId | Voice server / container identifier. |
Persistence failures
Failed to load SIP trunks from databaseERRORsip_trunk_health
When it fires
loadEnabledTrunks() cannot read the SipTrunk table (for example, the database is unreachable).
What it means
The monitor does not know which trunks to watch, so it marks all previously known trunks as unavailable due to lost visibility and returns an empty trunk list for this poll. Investigate database connectivity and the TelPhi → Prisma path.
Payload fields
| Field | Value |
|---|---|
serverId | Voice server / container identifier. |
error | Error object. |
Failed to persist health snapshotERRORsip_trunk_health
When it fires
persistSnapshot() cannot upsert a row in SipTrunkHealthSnapshot.
What it means
The latest probe result was not saved. TelAPI's trunk-health endpoint may serve stale data until the next successful snapshot. The event is still recorded separately, so alarms/notifications still fire.
Payload fields
| Field | Value |
|---|---|
serverId | Voice server / container identifier. |
trunkId | Affected trunk identifier. |
error | Error object. |
Failed to persist health eventERRORsip_trunk_health
When it fires
persistTransition() cannot insert a row in SipTrunkHealthEvent.
What it means
A status transition happened but the event audit row could not be written. The notification may still have been sent, but the history view in TelWeb or the API may be missing this transition.
Payload fields
| Field | Value |
|---|---|
serverId | Voice server / container identifier. |
trunkId | Affected trunk identifier. |
error | Error object. |
Status transitions
SIP trunk {trunkId} is unavailableERRORsip_trunk_health
SIP trunk {trunkId} is degradedWARNsip_trunk_health
SIP trunk {trunkId} is availableINFOsip_trunk_health
When it fires
After evaluating an AMI endpoint probe and updating the trunk state. The actual log message follows the pattern SIP trunk {trunkId} is {status} and may include a reason.
What it means
| Status | Meaning |
|---|---|
available | Endpoint is reachable and healthy. If the previous status was not available, this is a recovery. |
degraded | Endpoint is reachable but round-trip time exceeded degradedRttMs, or another non-fatal quality issue was detected. |
unavailable | Endpoint is unreachable, the AMI command failed, or consecutive failures crossed the configured threshold. |
The monitor uses consecutive-success/failure counters before flipping the evaluated status, so a single bad probe does not immediately mark a trunk unavailable.
Payload fields
| Field | Value |
|---|---|
serverId | Voice server / container identifier. |
trunkId | Trunk identifier. |
status | New status: available, degraded, or unavailable. |
previousStatus | Status before this transition. |
reason | Human-readable probe reason, for example AMI command failed: Permission denied or Empty endpoint status from Asterisk. |
rttMs | Round-trip time of the AMI probe in milliseconds, or null. |
event.name | Always alarm. |
alarm.name | sip_trunk_health. |
alarm.severity | critical for unavailable, warn for degraded, info for recovery. |
alarm.state | Same as status. |
alarm.previous_state | Same as previousStatus. |
alarm.resource.type | sip_trunk. |
alarm.resource.id | Same as trunkId. |
alarm.server.id | Same as serverId. |
A sendNotification('SIP_TRUNK', ...) call is also triggered for these transitions with codes TRUNK_DOWN, TRUNK_DEGRADED, or TRUNK_RECOVERED.
Notification failures
Notification {code} not sent: {message}WARNsip_trunk_health
When it fires
sendNotification returned { success: false } for a trunk health state change.
What it means
The alarm state transition was detected, but the configured notification channel did not accept it (for example, a missing channel, invalid recipient, or downstream provider error). The snapshot/event rows are still persisted; only the alert delivery failed.
Payload fields
| Field | Value |
|---|---|
serverId | Voice server / container identifier. |
trunkId | Affected trunk identifier. |
code | TRUNK_DOWN, TRUNK_DEGRADED, or TRUNK_RECOVERED. |
message | Failure message returned by the notification service. |
Failed to send trunk health notificationERRORsip_trunk_health
When it fires
sendNotification threw an exception.
What it means
The notification subsystem itself failed while trying to deliver a trunk health alarm. This is more severe than a rejected notification; it may indicate a bug or outage in the notification service.
Payload fields
| Field | Value |
|---|---|
serverId | Voice server / container identifier. |
trunkId | Affected trunk identifier. |
code | TRUNK_DOWN, TRUNK_DEGRADED, or TRUNK_RECOVERED. |
error | Error object. |
TelAPI
Source: apps/telapi/src/routes/trunk-health.ts (serves the /health/sip-trunks REST endpoint used by TelWeb and operators).
Failed to scan Redis for trunk health keysERROR
When it fires
The Redis SCAN for voiceai:services:*:sip_trunks keys fails.
What it means
TelAPI cannot reach Redis or the key pattern scan failed. The endpoint returns 503 Redis unavailable and no trunk health data is served. Check Redis connectivity and the getRedisClient() path.
Payload fields
| Field | Value |
|---|---|
err | Error message string. |
Ignoring malformed sip_trunks heartbeat payloadWARN
When it fires
A value stored under a :sip_trunks Redis key is not valid JSON.
What it means
One Voice server's heartbeat payload is corrupt. The key is skipped so the rest of the fleet's data can still be returned. If many keys are malformed, investigate the writing service (TelPhi heartbeat / ServerHeartbeat).
Payload fields
| Field | Value |
|---|---|
key | Full Redis key that contained the bad payload. |
Failed to enrich trunk health from DBERROR
When it fires
The optional database enrichment step (looking up friendly trunk names from SipTrunk) fails.
What it means
The endpoint still returns Redis-derived trunk health, but trunkName fields may be missing. Operators can still identify trunks by trunkId. This is best-effort; the endpoint does not fail the whole request.
Payload fields
| Field | Value |
|---|---|
err | Error message string. |
Tooling
Source: tooling/scripts/import-twilio-numbers.ts. These lines are not structured service logs and do not reach SigNoz; they appear in the terminal when running the import script.
TWILIO_TRUNK_SID not set — skipping SIP configurationERROR
When it fires
The script is run without the optional TWILIO_TRUNK_SID environment variable.
What it means
Phone numbers will be imported into the platform, but Twilio SIP trunk routing will not be configured on them. Set TWILIO_TRUNK_SID if you need the numbers attached to a trunk.
Configuring SIP trunk routing (trunk: {trunkSid})...INFO
When it fires
The script starts attaching imported numbers to the configured Twilio trunk.
What it means
The SIP routing step is beginning. Each number is checked against the trunk's current members and only missing ones are added.
{phoneNumber} — already on trunk, skippingINFO
When it fires
A target number is already associated with the configured Twilio trunk.
What it means
No change was made for this number. This is expected on re-runs of the script.
{phoneNumber} — SIP trunk configuredINFO
When it fires
A target number was successfully added to the configured Twilio trunk.
What it means
Inbound calls to this Twilio number will now route to the platform via the configured trunk SID.
Where to look
| Surface | Best for |
|---|---|
| SigNoz Logs | Filter attributes.alarm.name = "sip_trunk_health" or attributes.logType = "sip_trunk_health" for status transitions and connection failures. |
| TelWeb → platform admin → SIP trunks | Trunk configuration and historical health snapshots. |
TelAPI /trunk-health | Real-time fleet-wide trunk status aggregated from Redis heartbeats. |
| Voice container stdout | docker compose logs voiceai-telphi when SigNoz is unavailable. |
Useful SigNoz queries
attributes.alarm.name = "sip_trunk_health" AND attributes.alarm.state = "unavailable"
attributes.logType = "sip_trunk_health" AND attributes.alarm.resource.id = "twilio_us"
service.name = "telapi" AND attributes.msg = "Failed to scan Redis for trunk health keys"
See also
- SIP signaling reference — SIP methods, response codes, and capture points.
- Monitoring in SigNoz — structured logging schema, per-service log levels, PII redaction, and dashboards.
- Application and call-flow logging — call lifecycle logs from TelSys, TelPro, and RTPEngine.
- Voice operations — TelSys, ARI, and Voice host operations.