SIP signaling and trunk log reference
Use this reference to find and read the records that the SIP edge (TelPro) and the trunk health monitor write. It covers structured SIP proxy records, captured SIP messages, rejected calls and trunk health. To decide what a SIP response means, use SIP diagnostics.
Availability: you need access to SigNoz, or Super Admin or Super Viewer to use Admin Settings → Log Search (Trace Debug). Security tagging of rejected calls is available from 0.9.18. From 0.9.18, a call from an unknown address is logged as a warning instead of an error.
Where SIP records land
| Producer | Service in SigNoz | How to recognize the record |
|---|---|---|
| SIP proxy on the SIP edge | telpro | service.component is kamailio; recordKind is log or trace |
| SIP message capture on the edge | telpro | recordKind is sip_message; event is sip_message |
| SIP capture on the Voice host | telsys | recordKind is sip_message; capture point starts with telsys_ |
| Trunk health monitor in TelPhi | telphi | component is sip_trunk_health |
| TelAPI trunk health endpoint | telapi | Messages listed in TelAPI trunk health messages |
The SIP proxy writes its structured records as JSON lines to its container log. The TelPro collector reads them, flattens the JSON into attributes and sends them to SigNoz. How records reach SigNoz is described in Telemetry, SigNoz and alerts.
Structured SIP proxy records
Each structured record carries these fields when they apply:
| Field | Meaning |
|---|---|
timestamp | Milliseconds since the Unix epoch |
logType, recordKind | telephony and log for events, trace for span start and end records |
level | debug, info, warn or error |
component, event | Producing part of the proxy, for example sip-proxy, and the event name |
callId, sipCallId | SIP Call-ID; the collector copies it to call_id and sip.call_id |
msg | Readable description |
sipMethod, statusCode | SIP method and response code; statusCode is copied to sip.status_code |
fromUri, toUri, sourceIp, sourcePort | Addressing of the request |
trunkId, selectedServer, durationMs | Matched trunk, chosen Voice server and measured duration |
instanceId | Edge instance; copied to service.instance.id |
TELPRO_LOG_LEVEL filters which levels are written; see Set the level for each component. The collector removes trace-ID aliases from attributes; to find a call by trace ID, use Log Search (Trace Debug).
Common event values:
| Event | What it records |
|---|---|
call_start, call_end | Start and end of a call through the edge |
trunk_identify_start, trunk_identified, trunk_identify_failed | Matching the request to an inbound trunk |
number_identified, number_identify_failed | Matching the called number |
route_start, server_selected, route_failed | Choosing a Voice server |
load_balance, failover | Load balancing and failover between Voice servers |
trunk_call_rejected | A new call rejected by a trunk rule |
xcall_missing, xcall_invalid | Missing or unknown X-Call-ID on a browser call |
header_applied | A configured header rule changed a header |
trunk_probe_start, trunk_probe_success, trunk_probe_failure | Trunk OPTIONS probes; see Trunk probe records |
telsys_source_fallback, trace_correlation_failed | Fallback classification of a Voice server, or a trace ID that could not be resolved |
SIP message captures
A capture record holds one SIP message as it crosses a capture point. Captures have logType sip_capture, recordKind sip_message, component sip-capture and level info. The collector adds event sip_message and category sip.
| Attribute | Meaning |
|---|---|
sip.capture_point | Where the message was captured; see the list below |
sip.method | Request method |
sip.status_code | Response code, for responses |
sip.call_id | SIP Call-ID |
sip.captured_at_ms | Capture time in milliseconds since the Unix epoch |
sip.raw | The decoded SIP message |
Capture points:
telpro_ingress_from_outside: a message from a carrier or caller arrives at the edge.telpro_egress_to_telsys: the edge forwards a message to a Voice server.telpro_ingress_from_telsys: a message from a Voice server arrives at the edge.telpro_egress_to_outside: the edge sends a message to a carrier or caller.telpro_local_response: the edge answers on its own, for example with a reject. Only the status line is captured.telsys_ingress_from_telproandtelsys_egress_to_telpro: the same messages as seen on the Voice host.
On the Voice host, SIP_CAPTURE_REDACT_HEADERS removes authentication and cookie headers from captures, and SIP_CAPTURE_MAX_MESSAGE_BYTES (default 65536) limits the captured size. Captures still contain telephone numbers in the request line and in From and To. Read Protect diagnostic logs before you share one.
Native proxy lines
The SIP proxy also writes unstructured lines, for example Rejecting call from unknown trunk. These lines have no recordKind. The TelPro collector drops them, so they never reach SigNoz. They stay only in the SIP proxy container's log on the TelPro host, as long as the host's log rotation keeps them.
Rejected calls and security tagging
From 0.9.18, the TelPro collector marks rejected calls with security_event true and a security_category. The optional export collector forwards these records to your security destination; see Verify security event export and retention.
| Reject | Event and level | SIP response | security_category |
|---|---|---|---|
| Source matches no configured trunk | trunk_identify_failed, warn | 403 Forbidden - Unknown Trunk | authorization |
| Trunk TLS requirement not met | trunk_identify_failed, error | 403 with a TLS reason, for example Forbidden - TLS Required | certificate |
| Trunk in maintenance mode | trunk_call_rejected, warn | Configured code, default 503 Service Unavailable | authentication |
Invalid or missing X-Call-ID on a trunk that requires it | trunk_call_rejected, warn | 403 Forbidden - Invalid Call Token | authentication |
Unknown X-Call-ID on a browser call | xcall_invalid, warn | 403 Forbidden - Invalid Call Token when the trunk requires X-Call-ID | authentication |
Missing X-Call-ID on a browser call | xcall_missing, warn | 403 Forbidden - Invalid Call Token when the trunk requires X-Call-ID | authentication |
A trunk_identify_failed record whose msg mentions TLS, a certificate or a peer name gets certificate; any other gets authorization. A tag already set by the producer is never changed.
Rejects without a structured record
Five rejects write only a native proxy line. The collector drops these lines, so they are neither tagged nor exported:
- Outbound transfer REJECTED: source … is not a verified TelSys server, answered with
403 Forbidden - Not a TelSys Server. - Direct-URI transfer REJECTED: source … is not a verified TelSys server, answered with
403 Forbidden - Not a TelSys Server. - Inbound trunk … uses … but auth_user/auth_pass is missing, answered with
500 Inbound Auth Misconfiguration. - Inbound ip_and_digest rejected for trunk …: source IP … not authorized, answered with
403 Forbidden - Source IP Not Allowed. - Inbound digest username mismatch for trunk …, answered with a new digest challenge.
To investigate these, read the SIP proxy container log on the TelPro host for the call's time window. The telpro_local_response capture shows the response status line in SigNoz. The meaning of each response is described in Responses generated by the edge.
Trunk probe records
When OPTIONS health probes are enabled for an inbound trunk, the edge writes one record per probe result. Probe settings are described in Understand the health controls actually available.
| Event | Level | Message |
|---|---|---|
trunk_probe_success | info | SIP OPTIONS probe to sip:<host>:<port> for trunk <id> succeeded (<n>ms, status=<code>) |
trunk_probe_failure | error | SIP OPTIONS probe to sip:<host>:<port> for trunk <id> failed (<n>ms, evcode=<code>) |
Both records have component sip-proxy and recordKind trace. A final 2xx response is a success; any other response, or no response (evcode=0), is a failure. The SipTrunkDown alert counts failed probe spans per trunk; see Alert rules.
The edge also stores the latest result for each trunk for 60 seconds, with ok, rttMs, error (null, no_reply or status_<code>) and checkedAt. The TelPhi monitor reads this result.
Trunk health monitor records
TelPhi writes records with component sip_trunk_health as it evaluates stored edge probe results. The interval, failure threshold and degraded round-trip threshold use the Voice service configuration. Follow Diagnose trunk health to distinguish stale results from a carrier failure.
| Message | Level | Meaning |
|---|---|---|
SIP trunk <id> is unavailable[: reason] | error | The trunk changed to unavailable |
SIP trunk <id> is degraded[: reason] | warn | The trunk changed to degraded |
SIP trunk <id> is available[: reason] | info | The trunk is available again |
Starting SIP trunk health monitor, SIP trunk health monitor started, Stopping SIP trunk health monitor | info | Monitor lifecycle |
Initial poll failed, Poll cycle failed, Failed to load SIP trunks from database | error | The monitor could not evaluate trunks |
Failed to persist health snapshot, Failed to persist health event | error | The result was not saved |
Failed to send trunk health notification | error | The notification could not be sent |
Typical reasons are no probe result from edge and RTT trunkId, serverId, status, previousStatus, reason and rttMs, plus alarm attributes: event.name alarm, alarm.name sip_trunk_health, alarm.severity (critical, warn or info), alarm.state, alarm.previous_state and alarm.resource.id. Status changes also trigger the TRUNK_DOWN, TRUNK_DEGRADED and TRUNK_RECOVERED notifications; TRUNK_RECOVERED is sent when a trunk becomes available again.
TelAPI trunk health messages
The TelAPI endpoint /health/sip-trunks reports trunk health from the stored results. It writes these messages:
- Failed to scan Redis for trunk health keys (error): the endpoint answers
503withRedis unavailable. - Ignoring malformed sip_trunks heartbeat payload (warn): one stored entry could not be read and is skipped.
- Failed to enrich trunk health from DB (error): trunk details from the database are missing from the answer; the health data is still returned.
Find SIP records
- Note the call's time in UTC and its SIP Call-ID, for example
a84b4c76e66710@example.com. - In Admin Settings → Log Search (Trace Debug), search
Call-ID: a84b4c76e66710@example.comover a narrow time range. See Find a trace without an open conversation. - In the SigNoz logs view, filter
service.name = 'telpro' AND sip.call_id = 'a84b4c76e66710@example.com'to list proxy records and captures for the call. - For rejects, filter
service.name = 'telpro' AND event = 'trunk_identify_failed'orevent = 'trunk_call_rejected'. - For trunk health, filter
service.name = 'telphi' AND component = 'sip_trunk_health'and the trunk'strunkId.
To verify, confirm that the records you found share the Call-ID and fall inside the call's time window. If a reject appears in a capture but not as a structured record, it is one of the rejects without a structured record.
Related
- SIP diagnostics: to decide what a routing or termination response means.
- Configure SIP trunks and routing: to change probe and maintenance settings.
- Telemetry, SigNoz and alerts: to open SigNoz and see the trunk alert.
- Verify security event export and retention: to forward tagged rejects.
- Protect diagnostic logs: to share captures safely.