Diagnose SIP routing and termination

Use this procedure for an unexpected SIP routing response, when a call ends with an unexpected cause, an expected diagnostic header is missing, or optional termination headers are dropped.

Availability: Operators with access to service logs, SigNoz and SIP captures. Administrators can supply the trunk and route configuration but cannot read the edge captures themselves.

Before you start

Identify the deployment release, affected conversation or channel, and SIP call leg. Obtain access to the relevant application logs and SIP capture. Record the time of the event and whether the call had been answered. Avoid combining captures from different call attempts or transfer legs.

The termination reference defines cause values, generated headers, and default optional-header limits. A SIP cause inside Reason is not the same thing as the response status of the message carrying it.

Identify a routing response first

Confirm which component sent the response. A carrier response with the same numeric code is not automatically the Delphi branch described below. The table lists the responses that most often need a configuration check.

Response from the platform	Condition	What to do
403 Forbidden - Unknown Trunk	No configured trunk matched the request	Check inbound identification against the request's source, From and User-Agent
403 Forbidden - TLS Required	The selected trunk requires TLS and the request used a different transport	Compare the incoming transport with the selected trunk policy
500 Inbound Auth Misconfiguration	Required digest credentials are missing	Ask the authorized administrator to correct the trunk credentials
403 Forbidden - Source IP Not Allowed	The ip_and_digest source-IP check failed	Compare the source address with the allowed IP/CIDR configuration
404 No Outbound Route	Required route context or the corresponding egress payload is missing	Check outbound selection and its applied runtime state
404 No Outbound Identity	The resolved route has no outbound base-number identity	Check the route's base identity and endpoint configuration
503 Routing Error	An egress routing-data lookup failed	Investigate the routing dependency before changing the user's configuration
503 Service Unavailable during an instance drain	The TelPro instance is draining and rejects every new INVITE	Check the instance state with the telephony edge procedures; not every 503 is a drain
Trunk's configured response (default 503 Service Unavailable)	The trunk is in maintenance mode (from 0.9.18); the edge logs trunk_call_rejected	Check the Maintenance column and trunk maintenance

The complete list of responses that the edge generates itself is in responses generated by the edge.

A 481 response concerns the relevant transaction/dialog context; it does not, on its own, prove that a SIP method is unsupported. Similarly, a successful OPTIONS response is a probe result, not proof that a real call passed routing and authorization.

For a configuration correction, follow Configure SIP trunks and routing, then return to the verification step. For termination/header questions, continue below.

Diagnose trunk health

Use this check when a trunk alarm disagrees with call behavior. The monitor covers enabled external inbound trunks; it does not test egress trunks or change routing.

Availability: querying GET /health/sip-trunks requires an internal API token with read:trunk_health permission and access to the deployment's TelAPI endpoint. A browser session or an ordinary application API key does not supply this permission.

1. Confirm the edge probe configuration and target. The edge writes OPTIONS results to Redis; the Voice service evaluates those results. An absent, invalid or stale result produces no probe result from edge.
2. Request GET /health/sip-trunks with the internal token in the Authorization: Bearer header. The response contains generatedAt, count and servers. A server identifies its serverId, lastHeartbeat, source and trunks; each trunk includes trunkId, status, reason, rttMs and lastCheckedAt.
3. Check freshness before interpreting status. The server's source is redis, db or heartbeat_only; a database fallback is not a fresh probe. A newly discovered trunk starts as available before its first successful measurement. The monitor also keeps the previous status until enough consecutive failed or successful evaluations occur. A degraded result changes state immediately. Read the timestamps, reason and round-trip time together.
4. If tuning is necessary, have the deployment operator change the SIP_TRUNK_HEALTH_* settings in the environment reference, retain the previous values and restart the Voice service. The threshold counts evaluations, which can read the same edge result more than once. Changing it does not change the edge's probe interval. Restore the previous settings and restart if the new thresholds conceal a known failure or cause excessive alarms.
5. Verify a new probe result, a fresh monitor heartbeat and the expected notification after a controlled status transition. Then verify a real call separately. A 503 Redis unavailable response means the health endpoint cannot read Redis; inspect that dependency before changing trunk routing.

Use the trunk health log records to follow monitor transitions and notification failures.

1. Find the selected cause

Look for termination_signal_selected from termination_signaling_service for the affected channel. Its metadata includes method, causeId, category, sipCause, q850Cause, and reasonTextLength.

Compare the selected cause with the cause mapping. Use application events around the same time to determine why that cause was selected. Numeric causes alone cannot distinguish every failure; STT and TTS errors, for example, share the same SIP and Q.850 values.

If this event is absent, first confirm the channel, release, logging coverage, and whether the action used this termination path. Absence alone does not prove the call did not end.

2. Distinguish header preparation from publication

Event	What it tells you	Then check
optional_termination_headers_sanitized	Optional names or values were normalized, or entries removed	Compare intended headers with normalized output and the reserved-header rules
optional_termination_headers_dropped	The optional set exceeded the platform budget	Reduce the set according to the header limits
sip_call_id_unavailable	Header intent could not be indexed using a SIP Call-ID	Check that you are inspecting the correct dialog and whether the channel's Call-ID was available
sip_egress_intent_published	The service published a header intent	Continue to the outgoing SIP capture; this event is not proof of peer delivery
sip_egress_intent_failed	Header intent publication failed	Inspect the accompanying failure and dependencies; termination may still have continued

Record the event's method and identifiers rather than assuming all events refer to the original inbound dialog. A transfer can create a new outbound leg.

3. Inspect the corresponding SIP message

Compare the observed message with the generated header format:

• For an answered termination, inspect the relevant BYE path.
• For a rejection before answer, inspect the final response; do not require a BYE as evidence.
• For an in-dialog REFER, inspect that request.
• For a new transfer INVITE, look for X-Transfer-Reason on the new leg rather than assuming the old dialog's Reason was copied.

A service log, a platform-side capture, and the peer's received message are different evidence. Record the capture point and direction when reporting a discrepancy. The SIP log reference describes the capture records and how to find them.

Follow an action from intent to observed outcome

For a disputed hangup or transfer, separate the requested action, header preparation, edge processing and observed SIP transaction. A call disappearing from the UI or a hangup_success event is not proof that the remote peer received the intended headers.

Selected termination. Record termination_signal_selected, method/cause, channel and time. The service selected a signal; this does not prove the peer saw it.

Published header intent. Record sip_egress_intent_published, method, intentId, headerCount and indexedBySipCallId. The publish helper returned; this does not prove that the edge found or materialized those headers.

Edge application. Read Applied SIP egress intent … for … call=… together with adjacent warnings. The edge reached the end of the intent application path. The same path logs Failed to materialize SIP egress intent … if msg_apply_changes() fails, then still emits the applied line. The applied line alone is insufficient.

Platform-side SIP capture. Record recordKind=sip_message, logType=sip_capture, sipCallId, capturePoint, method, statusCode, time and the raw message where authorized. A message was observed at that capture point. Pair the request and response on the same SIP transaction; a response on another leg or method is not an acknowledgement of this action.

Peer/application outcome. Collect relevant peer-side evidence and the expected disconnected/transferred state. This confirms the separate outcome you are investigating. A platform-local capture or API acknowledgement does not establish every downstream business/media result.

Investigate a BYE without assuming a universal three-event guarantee

1. Start with an existing affected conversation/channel and a bounded time window. Record the relevant SIP Call-ID and dialog, not only the higher-level conversation ID. A transfer can add a separate SIP leg; keep its identifiers distinct.
2. Check the selected cause and header-publication events. For an indexed BYE, expect the method to be BYE and inspect indexedBySipCallId; sip_call_id_unavailable means the lookup index could not be formed. The intent may still have been stored under its generated ID, so do not equate the warning with no Redis write at all.
3. Match the intent ID and method to edge processing for that SIP Call-ID. Inspect lookup/method-mismatch and materialization warnings before relying on the applied line. The edge applies action headers only to requests from its private 10.* source path, so deployment placement and actual routing matter.
4. Find the corresponding outbound BYE capture and matching response. Inspect the raw SIP transaction identifiers where available, including CSeq method/number, rather than counting any nearby 200. A matched response from the relevant peer supports acknowledgement of that transaction; independently confirm the expected call state and headers.
5. If the chain stops, investigate that boundary. Missing capture can also mean missing instrumentation, ingestion, permissions or an incorrect query. Do not conclude “the carrier received nothing” solely because one stored log search returned no rows.

hangup_success is emitted after the platform's hangup call returns successfully. It does not itself verify the carrier-facing transaction. Intent publication is deliberately fail-open: sip_egress_intent_failed can coexist with a completed hangup. Redis intent/index entries expire after 15 minutes; inspect timely evidence rather than expecting them to remain as a durable audit trail. The publisher awaits its Redis pipeline but does not inspect each command's returned error slot, so its success log is not an independent Redis integrity check.

For a header that the peer did not receive, check whether its name was changed by proxy-compatible header-name normalization. Compare the original callback, the normalized and merged headers, the runtime result and the peer capture under the same call identifier. For HangUp, confirm the BYE and its matching peer 200 OK; for Transfer, confirm the REFER and the destination outcome described below. If dispatch is logged but the peer sees no request, check the edge diagnostics for invalid header name and parse_headers failed.

Check terminal-command dispatch

From 0.9.18, a hangup or transfer runtime command that is rejected or fails is reported in the logs. Two events mark the dispatch of such a terminal command:

Event	What it tells you
terminal_command_dispatch_started	The command handler started after pending playback finished. Metadata: channelId, commandType, commandSource, sipHeaderCount.
terminal_command_dispatch_result	The handler returned. runtimeStatus and actionSuccess record the result. A rejected or failed command is logged at error level with failureMessage.

An error-level result means the command did not run as requested; it is not proof of a disconnect or transfer. sipHeaderCount counts the bot's headers before the configured headers are merged; compare sip_headers_prepared for the merged set. For a header-budget problem, follow the command from dispatch through header preparation to the observed BYE or REFER and the peer response for the same call.

Follow the transfer method and the new leg

For an in-dialog REFER, the intent is indexed by the existing SIP Call-ID. For a new INVITE, the platform cannot know the new SIP Call-ID in advance: it supplies an X-Delphi-Egress-Intent-ID bootstrap value, which the edge uses to bind the new request and then removes. Do not expect that internal bootstrap header at the carrier or require indexedBySipCallId=true on the original INVITE publication.

Use transfer routing to establish the selected method, destination and egress path, then inspect that method's request/response and the intended transfer result. The generated new-leg header is X-Transfer-Reason; the original dialog's Reason is not automatically its substitute. A successful REFER request response is not by itself evidence of a completed destination conversation. Keep result-aware notification/outcome evidence, when applicable to the configured transfer path, distinct from request acceptance.

Transfer header-budget failure can return transfer_blocked with sip_header_budget_exceeded before dispatch. That is different from fail-open intent publication failure. Use the termination/header reference and observed method before deciding whether missing optional headers or an entirely blocked action is the problem. For a rejection before answer, follow the final-response path rather than demanding a BYE.

Use evidence gaps as diagnostic signals, not assumed deployed alerts

An intent without a matching observed action, publication/index warnings, or a rise in dropped headers can inform an operator's alert design. They are not a claim that this deployment has those alerts installed. Counting intents minus captures is not a reliable delivery proof without accounting for retries, duplicate captures, multiple legs, time windows and telemetry loss. Establish the actual alert definition, data source and tested behavior separately; preserve the bounded event chain for investigation through logs.

4. Send configuration questions to the right owner

If the destination, outbound identity, or selected route is unexpected, inspect the transfer-routing configuration before changing diagnostic headers. Inbound identification and outbound egress selection are separate configuration questions.

If the intended action came from a flow, identify that control's configuration and permissions.

5. Verify after a correction

Repeat the affected scenario with a test destination approved for your environment. Compare the selected cause, header publication result, and observed SIP message on the same relevant call leg. Confirm the application outcome independently—for example, that the channel disconnected—not merely that a header-publication log exists.

For a remaining discrepancy, retain the release, scenario, time, channel/conversation identifier, SIP Call-ID where available, capture point, selected cause, publication event, and observed message. Share only the diagnostic data appropriate for the recipient; recordings, transcripts, credentials, and unrelated call data are not needed for this comparison.

For transport, reliable provisional response, codec or network assumptions, consult the SIP signaling and media reference before interpreting a successful probe as proof of a working call.

Investigate a 100rel configuration mismatch

Use this check when the peer's reliable provisional response requirements differ from the behavior seen during call setup.

1. Identify the TelSys instance handling the call and record its deployed version. Keep evidence for both sides of the TelPro-to-TelSys boundary; one peer-facing capture does not establish the internal endpoint's configuration.
2. Ask the deployment operator to check that instance's effective SIP_100REL_MODE. Compare it with the canonical values and fallback behavior, and check the startup log for a warning about an unrecognized value.
3. Confirm the generated and loaded 100rel setting for the TelPro-facing TelSys endpoint. An environment value or a generated file alone does not show which configuration the running process is using. Resolve an environment/load mismatch through the deployment's configuration procedure before retesting.
4. If the setting matches the intended peer requirements but setup still fails, collect the complete provisional-response/PRACK exchange and final result for each relevant call leg. Include timestamps, deployed versions, intended mode, observed endpoint setting and sanitized signaling when escalating the mismatch.

Verify the result with a controlled call using the same peer and call path. A successful OPTIONS probe does not exercise this negotiation.

Diagnose missing or unexpected keypad input

1. Identify the collecting stage: pre-flow extension selection, the SIP flow's dtmf_collect node, or a separate provider/browser input path. The collection reference explains why their retry rules differ.
2. For a SIP call, inspect telephone-event negotiation and the relevant call leg using the SIP media reference. A keypad tone heard locally is not proof that a digit event reached the collector.
3. Correlate the channel and time with dtmf_received and the active collection node. Input before the collector registers its listener is not evidence of input accepted by that node. Verify prompt timing rather than changing retry settings to compensate for missing transport events.
4. Compare the result and transition with collection outcomes. digits_collected can represent partial input. A no-input timeout log can accompany aborted; check the actual signal. After an abort, a previous context value can remain and must not be treated as a newly collected answer.
5. Reproduce with non-sensitive test digits, exercising silence, partial input and the termination key. Check downstream validation separately from collection. Retain the release, node identifier, configured limit/deadline/attempt count, relevant call leg, observed signal and result length; omit real keypad values.

If transport events are missing, escalate with the negotiated DTMF method and sanitized call-leg evidence. If events arrive but the flow takes an unexpected branch, investigate collection outcome and transition configuration first.

Diagnose a caller media timeout

The media-connection monitor checks received RTP on the caller SIP leg, not the amount of recognized speech. A silent caller sending RTP is different from a network path delivering no packets. Bot output and STT/TTS activity do not establish that caller packets are arriving.

1. Correlate the call/channel and media_connection system events. monitor_started records the effective warning/disconnect milliseconds. Compare these with the platform Audio connection settings, allowing for the call's snapshot and any application warning override.
2. A warning event means the warning threshold was reached. A later recovered event follows renewed incoming RTP; it is not proof of successful transcription. A recognized hold pauses the watchdog and resets its timing; unhold starts a fresh interval.
3. On termination, distinguish no_media (no caller RTP observed since monitoring began) from media_connection_broken (RTP had arrived and then stopped). The conversation termination reason is Media, with that detail. Check negotiated addresses/ports, RTP forwarding, firewall/NAT and the caller path using the earlier SIP/media procedure.
4. Treat rtp_observation_failed as a telemetry problem, not evidence of missing packets: a failed statistics request does not itself establish a media timeout. Check ARI access and the next successful observation. prompt_unavailable means warning audio could not be prepared; timeout enforcement remains active. disconnect_failed needs termination-path investigation.

Do not extend the platform deadline merely to hide a broken connection. Preserve a bounded trace, effective timing and media evidence, correct the actual path, then verify a new controlled call. Do not deliberately interrupt a live customer call to exercise this check.

Related

• SIP log reference: to find capture records, trunk events and their fields.
• SIP termination reference: to interpret cause values and optional-header limits.
• SIP signaling and media: to compare observed signaling with the configured protocol behavior.
• Configure SIP trunks and routing: to correct trunk, pool or base-number configuration.