Application and call-flow logging

Retained Next source

This page preserves the merged Next documentation snapshot at e1ddcf6940adb7b619ce509063749e740eecc795. It can describe implementation beyond the source revision selected for the rebuilt Next guides. Retain the implementation-scope notes below and confirm your deployed version before applying these details. This is not a current-release availability or deployment claim. For the task-based entry point, see the rebuilt guide.

Use this page to understand what is logged during a call’s application and media path, what each log level means, and how to correlate those logs with SIP signaling. It covers TelSys (voice PBX), TelPro (SIP edge), and RTPEngine (media).

For JSON field names, SigNoz queries, and per-service *_LOG_LEVEL variables, see Monitoring in SigNoz. For SIP ladder captures and response codes, see SIP signaling reference.

Purpose

Structured logs let operators reconstruct:

• Call lifecycle on the voice PBX (answer, AI session, hangup, transfer)
• Edge routing and call-control outcomes (trunk/number selection, outbound transfer, BYE handling, failover)
• Media quality and media-layer failures (MOS, ICE/DTLS/SRTP, DTMF)

Together with the Debug SIP ladder, these logs support end-to-end investigation of a single conversation.

Log levels

SigNoz uses canonical uppercase severity labels. Raw Pino JSON can retain lowercase level values; the collector normalizes telemetry severity. See severity casing and historical records.

Level	Meaning
ERROR	Failed call path or hard failure (auth reject, media/control failure).
WARN	Degraded but recoverable (failover, sticky miss, non-SUCCESS transfer, poor MOS).
INFO	Normal lifecycle milestones (call start/end, successful route, hangup).
DEBUG	Troubleshooting detail (session teardown chatter, temporary verbosity).
TRACE	Rare deep diagnostics.
FATAL	Process- or edge-level unrecoverable failure.

Control verbosity with the global LOG_LEVEL or service overrides such as TELSYS_LOG_LEVEL, TELPRO_LOG_LEVEL, RTPENGINE_LOG_LEVEL, and TELPHI_LOG_LEVEL. See Per-service log levels and the environment variable index.

SIP ladder capture is independent of general log verbosity: lowering info/warn filtering does not remove sip_message captures used by the Debug SIP ladder.

Correlation

Use the same identifiers across TelPro, TelSys, media, and application logs:

Identifier	Use
trace_id	Distributed trace across edge, voice, and app logs.
sipCallId	SIP Call-ID (also shown as call id in TelWeb / Debug).
Conversation	TelWeb conversation id when investigating from a call detail page.

Start from TelWeb Conversations → call detail → Debug (Logs / Spans / SIP ladder), then search SigNoz with the same trace_id, Call-ID, or transactionId.

What is logged

TelSys (voice PBX)

Call and AI-session lifecycle events (typically info):

Area	Examples	Typical level
Call entry	Call start, channel created, channel answer	info
AI session	Session enter / exit with the voice AI app	info
Hangup	Hangup with cause / optional hangup reason	info
Transfer	Transfer initiated; transfer result	info / warn if not SUCCESS
Dialplan errors	Error-handling paths	error

Operational bootstrap and config notes use operational log records (see Structured logging schema).

TelPro (SIP edge)

Routing and call-control outcomes (not the raw SIP frames — those are in the SIP ladder):

Area	Examples	Typical level
Ingress / routing	Trunk identified, number/customer route, server selected	info
Rejects	Unknown trunk, TLS/auth failure, unverified internal source	warn / error
Outbound / transfer	Outbound or direct-URI transfer detected or rejected	info / warn
BYE / disconnect	Carrier-bound BYE routing; loop prevention when edge misroutes	info / warn / error
Failover	TelSys server failure, failover attempt or abort	warn / error
Media control (via TelPro)	RTPEngine offer / answer success or failure	info / error

RTPEngine (media)

Data-plane media outcomes. Control-plane offer/answer commands are logged from TelPro (see SigNoz monitoring).

Area	Examples	Typical level
Call quality	End-of-call stats (rtpengine_call_stats: MOS, jitter, loss)	info; warn / error when MOS is poor
Media errors	ICE / DTLS / SRTP failures	error
Resource errors	Port / socket exhaustion	error
Control errors	Unknown call, offer/answer rejected at the media engine	error
DTMF	RFC4733 media DTMF events	info
Session end	Media session teardown notices	debug

Transaction logging schema

Next implementation scope

The transaction fields and explicit security-event tagging described below refer to platform PR 554 (internal implementation reference), reviewed at commit 300e8a9956ada4ac628fd5fceda13855510a73b0. They are not a statement that the current released platform or a customer deployment contains that change. Confirm the deployed implementation before relying on these fields; source checks do not establish live ingestion or delivery.

Instrumented application-level interactions within a conversation emit transaction records for the legs their producer observes. An outbound exchange can produce request-out and response-in records, but transport failure, missing context or an uninstrumented host can leave a partial sequence. The implemented paths include bot traffic, speech services, webhooks and HTTP tool calls; Limits and gaps describes coverage boundaries.

Transaction records are ordinary structured log entries extended with the properties below. There is no parallel log format and no separate logType, so existing SigNoz queries, TelWeb log-type chips, and the AppLog sink are unchanged; the new properties are additive and appear only on records that carry them.

Properties

The helper requires category, logpoint, component and operation; other context and transport fields are conditional. Undefined inputs are omitted, not emitted as empty values. Aliases are added only when their source value exists and do not overwrite an explicit value. “Response only” means the field is eligible on a response, not guaranteed on every response.

Property	Type	Value and notes	Present on
category	string	bot, stt, tts, sip — plus the Delphi extensions webhook (a webhook exchange that is not a bot platform call) and http (any other outbound HTTP call made on behalf of a conversation). A transaction record gets its category from the emitting service; the collector sets sip on SIP captures, which are not transaction records.	All legs
logpoint	string	request-in, request-out, response-in, response-out. See directions.	All legs
service	string	Emitting component (telphi, telapi, tasker, telpro, …).	When available from producer/context
server-name	string	Instance that executed the transaction (the container host name).	When available from producer/context
operation	string	Executed operation, for example create_conversation, send_message, synthesize, fetch. On system-style records it is also written to the structural action field.	All legs
verb	string	HTTP verb of the request (GET, POST, …); WS for a websocket lifecycle event.	When available from producer/context
url	string	Target URL. Credentials in the query string are scrubbed before the record is written.	When available from producer/context
botName	string	Bot or app that owns the conversation.	When available from producer/context
transaction-Id	string	Alias of transactionId when supplied by the producer/context; its scope depends on that producer. Do not infer one universal per-exchange or per-conversation lifetime.	When available from producer/context
conversation-id	string	Delphi conversation id. Alias of conversationId.	When available from producer/context
call-id	string	Alias of callId, falling back to sipCallId when callId is null/undefined. callId can be a Delphi call/session ID or a producer-specific SIP ID; this alias is not universally the SIP Call-ID.	When available from producer/context
trace-id	string	Distributed trace id. Alias of trace_id, and the string to search on — the OTLP collector removes the trace_id attribute itself.	When available from producer/context
caller-id	string	Identifier of the credentials in use (for example the variable key name) — not the caller’s number, and never the secret value.	When available from producer/context
payload	string	Request or response body, redacted and truncated. See Payload and detail policy.	Flag or failure
headers	object	Request or response headers, redacted. Same switch as payload.	Flag or failure
status	string	success or error. The ticket’s ok is spelled success here, because the logger derives the log level from this value.	Response legs when supplied
status-code	number	Response status code. Alias of statusCode.	Response legs when supplied
duration	number	Response latency in milliseconds. Alias of durationMs — on a response leg only, because durationMs means something else on other records.	Response legs when supplied
timestamp	number	Event time in epoch milliseconds, the platform convention, for filtering and ordering.	All records
timestamp-iso	string	The same time as ISO-8601 with offset, for example 2026-07-28T12:13:06.301+02:00. Delphi writes millisecond precision.	When available from producer/context
attempt	number	1-based attempt number, present only when the request was retried.	Retried requests
bodyOnly	bool	Marks a companion record that carries only a successful response body.	Success path

Available aliases are written alongside their canonical fields; an explicit alias is not overwritten. A missing source identifier remains absent.

logpoint directions

logpoint is read from Delphi’s point of view, so the same category can appear in both directions — for example a bot send_message is request-out, the bot platform’s reply is response-in.

logpoint	Meaning	Example
request-in	Something called us.	Inbound bot-platform webhook at TelAPI.
request-out	We called someone.	send_message to the bot platform, synthesize to TTS, a sandbox tool fetch.
response-in	The answer to our request.	The bot platform’s HTTP reply; a websocket upgrade or close.
response-out	Our answer to an inbound request.	The TelAPI webhook reply.

Finding transaction records only

Transaction records are exactly the records that carry a logpoint, so filter on its presence:

logpoint IS NOT NULL

The retry-control records (provider_request_retry, provider_request_error) deliberately carry no logpoint, so retry noise never appears in that result set. Within it, key alarms on the response-in record that carries status-code: on the success path a companion bodyOnly record may be written after it when the response body was buffered, and that companion has no status-code.

Limits and gaps

• STT has no HTTP path. Both STT providers are websocket-only, so speech-to-text cannot produce an HTTP verb, a request payload, or an ordinary status-code. Its transaction records are the websocket lifecycle (request-out at connect, response-in at open or close) with verb: WS; the upgrade response supplies status-code where there is one.
• A websocket transport yields lifecycle records only — one request-out and one response-in per connection, never one record per frame.
• Payloads are guaranteed on error and best-effort on success. Binary audio and JSON-lines streams are never buffered; such a record carries status, duration and the content type instead of the body.
• SIP capture records are categorised, not transactions. The SIP edge does not emit transaction records: the OTLP collector tags recordKind=sip_message captures with category: "sip". They carry no logpoint, because a SIP frame is not one of the four transaction legs and a fabricated leg would put non-transaction traffic into the transaction-record result set. Correlate a capture by trace_id and SIP Call-ID instead — see Correlation.
• Not every HTTP call is a transaction. Only interactions within a conversation are recorded. Service-level calls such as an access-token fetch are logged as ordinary structured logs.
• The inbound webhook leg carries no headers map. The TelAPI callback projects the headers that identify the exchange onto named fields (transaction-Id, call-id, botName) instead, so a callback token value can never reach a transaction record.
• Flow-engine runner records depend on the host that runs the flow. The engine emits them only when the flow host injects a logger, and never throws when it does not: TelPhi's in-call flow host injects one, TelAPI web-chat sessions do not.

Payload and detail policy

Policy	Behavior
Metadata always	The payload switch does not suppress supplied metadata. URL, verb, status codes, bot identity and handling labels remain conditional on the producer/context; requests do not gain response-only fields.
Bodies on failure	At instrumented failure paths, the producer can force available payloads and headers even with LOG_INTEGRATION_PAYLOADS=false. Missing body/header inputs stay absent; the helper does not reconstruct them.
LOG_INTEGRATION_PAYLOADS	Set to true to also log payloads and headers on successful legs. Default false. Set per service (the same variable name in each service’s deployment), so it can be raised for one service and one short support window. Expect higher volume and cost.
PII	Redaction before SigNoz export is controlled separately — see PII redaction.

LOG_INTEGRATION_PAYLOADS is a volume control, not a privacy control

Transaction payloads are PII-bearing by construction. Speech transcripts and TTS bodies go through the transcript classifier and are stored as [REDACTED] unless ENABLE_PII_LOGGING=true; credentials in headers and query strings (authorization, Ocp-Apim-Subscription-Key, cookies, tokens, API keys) are always redacted, whatever either switch says, and bodies are truncated before storage. As defence in depth the OTLP collector replaces every payload value with [REDACTED] while ENABLE_PII_LOGGING is not true, in each of the three envelopes a record can arrive in (the flat attribute, metadata.payload, pino.payload). The payload flag controls volume; it is not a privacy guarantee. Review emitted fields and effective redaction before enabling it.

Security-relevant events

Security-relevant events are marked with two log attributes, so they can be selected directly instead of by the wording of a message: security_event = true, plus one of four security_category values (authentication, authorization, audit, certificate). Filter on security_event in SigNoz. Security event export has the full contract, the action-to-category mapping, and what is deliberately not exported.

The integration-side refusals that reach this path are Kamailio's trunk-admission rejects, logged as structured warn or error records:

Event	security_category	Refused because
trunk_identify_failed	certificate	The peer's TLS certificate or identity did not verify
trunk_identify_failed	authorization	The trunk is unknown or unauthorized to this platform
trunk_call_rejected	authentication	The trunk requires an X-Call-ID and it was invalid or missing
xcall_invalid	authentication	A Janus call's X-Call-ID had no entry, so it did not resolve
xcall_missing	authentication	A Janus call arrived without an X-Call-ID at all

Application-side refusals — failed sign-ins, session terminations, permission denials — are emitted as security events by the services themselves through @delphi/logger, and carry the same attributes. They are not SIP-level events, so they are not listed here.

Not recorded as structured logs. Five further rejects in kamailio.cfg are written with xlog only. They produce a plain log line, are dropped before the structured log path, and therefore cannot be tagged or exported: the transfer-from-unverified-source rejects (kamailio.cfg:617, :646), authentication misconfiguration (:810), source IP not authorized (:816), and digest username mismatch (:835). There is no BYE self-relay reject at all — the route that would classify one is defined and never called. Making these exportable means emitting them as structured records, which is a change to the voice application and its image.

Where to look

Surface	Best for
TelWeb → conversation → Debug	Per-call logs, spans, and SIP ladder for one conversation.
SigNoz Logs	Cross-call search by trace_id, Call-ID, level, or event.
SIP signaling reference	Which SIP messages were captured and what response codes mean.
Monitoring in SigNoz	Schema keys, log-level env vars, dashboards, and PII.

Integration-specific field mapping

Log field value definitions

These generic value tables describe the Next implementation scope above.

category — the integration domain the event belongs to:

Value	Meaning
stt	Speech-to-text: audio → text (recognize / streaming recognition).
tts	Text-to-speech: text → audio (synthesize / streaming synthesis).
sip	SIP signaling / call control (invite, bye, register, …).
webhook	Delphi extension: a webhook exchange that is not a bot platform interaction.
http	Delphi extension: any other outbound HTTP call made on behalf of a conversation — a flow runner, a sandbox tool fetch.

logpoint — which leg of the exchange the record is. See logpoint directions for the full table; the four values are request-in, request-out, response-in, response-out. The implemented spelling is response-out.

service — the Delphi component that emitted the event:

Value	Component
telpro	SIP edge / SBC (Kamailio).
telsys	Voice PBX (Asterisk).
telphi	Voice AI application (STT / TTS / bot orchestration).
telweb	Web UI.
tasker	Background task runner.
scaler	Autoscaling.
rtpengine	Media engine (data plane; control-plane offer/answer is logged from telpro).
audioproc	Audio processing.
portal	Portal.

operation — the executed operation, per category. Representative values; the exact value is always on the record's operation field, which is also written to the structural action.

category	Representative operations
bot	create_conversation, send_message, end_conversation, refresh_access_token
stt	connect, transcribe (websocket lifecycle — see limits)
tts	synthesize, connect
sip	SIP methods: invite, ack, bye, cancel, register, update, reinvite
webhook	execute (flow-engine webhook runner)
http	fetch (sandbox tool), execute (flow-engine LLM query runner), conversation scoring

Cross-service logging schema

Canonical Node service names: telpro, telsys, telphi, telapi, telweb, tasker, scaler.

Telephony (Kamailio / TelSys) uses a parallel JSON shape with recordKind:

recordKind	Meaning
log	Operational / lifecycle event
trace	Call-lifecycle span-like event (TelSys dialplan)
sip_message	Captured SIP frame for the Debug ladder

App logs from @delphi/logger use logType values such as conversation, model_usage, system, telephony, audio, error, audit, and cdr, plus injected trace_id / span_id. See Service logs and Monitoring in SigNoz.

Transaction records are this same app-log shape — they add the properties above to the existing entries rather than introducing a new logType, which is why no log-type chip, dashboard, or sink changed. Remaining instrumentation gaps require separate implementation and verification.