Application and call-flow logging
Use this page to understand what is logged during a call’s application and media path, what each log level means, and how to correlate those logs with SIP signaling. It covers TelSys (voice PBX), TelPro (SIP edge), and RTPEngine (media).
For JSON field names, SigNoz queries, and per-service *_LOG_LEVEL variables, see Monitoring in SigNoz. For SIP ladder captures and response codes, see SIP signaling reference.
Purpose
Structured logs let operators reconstruct:
- Call lifecycle on the voice PBX (answer, AI session, hangup, transfer)
- Edge routing and call-control outcomes (trunk/number selection, outbound transfer, BYE handling, failover)
- Media quality and media-layer failures (MOS, ICE/DTLS/SRTP, DTMF)
Together with the Debug SIP ladder, these logs support end-to-end investigation of a single conversation.
Log levels
All structured logs use lowercase Pino-style labels:
| Level | Meaning |
|---|---|
error | Failed call path or hard failure (auth reject, media/control failure). |
warn | Degraded but recoverable (failover, sticky miss, non-SUCCESS transfer, poor MOS). |
info | Normal lifecycle milestones (call start/end, successful route, hangup). |
debug | Troubleshooting detail (session teardown chatter, temporary verbosity). |
trace | Rare deep diagnostics. |
fatal | Process- or edge-level unrecoverable failure. |
Control verbosity with the global LOG_LEVEL or service overrides such as TELSYS_LOG_LEVEL, TELPRO_LOG_LEVEL, RTPENGINE_LOG_LEVEL, and TELPHI_LOG_LEVEL. See Per-service log levels and the environment variable index.
SIP ladder capture is independent of general log verbosity: lowering info/warn filtering does not remove sip_message captures used by the Debug SIP ladder.
Correlation
Use the same identifiers across TelPro, TelSys, media, and application logs:
| Identifier | Use |
|---|---|
trace_id | Distributed trace across edge, voice, and app logs. |
sipCallId | SIP Call-ID (also shown as call id in TelWeb / Debug). |
transactionId | TOBi per-conversation correlation id. Stable across all BOT requests, SIP headers, and spans. See Transaction correlation. |
| Conversation | TelWeb conversation id when investigating from a call detail page. |
Start from TelWeb Conversations → call detail → Debug (Logs / Spans / SIP ladder), then search SigNoz with the same trace_id, Call-ID, or transactionId.
What is logged
TelSys (voice PBX)
Call and AI-session lifecycle events (typically info):
| Area | Examples | Typical level |
|---|---|---|
| Call entry | Call start, channel created, channel answer | info |
| AI session | Session enter / exit with the voice AI app | info |
| Hangup | Hangup with cause / optional hangup reason | info |
| Transfer | Transfer initiated; transfer result | info / warn if not SUCCESS |
| Dialplan errors | Error-handling paths | error |
Operational bootstrap and config notes use operational log records (see Structured logging schema).
TelPro (SIP edge)
Routing and call-control outcomes (not the raw SIP frames — those are in the SIP ladder):
| Area | Examples | Typical level |
|---|---|---|
| Ingress / routing | Trunk identified, number/customer route, server selected | info |
| Rejects | Unknown trunk, TLS/auth failure, unverified internal source | warn / error |
| Outbound / transfer | Outbound or direct-URI transfer detected or rejected | info / warn |
| BYE / disconnect | Carrier-bound BYE routing; loop prevention when edge misroutes | info / warn / error |
| Failover | TelSys server failure, failover attempt or abort | warn / error |
| Media control (via TelPro) | RTPEngine offer / answer success or failure | info / error |
RTPEngine (media)
Data-plane media outcomes. Control-plane offer/answer commands are logged from TelPro (see SigNoz monitoring).
| Area | Examples | Typical level |
|---|---|---|
| Call quality | End-of-call stats (rtpengine_call_stats: MOS, jitter, loss) | info; warn / error when MOS is poor |
| Media errors | ICE / DTLS / SRTP failures | error |
| Resource errors | Port / socket exhaustion | error |
| Control errors | Unknown call, offer/answer rejected at the media engine | error |
| DTMF | RFC4733 media DTMF events | info |
| Session end | Media session teardown notices | debug |
Payload and detail policy
| Policy | Behavior |
|---|---|
| Metadata always | Targets, status codes, event names, and handling labels (for example bot / STT / TTS) are logged when available. |
| Bodies on failure | Request/response payloads are included when a response indicates failure. |
| Optional fuller payloads | Broader payload logging can be enabled for short support windows; expect higher volume and cost. |
| PII | Redaction before SigNoz export is controlled separately — see PII redaction. |
Security-relevant events
These appear as structured warn or error logs and are searchable in SigNoz (filter by service and event text):
- Unknown or unauthorized trunk / source
- Inbound authentication or TLS peer verification failures
- Outbound or transfer attempts from an unverified internal source
- BYE self-relay blocked (misrouted disconnect toward the platform instead of the carrier)
Where to look
| Surface | Best for |
|---|---|
| TelWeb → conversation → Debug | Per-call logs, spans, and SIP ladder for one conversation. |
| SigNoz Logs | Cross-call search by trace_id, Call-ID, level, or event. |
| SIP signaling reference | Which SIP messages were captured and what response codes mean. |
| Monitoring in SigNoz | Schema keys, log-level env vars, dashboards, and PII. |
SCRUM-738 / VGWK-36 coverage
What Vodafone asked for on application and integration flow logging, and where it lands today. Use this table when reviewing C2 / SignOz — do not treat an unused Kamailio flowId field as a gap unless VF asks for it.
| Requirement | Status | Where to verify |
|---|---|---|
| Reconstruct application-level bot / STT / TTS requests for a conversation | Shipped | TelPhi component such as vodafone_tobi_llm, STT, TTS; TelWeb Debug Logs |
| Correlate application logs with SIP signaling | Shipped | Same trace_id and SIP Call-ID (sipCallId / callId) on TelPro recordKind=sip_message and TelPhi logs |
| Documented log levels and payload policy | Shipped | Log levels and Payload and detail policy above; *_LOG_LEVEL in the env-var index |
| Always log metadata (target, status, bot, STT/TTS handling) | Shipped | Structured fields on TelPhi / TelAPI system logs |
| Log request/response payload on error | Shipped | Payload and detail policy |
| Security events | Shipped | Security-relevant events; Security event export |
| Single correlation id across components | Shipped | trace_id originates at TelPro; transactionId provides a second stable per-conversation id on the managed LLM path. See Transaction correlation. |
| Transaction-style BOT/SIP/TTS/STT events | Shipped | transactionId propagated as OTel span attribute on all pipeline spans and log-to-span spans. See Transaction correlation. |
| Applicable to all components (not only RTP / TelPro / TelSys) | Partial | Node services (TelPhi, TelAPI, TelWeb, Tasker, Scaler) export OTEL when OTEL_ENABLED=true. audioproc and portal do not have native OTEL |
VF transaction-field mapping
| VF property | Delphi field / equivalent |
|---|---|
category (STT / TTS / BOT / SIP) | logType plus component / serviceType (stt, tts, llm, telephony / recordKind=sip_message) |
service | service / service.name (telpro, telsys, telphi, telapi, telweb, tasker, scaler) |
operation | action or operation on the structured record |
server-name | service.instance.id / instance hostname |
transaction-Id | transactionId — per-conversation UUID generated by the TOBi LLM component, propagated through Redis, SIP headers, and OTel spans. See Transaction correlation. |
botName | App / flow name on the conversation and TelPhi context |
conversation-id | Delphi conversationId (TelWeb conversation) |
provider-conversation-id | tobiConversationId on TelPhi / CDR (not on raw Kamailio SBC lines) |
Conversation and call identifiers (VGWK-62)
Four distinct identifiers appear in Delphi logs. They are never interchangeable:
| Identifier | Log field | What it is | Where it appears |
|---|---|---|---|
| Delphi conversation id | conversationId | The TelWeb conversation record. For voice calls this equals the voice-PBX channel id. | TelAPI webhook lines (tobi.callback.*), TelPhi call context, CDR |
| TOBi conversation id | tobiConversationId | The conversation id issued by the TOBi platform (conversation.identifier.id) at create time. | Every TelAPI TOBi webhook line once the session is loaded, TelPhi call context and DTMF/log metadata, CDR |
| Delphi call id | callId | The resumable call/session id (X-Call-ID), keying voiceai:calls:{id} in Redis. | TelAPI, TelPhi, TelPro structured lines (xCallId) |
| SIP Call-ID | sipCallId (TelPro also callId) | The SIP dialog Call-ID header of one SIP leg. | TelPro / Kamailio recordKind=sip_message lines, SIP ladder |
On raw SBC (Kamailio) lines, callId is the SIP dialog Call-ID, not a conversation id, and the TOBi conversation id is not available at the SIP edge. To follow one conversation across layers, pivot on transactionId or callId (X-Call-ID); the tobiConversationId joins in on the TelAPI/TelPhi lines of the same conversation. All TelAPI TOBi webhook log lines — including auth failures and error responses — carry channelId, conversationId, and (once the callback session is resolved) tobiConversationId.
| logpoint | Directional action values (e.g. stream start/complete) rather than request-in / response-out literals |
| verb / url | Logged on outbound HTTP integrations when the client records them |
| call-id | SIP Call-ID (callId / sipCallId) |
| payload / headers | On error, or when fuller payload logging is enabled |
| status / status-code | status plus HTTP/SIP code when applicable |
| timestamp | timestamp (epoch ms in pino; ISO in some telephony JSON) |
| duration | durationMs / latencyMs on responses |
| trace-id | trace_id |
Cross-service logging schema (spike)
Canonical Node service names: telpro, telsys, telphi, telapi, telweb, tasker, scaler.
Telephony (Kamailio / TelSys) uses a parallel JSON shape with recordKind:
recordKind | Meaning |
|---|---|
log | Operational / lifecycle event |
trace | Call-lifecycle span-like event (TelSys dialplan) |
sip_message | Captured SIP frame for the Debug ladder |
App logs from @delphi/logger use logType values such as conversation, model_usage, system, telephony, audio, error, audit, and cdr, plus injected trace_id / span_id. See Service logs and Monitoring in SigNoz.
Please review this schema with VF. Agreed gaps become follow-up stories — this page is the spike deliverable, not a new logger.