Transaction correlation

The transactionId is a stable per-conversation correlation id generated by the TOBi managed LLM provider. It travels from the bot integration through TelAPI, Redis, SIP headers, and OTel spans so that every event in a single conversation — across all services — can be filtered under one id in SigNoz.

This page documents the full propagation chain and how to search by transactionId in SigNoz. For the SIP header mechanics, see SIP signaling reference. For the webhook endpoint that carries the id back into Delphi, see Webhooks.

Why a separate id?

The distributed trace_id already correlates most services, but it has limitations on the managed LLM path:

• The trace_id is generated at the SIP edge (Kamailio) and may not be known to the managed LLM provider until after call setup.
• The managed LLM provider sends its own outbound BOT requests to the TOBi API and needs a single stable id to correlate those requests end-to-end — including callbacks that arrive asynchronously.
• Early call-setup spans (before the TOBi provider is instantiated) and media-only spans (RTPEngine) don't carry the transactionId in their own logs.

The transactionId solves this: it is a UUID generated once per conversation by the TOBi LLM component, sent to the TOBi API as the vf-trace-transaction-id header on every outbound BOT request, and echoed back on webhook callbacks. Delphi then propagates it through the full voice stack.

Propagation chain

Step-by-step

Step	Service	What happens
1	TOBi LLM component	Constructor generates transactionId = randomUUID(). One per conversation.
2	TOBi LLM → TOBi API	Every outbound BOT request carries vf-trace-transaction-id: <transactionId> in the HTTP headers.
3	TOBi API → TelAPI webhook	The TOBi API echoes the header on callback responses. TelAPI extracts it from vf-trace-transaction-id on POST /api/v1/webhooks/tobi/:channelId.
4	TelAPI → Redis	TelAPI writes transactionId into the voiceai:calls:{callId} Redis key so downstream voice infrastructure can read it.
5	Kamailio (TelPro)	The REDIS_VALIDATE_XCALL_ID route reads transactionId from the Redis call record. The LOG_EMIT route includes it in structured log payloads. The ADD_TRACE_HEADER route forwards it as the X-Transaction-ID SIP header toward TelSys.
6	TelSys (Voice)	Asterisk reads X-Transaction-ID at INVITE time into channel variable X_TRANSACTION_ID. All structured log events (channel_created, stasis_enter, transfer_initiated, hangup, error) include "transaction_id":"${X_TRANSACTION_ID}".
7	TelPhi	Sets X_TRANSACTION_ID via ARI setChannelVar when the TOBi provider generates the id (covers resumed sessions and fresh provider paths). Attaches transactionId as an OTel span attribute on the call span and on all four modular audio pipeline spans (user.speech, user.transcript, ai.transcript, ai.response).
8	log-to-span	The sidecar caches transactionId per Call-ID and injects it as a transactionId span attribute on spans from services that don't carry it in their own logs (RTPEngine, early call-setup spans).
9	SigNoz	All spans and logs carry transactionId as a searchable attribute. Filter by transactionId to see every event for one conversation across all services.

Where transactionId appears

OTel span attribute

transactionId is set as a span attribute on:

Span	Service	How
call (root call span)	TelPhi	callSpan.setAttribute('transactionId', …) in stasis-start handler
user.speech	TelPhi	transactionSpanAttributes spread into span attributes
user.transcript	TelPhi	same
ai.transcript	TelPhi	same
ai.response	TelPhi	same
kamailio.call and child spans	log-to-span	YAML config maps transactionId from log field transactionId
telsys.channel and child spans	log-to-span	YAML config maps transactionId from log field transaction_id
rtpengine.call_stats	log-to-span	Injected from callTransactionIDCache (Call-ID lookup)

Structured log field

Service	Log field name	Notes
TelPhi (Node)	transactionId	Set via logger context (channelLogger.setContext({ transactionId }))
TelAPI (Node)	transactionId	Set on the per-request child logger for the webhook handler
Kamailio (TelPro)	transactionId	Included in LOG_EMIT structured payload when present
TelSys (Voice)	transaction_id	Included in all recordKind=trace and recordKind=log dialplan events

SIP header

Header	Direction	Set by	Read by
X-Transaction-ID	TelPro → TelSys (SIP INVITE / in-dialog requests)	Kamailio ADD_TRACE_HEADER route	Asterisk dialplan at INVITE time
X-Transaction-ID	TelPhi → TelSys (ARI channel variable)	TelPhi setChannelVar	Asterisk dialplan (covers resumed sessions)

Searching in SigNoz

By span attribute

attributes.transactionId = "<uuid>"

This returns every span across TelPhi, TelPro (Kamailio), TelSys (Asterisk), and RTPEngine that carries the transactionId attribute — the full conversation trace across all services.

By log field

service.name = "telphi" AND transactionId = "<uuid>"
service.name = "telpro" AND attributes.transactionId = "<uuid>"
service.name = "telsys" AND attributes.transaction_id = "<uuid>"

Field name differs by service

Node services (TelPhi, TelAPI) use transactionId (camelCase). TelSys dialplan logs use transaction_id (snake_case). log-to-span normalises both to the transactionId span attribute.

From TelWeb Trace Debug

If you have a conversation row open, the TelPhi call span carries transactionId as a span attribute. Copy it from the Debug → Spans pane and search SigNoz with it. See Trace debug search for the superadmin search box.

Webhook and OpenAPI reference

The TOBi webhook endpoint (POST /api/v1/webhooks/tobi/{channelId}) is documented in the generated OpenAPI reference. The vf-trace-transaction-id header is a provider-side contract header — it belongs to the TOBi API, not to Delphi's own API surface, so it is not modelled as a Delphi OpenAPI parameter. Delphi accepts it on the callback and uses it as the transactionId for correlation.

Reference	Link
TOBi webhook endpoint (Redoc)	WebRTC profile · Webhooks-only profile
TobiCallbackBody schema	WebRTC profile · Webhooks-only profile
TobiBotControlEvent schema	WebRTC profile · Webhooks-only profile
API reference status	API reference

The callback body schema (TobiCallbackBody) does not contain transactionId as a body field — the id travels as the vf-trace-transaction-id HTTP header, not inside the JSON payload. See Webhooks for the header contract and TOBi managed LLM for provider configuration.

log-to-span diagnostic logging

When log-to-span is not producing spans for a service, enable diagnostic output to see why lines are being skipped:

# Set LOG_LEVEL=debug on the log-to-span container
docker compose exec log-to-span sh -c 'export LOG_LEVEL=debug && kill 1'

Or set LOG_LEVEL=debug in the docker-compose environment and restart. Diagnostic messages show:

• whether JSON was extracted from the log line
• whether the JSON unmarshal succeeded
• whether trace_id is present and valid (32 hex chars)
• whether the event type is configured in the YAML

This is useful when troubleshooting missing TelPro or TelSys spans in SigNoz. See SIP signaling reference → Configuration for the relevant env vars.

See also

• Application and call-flow logging — what each service logs and at which level.
• SIP signaling reference — SIP headers, capture points, and the X-Transaction-ID header.
• Monitoring in SigNoz — structured logging schema and correlation identifiers.
• Trace debug search — superadmin SigNoz search by trace ID, call ID, or header value.
• TOBi managed LLM — provider configuration, application metadata, and the vf-trace-transaction-id header.
• Webhooks — the inbound callback endpoint and auth model.
• Managed callback actions — field-level reference for transfer, hangup, and runtime config callbacks.