Skip to main content
Version: 0.9.17-patch2

Trace debug search

The TelWeb Trace Debug view has a SUPERUSER-only search box for retained SigNoz data. Use it when you have a trace ID, call ID, or SIP/header value but no conversation row open yet. The search uses the same SigNoz URL and API key configured under Platform settings, and respects the time-range picker next to the box (default: last hour).

The box has two modes:

ModeWhen to usePrefix
Friendly headerYou pasted a SIP header line or a name: value / name = value pair. Best default for call investigations.none
Raw ClickHouseYou need exact column predicates, Map lookups, LIKE/positionCaseInsensitive, or to combine many fields.sql:

Friendly header mode (default)​

Paste a SIP header line — or any name: value / name = value pair. The parser:

  • accepts both name: value and name = value separators (with optional surrounding quotes on the value);
  • lowercases the name and treats ./_ like -, so call_id, call-id, and callId are equivalent;
  • splits SIP URIs into the address and each ;tag/param fragment, and splits Call-ID on @, so partial values still match;
  • searches both the structured attributes_string map (e.g. call_id, xCallId, headerValue) and the log body with a contains fallback — so a header that lives only inside the raw SIP message still hits.

Recognized SIP headers get tailored filters:

Header (name:)Extra filters added
Call-IDcall_id / sipCallId / sip.call_id attribute variants + Call-ID: <value> in body.
X-Call-IDxCallId + headerValue attributes + X-Call-ID: <value> in body.
From / To / Contact / User-Agent / CSeqheaderValue attribute + <Name>: <value> in body.

Examples​

call_id: 39-742@172.19.14.50
xCallId: '0'
userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36'
From: sip:1291@0.0.0.0:49735
To: sip:011442038070440@91.107.204.150:5060
CSeq: 2 INVITE

The same lines work with = instead of : — for example xCallId = '0' or userAgent = 'Mozilla/5.0 ...'.

Free text with no separator (e.g. a raw trace ID, call ID, or transaction ID) is also fine: a 16–36 char hex string is treated as a trace ID, a UUID-format string is matched against transactionId attributes, and any other term of length ≥ 6 is matched against call-ID / xCallId / headerValue attributes and the body. See Transaction correlation for how transactionId propagates.

Flow and TOBi conversation IDs​

From v0.9.17-patch2, search exact stored application IDs even when they appear only in structured log attributes:

flowDefinitionId: 11111111-2222-4333-8444-555555555555
tobiConversationId: example-bot-conversation-123
flow_definition_id = "11111111-2222-4333-8444-555555555555"

Named searches accept case differences, underscores, hyphens and an optional pino. prefix. Values match exactly; partial IDs do not match these attributes. Named searches check the current and legacy stored attribute forms without scanning unrelated SIP or log-body text. Bare IDs of at least six characters also search these attributes alongside the existing call/trace search paths.

Choose a time range containing the original call. A release cannot recover expired logs or attributes that were never recorded.

Raw ClickHouse SQL mode (sql:)​

Prefix the query with sql:. Everything after the prefix is used verbatim as the WHERE clause of a query against the logs table (signoz_logs.distributed_logs_v2). The selected columns include body, attributes_string, attributes_number, attributes_bool, resources_string, trace_id, span_id, and timestamp. The time-range picker constrains start/end of the request — do not add a timestamp filter yourself (timestamp is UInt64 nanoseconds; a hand-written {{.start_datetime}} DateTime string compares as text and silently returns nothing).

attributes_string is a ClickHouse Map(String, String). Read a single key with bracket syntax; test whether a key exists with mapContains:

sql: attributes_string['call_id'] = '39-742@172.19.14.50'
sql: mapContains(attributes_string, 'xCallId')
sql: attributes_string['userAgent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36'
sql: attributes_string['transactionId'] = '<uuid>'

Equality (field = value)​

Use exact equality on a stored attribute key. Confirm the key is actually present in the stored map first — see Stored attributes vs. body below.

sql: attributes_string['call_id'] = '39-742@172.19.14.50'
sql: attributes_string['cseq'] = '2 INVITE'
sql: attributes_string['fromUri'] = 'sip:1291@0.0.0.0:49735'
sql: attributes_string['toUri'] = 'sip:011442038070440@91.107.204.150:5060'
sql: attributes_string['userAgent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36'

Header-style (field: value)​

The friendly field: value form also works inside raw mode only if it is not prefixed with sql: — i.e. drop the sql: prefix and the same line is parsed by the friendly header engine (see above). Inside sql: mode there is no header engine; write the SQL predicate yourself.

Search in body​

Most SIP header values (From, To, User-Agent, X-Call-ID, CSeq) are not stored as their own attributes_string keys — they live inside the raw SIP message in body. Search the body with like (case-sensitive) or positionCaseInsensitive (case-insensitive):

sql: like(body, '%39-742@172.19.14.50%')
sql: positionCaseInsensitive(body, 'X-Call-ID') > 0
sql: like(body, '%sip:1291@0.0.0.0:49735%')
sql: positionCaseInsensitive(body, 'sip:011442038070440@91.107.204.150:5060') > 0
sql: positionCaseInsensitive(body, 'CSeq: 2 INVITE') > 0
sql: like(body, '%Chrome/150.0.0.0 Safari/537.36%')

Combine predicates with AND / OR / NOT:

sql: positionCaseInsensitive(body, 'X-Call-ID') > 0 AND NOT like(attributes_string['fromUri'], '%hydra%')

Stored attributes vs. body​

A recurring gotcha: keys like xCallId, fromUri, toUri, userAgent, and CSeq are synthesized at read time by the API (from the SIP capture payload), not necessarily present in the ClickHouse attributes_string map that sql: mode queries. The stored map reliably carries service-level keys (call_id, sipCallId, sip.call_id, trace_id, headerValue, level, service.name); SIP-header values are usually only in body.

So:

  • prefer friendly header mode for From / To / User-Agent / CSeq / X-Call-ID — it already falls back to body contains;
  • in sql: mode, prefer a body predicate for those headers, or first verify the key is stored with mapContains(attributes_string, '<key>');
  • call_id / sipCallId are safe to query as attributes_string['call_id'] equality.

Errors​

ClickHouse errors from sql: mode are surfaced in the search panel (not swallowed as an empty result). If a query returns nothing, narrow the time range, simplify the predicate, or run sql: 1=1 to confirm rows exist in the window.

See also​