Logs and diagnostics

Start with the question you need to answer. A log level describes severity; an event type describes what happened. Neither tells you, by itself, where a record is stored or which call it belongs to.

Choose the right evidence

Your question	Evidence to inspect	Start here
What failed inside a service?	Service/runtime messages and their error context	Log levels and object structure, then service inventory (existing guide)
Who signed in or changed configuration?	Audit events: actor, action, category, target and details	Access Logs filters and feed scope; Audit logging (existing guide)
What SIP request or response crossed the edge?	SIP capture and signaling records	SIP signaling (existing guide); use SIP diagnosis for routing and termination
What happened during this call?	Call-flow events, conversation records and traces	Object and correlation boundaries; begin with the known call and time window
How did the call finish?	Call detail record (CDR)	Interpret the CDR summary
Is a trunk unhealthy?	Probe results, health transitions and alarms	SIP trunk logs (existing guide)
Why are records missing or duplicated?	Producer, transport, collector, storage and query filters	Check the delivery path
What may be retained or exported?	Redaction, retention and export configuration	Protect diagnostic logs and security event export and retention

Read a record in four steps

1. Identify its producer and representation: service JSON, an OpenTelemetry log record, a stored application/audit row, a SIP capture or a trace span.
2. Read its severity and event discriminator. Use levels and object structure; do not interpret a logType as a severity.
3. Find the available correlation identifiers and a narrow time range. Missing identifiers do not mean that another field can be substituted unchanged.
4. Follow the event-specific reference for payload fields and expected outcomes, then the configuration task if a change is needed.

Check the delivery path

Check each stage separately: was the event emitted, was that destination enabled, did collection succeed, and does the query search the right service, time range and fields? A database row, a stdout message and an exported log are not guaranteed copies of one another. Browser console output is another source; do not assume it is collected merely because server logging is enabled.

An enabled telemetry export does not prove a record reached storage. A logger setting does not establish collector retention or user access. See Monitoring in SigNoz (existing guide) for the deployed collection and query setup.

Establish the producer and transport

Availability: this procedure is for the deployment/observability operator with access to the selected host's service configuration and telemetry backend. TelWeb team membership or a visible conversation does not grant that access. Ask the operator when the backend, host or required query view is unavailable.

Choose one known, non-sensitive event and record its UTC time, producer, environment and available call/trace identifier. Use an existing event or an approved synthetic request; do not generate a failed call or change a customer's flow just to test collection.

Evidence path	What to check before searching
Application → OTLP	Check the running application's OTEL_ENABLED and OTEL_EXPORTER_OTLP_ENDPOINT, then its instrumentation startup. The shared export switch requires the literal value true; a configured endpoint alone does not enable export. TelWeb's Node instrumentation also requires a nonempty endpoint and appends /v1/logs and /v1/traces. Use the receiver base URL, not an already suffixed signal URL.
Docker stdout/stderr → filelog	Check the actual Docker logging driver and mounted log files. The supplied collectors read Docker JSON files under /var/lib/docker/containers/*/*.log; a different logging driver or unreadable mount is not covered by this file receiver. start_at: end is not a promise to backfill pre-existing history.
Browser console	Inspect the browser locally. Server instrumentation and Docker file collection do not automatically export browser console output.
Application/audit database	Use the appropriate product view and permissions. Persistence and telemetry export are separate paths; finding a database row does not prove backend ingestion.

For application endpoints, check reachability from the application's network context. localhost in a host-network service and localhost inside a bridged container are different destinations. The supplied Web, API, Voice and TelPro collector configurations accept OTLP on gRPC 4317 and HTTP 4318; these are configuration defaults, not instructions to expose either port publicly.

Ops bundle limitation: its supplied collector has filelog/host/container-metric receivers and no OTLP receiver or traces pipeline, although Tasker/Scaler Compose settings can point applications at local HTTP 4318. That environment variable does not create a receiver. Verify the deployed collector configuration before relying on direct application export or expecting traces from this role. Treat an absent receiver as a configuration gap for the operator; do not enable another production listener during diagnosis without reviewing its network and access policy.

Locate the first missing stage

1. Producer: confirm the event should be emitted at the effective threshold and inspect a bounded local sample. Use severity and representation rather than treating raw Pino output as the final normalized record. A suppressed event cannot be recovered by changing a search filter.
2. Receiver: identify the mounted collector configuration and the enabled pipeline for that signal. A receiver defined in YAML must also belong to the pipeline. Check startup/configuration errors and application exporter failures. A running container or a healthy collector endpoint proves neither ingestion nor storage.
3. Processors: inspect filters and identity before assuming loss. Supplied collectors restrict accepted log service names to telpro, telweb, telapi, tasker, scaler, telsys and telphi; file records without service identity can be dropped. The Web collector also filters the audit filelog mirror when direct OTLP is the selected copy. These filters are not a universal duplicate-removal service. Keep the protection layers intact when diagnosing missing fields.
4. Exporter and backend: distinguish the application's receiver URL from the collector's OTEL_BACKEND_ENDPOINT. Inspect collector logs directly for export errors; self-log filtering means those errors may be absent from the central search. The supplied backend exporter has a bounded queue/retry window, not durable unlimited retention. Correct destination, protocol, network and trust settings against the generated deployment configuration; never copy a sample private IP as the customer backend address.
5. Query: start with the correct environment, canonical service name and a narrow incident window. Inspect one returned record's timestamp, resource attributes, body and identifiers before adding filters. Broaden one condition at a time. Logs, traces and stored audit rows are separate views; an empty trace view does not establish an empty log store.

The supplied collector backend configuration uses tls.insecure: true. Verify the deployed transport and network protection explicitly; the setting is not evidence of encrypted backend delivery. The pipeline diagram above describes responsibilities, not a universal host topology or TLS guarantee.

Verify recovery and avoid duplicates

After an approved configuration change, use the role's service health and activation procedure, repeat the same safe event and compare producer, receiver and backend timestamps. Confirm the intended service/environment and expected representation, then restore temporary diagnostic verbosity. A new record proves that tested path at that time; it does not recover expired data or establish complete historical capture.

For duplicates, compare event time, message/discriminator, identifiers and producer/transport before deleting or filtering anything. A stdout copy and direct OTLP copy may describe one event; two retries may describe distinct events. Do not disable audit persistence or redaction to make counts match. If collection works but a call remains hard to find, continue with Trace Debug search and its explicit time/result limits.

Find a trace without an open conversation

Open Admin Settings → Log Search (Trace Debug) when you have a trace ID, SIP Call-ID or header value but no conversation selected. Super Admin and Super Viewer platform-wide readers can use this search; a team role alone is insufficient. The configured SigNoz URL/key and retained telemetry are prerequisites, separate from the account's authority. See platform observability settings.

A — Search field. B — Time range. Search is disabled for empty input. This released-interface capture uses a synthetic Super Admin account. Select the image to read it at full size.

1. Set the time range around the incident. The initial selection is last hour; presets include 15 minutes, 1 hour, 6 hours, 24 hours and 7 days. Custom date/time selections use the browser's Date/time context; note your timezone when comparing UTC evidence. A reversed custom range cannot be applied.
2. Enter a narrow identifier or header in Search SignOz and choose Search or press Enter. Whitespace-only input is disabled; the API accepts at most 500 characters. Editing the input alone does not submit it. A selected time-range change updates the active search bounds.
3. Select a result and inspect SIP, Logs or Spans. Selection initially opens SIP. Where a matching database conversation is found, use its conversation link and conversation review; a trace can exist without a conversation row.
4. Confirm identifiers, time and producer before concluding that grouped records describe the intended call. A list item is a search result, not proof that every event, span or SIP message was retained.

Choose a query mode

SIP Call-ID: Call-ID: synthetic-call-123@example.test combines selected structured-attribute matches and body searches. = is also accepted as a separator; surrounding quotes can be stripped.

Other SIP headers: From: sip:example@example.test includes body fallback for headers present only in captured SIP text. Parsed URI fragments can broaden matches; verify the exact result.

Bare identifiers: a trace/call identifier builds applicable trace/call/application-attribute and body filters. This is not strict equality across every field or an interchangeable-ID guarantee.

Named application attributes: flowDefinitionId: 11111111-2222-4333-8444-555555555555 matches the named attribute exactly, including its pino. form. Name matching tolerates case and punctuation variants; partial values do not match those attributes.

Raw predicate: sql: attributes_string['call_id'] = 'synthetic-call-123@example.test' uses a ClickHouse predicate. Text after sql: becomes the WHERE predicate, not a friendly-header expression. A full SELECT statement is not the documented input.

Friendly mode is the starting point for SIP header investigations. Its body/attribute passes are combined, not an AND condition requiring every candidate field. Header-name normalization does not establish that every spelling has identical specialized filters; prefer the canonical Call-ID and X-Call-ID names.

Raw mode uses signoz_logs.distributed_logs_v2. Stored attributes_string is a map: mapContains(attributes_string, 'call_id') checks key presence; attributes_string['call_id'] reads the value. A header shown by a parsed UI can be derived from body and absent from that stored map. For a synthetic body search, use sql: positionCaseInsensitive(body, 'synthetic-call-123') > 0. Inspect the actual stored representation rather than assuming every UI field is a stored column.

Raw time boundary: the client sends the selected start/end to SigNoz, but its raw SQL text does not inject a timestamp predicate. Do not assume that request metadata alone enforces the selected window on every SigNoz deployment. Confirm returned timestamps and have the observability operator validate time scoping before a broad raw query. Do not copy an unverified timestamp macro or run an unrestricted query as the first troubleshooting step. Raw clauses can also appear in server error diagnostics: never put credentials or private conversation text into them.

Understand limits and empty results

The UI does not expose pagination or a result-limit input. The backend defaults to 500 rows per query pass (API limit 1–500), so this is not an export of all matching telemetry. Friendly search can make several passes, merge matching timestamp/body pairs and group by call/trace; returned group count is not the same as total matching log rows. Bounded identifier expansion can add related records, but does not prove a complete call history.

Friendly results require a usable trace identifier after grouping. If matching logs have no normalizable trace ID, they can fail to become friendly trace results. Raw SQL can return matching rows without one. For searches other than named application-ID searches, a raw-SIP attribute fallback runs only when the initial friendly passes find no rows, and its errors can be swallowed as an empty result; it is not guaranteed to find every raw capture when some other matches already exist.

Observed result	Next check
SigNoz not configured	Ask the operator to verify effective URL and secret reference; account-role changes will not configure telemetry.
Query error	Simplify the query and preserve a redacted error. Raw ClickHouse failures are surfaced; the client disables automatic retries for this search.
No matching traces	Verify time conversion, identifier spelling, retention and producer/delivery path. An empty result does not prove the call never existed.
Header visible elsewhere but attribute equality finds nothing	Try friendly canonical-header search or a narrow body predicate after confirming the stored shape.
Result lacks a conversation link	The API did not associate it with a conversation by trace/call identity; investigate the trace itself rather than creating a conversation or guessing another ID.

Use diagnostic-log protection before sharing results and correlation boundaries before joining identifiers. Search cannot recover expired records, undo redaction or manufacture attributes that the producer never emitted.

Call and transaction correlation

Use Correlation boundaries to distinguish call identifiers, trace context and integration transaction identifiers. Integration-specific propagation and exact fields need their own permitted contract; they are not a universal schema for every Delphi call.

Protect diagnostic logs before sharing

Keep diagnostic evidence useful without treating it as automatically safe to export. A logger's redaction, a collector's transformations, stored conversation content and recording retention are separate controls. First identify the producer and storage copy you intend to inspect or share.

Check the effective protection layer

Shared structured logger

Runs redactForLogs before its Pino output, configured OpenTelemetry emission and optional database persistence. Recognized credential field names and sensitive URL query parameters are scrubbed even when operational PII logging is enabled.

Native SIP/process output, arbitrary console messages and direct database writes do not automatically pass through this helper. Unknown field names or unrecognized free text are not covered by a universal secret detector.

PII handling in that helper

Unless ENABLE_PII_LOGGING is exactly true, replaces recognized transcript/text fields, masks phone-like values in recognized fields/paths and SIP/TEL URIs, and masks recognized DTMF fields/patterns.

Redaction is not proof of anonymity, complete removal of personal data or protection of every nested application payload.

Voice collector

Configured log and trace pipelines apply separate PII transforms. Selected SIP Signal= bodies are masked when the flag is not literal true; conversation text and selected telephone/preview attributes have their own rules.

A collector transformation cannot retroactively protect raw output that was already retained or delivered elsewhere. Other service pipelines and effective deployed configuration need separate inspection.

The reference Voice log processor has rules for email-like local parts, hyphenated card-shaped numbers, recognized phone attributes and textPreview/responsePreview. For logType=conversation, its intended substitutions include text: "***" and a conversation (redacted) body. These are specific patterns, not detection of every email address, payment identifier or transcript. Check the final exported record: later processors can change its body and attributes.

SIP captures are a deliberate exception to the ordinary body-pattern rules: event=sip_message preserves routing evidence while selected Signal= values are masked. A masked DTMF signal therefore does not make the rest of a SIP frame safe to share. Inspect the body and any remaining sip.raw, message or other copies together. Voice trace processing has a separate attribute pipeline; do not infer span protection from one masked log row.

For deployment review, inspect the effective transform/pii, transform/pii_traces and their pipeline ordering in .infrastructure/services/voice/otel-collector-config.yaml. The shared helper lives in packages/logger/src/log-redaction.ts, with logDTMF in packages/logger/src/structured-logger.ts. Use the files from the selected product revision and the configuration actually delivered to that host; a snippet from another release is not an installation instruction.

Conversation/recording stores

Have their own selection, content and retention rules.

A log flag does not prove stored recordings/transcripts were disabled or existing objects were deleted. Use recording selection and storage.

The reference Voice environment defaults ENABLE_PII_LOGGING to false and forwards it to TelPhi and the collector. Confirm the actual consumer environment after the reviewed deployment/refresh procedure; changing an upstream secret or parameter is not proof that a running process adopted it. Enabling operational PII logging expands what can be emitted; do not enable it merely because a support ticket asks for “full logs.” Establish the affected destinations, approved audience and cleanup/retention plan first.

MSISDN_UNMASKED_DIGITS defaults to 4 in the shared helper and reference Voice deployment. The helper accepts an all-digit configured value from 0 to 15 and otherwise falls back to 4. It retains that many trailing digits for matching phone values with at least five digits; a short value can remain unchanged, and a large suffix can expose the entire matched number. This helper validation is not a guarantee that every collector/native consumer validates the variable the same way. Do not infer that every number should have the same visual mask.

Interpret masked records correctly

[REDACTED] means a recognized value was replaced. [Circular] severs a circular object reference; deep object subtrees are replaced at the helper's default depth boundary of 16. These markers are not evidence that the original field was empty. pii_redacted and redacted_fields_count, when attached by the structured logger, describe that emission's replacements; they are not a certification of every field in the record. A record without the marker is not automatically safe.

DTMF helpers and the generic walker both matter. logDTMF masks a supplied string metadata.digit or metadata.digits when its effective PII setting is off; missing expected fields can emit a diagnostic error. The generic walker also masks recognized DTMF keys and selected message patterns. DTMF protection therefore does not depend exclusively on calling logDTMF. Per-call/helper configuration and the final logger pass can differ; inspect emitted evidence rather than assuming one override disables every layer.

Synthetic interpretation example: a transcript field may become [REDACTED], a recognized digits: "12#" value may become "***", and a recognized telephone field may keep only its last four digits. These are classifier examples, not an exhaustive schema or a reason to put real secrets into a test log. Keep executable credentials out of logs at the producer even when a redactor is configured.

Prepare a bounded support extract

1. Identify the necessary event, component, time window and correlation identifiers using log structure. Prefer the smallest relevant excerpt over a complete raw dump.
2. Inspect the actual representation: structured record, rendered message, collector output or stored row. Check headers, URL queries, nested metadata, before/after objects, phone numbers, text and attached files for sensitive content the selected layer may not classify.
3. Remove credentials and private conversation/customer content before sharing through the approved channel. A screenshot or copied browser response is a new copy with its own audience; it does not inherit backend authorization.
4. If content appears unexpectedly unmasked, record the producer, effective settings and redacted example shape. Investigate whether it bypassed the logger, matched a different classifier or used another collector route. Do not paste the leaked value into a general issue as proof.
5. Verify retention and destination access for each existing copy through security export and retention. Restoring a redaction setting changes future handling; it does not erase prior exports or downloaded material.

For an approved diagnostic exception, return the consumer settings to the intended policy afterward and verify a new synthetic event at the relevant output stages. No source inspection or single masked sample establishes complete protection across the platform.

Related tasks

• Find the relevant UI screen when you need to locate a conversation or access-log view.
• Diagnose missing recordings for stored audio/transcripts; recording artefacts and diagnostic logs have different controls.
• SIP and call handling connects protocol behavior, configuration and diagnosis.

Migration status

The overview, common logger reference and CDR interpretation are rebuilt here. Detailed audit, SIP and service references still link to existing guides while their fields and release applicability are checked. This page does not certify a complete event catalogue or a deployed telemetry pipeline.

For a browser disconnect or failed voice restoration, use Browser-session recovery to distinguish token validation, retained session state and media resume before retrying or starting another conversation.

For a security or configuration change, use Reconstruct an audited change before inferring success, current authority or a full rollback snapshot from an individual row.