Monitoring in SigNoz

Next implementation scope

The transaction fields and explicit security-event tagging described below refer to platform PR 554 (internal implementation reference), reviewed at commit 300e8a9956ada4ac628fd5fceda13855510a73b0. They are not a statement that the current released platform or a customer deployment contains that change. Confirm the deployed implementation before relying on these fields; source checks do not establish live ingestion or delivery.

SigNoz is the central place to investigate a Delphi deployment. Every service host runs an OpenTelemetry collector that forwards logs, traces, metrics, and health signals to the SigNoz service at 10.0.1.10:4317 (OTLP gRPC) / 10.0.1.10:4318 (OTLP HTTP). The SigNoz UI is exposed through the Web service Caddy proxy at https://${DOMAIN_SIGNOZ}.

For running, upgrading, or migrating the SigNoz backend itself, use SigNoz operations. This page is about using SigNoz once telemetry is flowing.

What lands in SigNoz

Signal	Examples	Typical use
Traces	TelAPI WebSocket lifecycle, TelPhi conversation spans, provider calls, flow execution	Follow a request or call end-to-end.
Logs	Application logs, structured TelPro/TelSys JSON, RTPEngine QoS logs (rtpengine_call_stats), tasker jobs	Search for failures by service, trace ID, call ID, or media metrics.
Metrics	Container CPU / memory, request rates, error rates, collector health, ClickHouse storage	Dashboards and alerts.
SIP / call spans	Kamailio SIP ladder, RTPEngine rtpengine.call_stats spans, Asterisk / TelSys channel handling, DTMF	Debug call setup, teardown, routing, keypad input, and no-audio cases.

Kamailio and TelSys SIP captures, method/response-code reference, and capture-point glossary: SIP signaling reference.

Kamailio, RTPEngine, and Janus logs are converted into OTLP spans by the log-to-span sidecar on TelPro. Voice-side call handling lands through Voice collectors.

Structured logging schema

Delphi voice services emit one JSON object per line with aligned keys so SigNoz filters work across TelPro, TelSys, RTPEngine, and Node apps.

For what each log level means and which call-flow events TelSys / TelPro / RTPEngine emit, see Application and call-flow logging.

For the SIP trunk health monitor and TelAPI trunk-health log catalog, see SIP trunk log reference.

Key	Meaning
timestamp	Milliseconds since epoch
level	Pino label: trace, debug, info, warn, error, fatal
logType	Domain: telephony (TelPro), pbx (TelSys), media (RTPEngine), sip_capture
recordKind	trace → spans only; log → SigNoz Logs tab; sip_message → SIP ladder
service	Logical service (telpro, telsys, rtpengine, …)
service.component	Implementation (kamailio, asterisk, rtpengine, entrypoint, …)
trace_id, span_id, parent_span_id	Cross-service correlation (32- / 16-char hex)
logpoint	Transaction leg: request-in, request-out, response-in, response-out. Records carrying it are the application transaction records — see Transaction logging schema.
category	Transaction family: bot, stt, tts, plus the webhook and http extensions; the collector also sets sip on SIP captures, which carry no logpoint
payload, headers	Transaction body and headers, redacted and truncated; present on failure always, on success only while LOG_INTEGRATION_PAYLOADS=true
transaction-Id, trace-id, call-id	Conditional aliases of transactionId, trace context and callId (falling back to sipCallId); absent source values are omitted. call-id is not universally the SIP Call-ID. trace-id survives the collector's trace_id removal
event, component, msg	Queryable semantics and human-readable text
Call context	sipCallId, channelId, trunk/customer fields as applicable

RTPEngine call quality appears in both SigNoz Traces and Logs for the same call:

• Trace span: rtpengine.call_stats (flame graph child of kamailio.call)
• Log entry: event = "rtpengine_call_stats" with rtp.codec, rtp.mos.*, rtp.jitter.*, rtp.packet_loss.*, packet/byte counts

RTPEngine control plane (offer/answer/delete) is logged from Kamailio (rtpengine_offer, rtpengine_answer, …). The RTPEngine container contributes data plane QoS, DTMF (dtmf_media_event), and media errors (rtpengine_media_error).

Severity casing and historical records

The standard SigNoz severity labels are TRACE, DEBUG, INFO, WARN, ERROR and FATAL. Direct shared-logger OpenTelemetry export uses uppercase severityText; raw Pino JSON still uses lowercase level values. Collector normalization produces the canonical severity, while a retained source level attribute may remain lowercase. Missing severity defaults to INFO in the configured normalization path; this fallback does not establish that the event succeeded.

Historical records are not rewritten, so a time range spanning older records can show both info and INFO in the filters. If new records still show lowercase normalized severity, verify the actual collector path and deployed configuration, then query a period starting after its update. Distinguish normalized severity from the raw level attribute. A processor that only fills empty severity does not normalize existing lowercase values.

Configuration values below describe verbosity controls, not the casing of the displayed severity.

Per-service log levels

Delphi application/telephony log controls accept trace, debug, info, warn, error, fatal in increasing severity. A threshold includes that level and more severe events. Use info for routine operation; use debug/trace temporarily for a specific investigation. Structured application logging and raw packet/capture logs are separate surfaces.

Generated production Compose files pass the service variable into each container's LOG_LEVEL. Change the service-specific variable in its configured source, usually SSM via vars.yaml; a host-level LOG_LEVEL alone is not a universal production override. Development bundles can provide a global fallback.

Variable	Host / component	Level handling
TELWEB_LOG_LEVEL	Web / TelWeb	Node structured logger.
TELAPI_LOG_LEVEL	API / TelAPI	Node structured logger.
TELPHI_LOG_LEVEL	Voice / TelPhi	Node structured logger.
AUDIOPROC_LOG_LEVEL	Voice / AudioProc	Audio-processing service logger.
TELSYS_LOG_LEVEL	Voice / Asterisk and its telemetry sidecars	Asterisk channels and collector filtering; SIP capture is separate from operational severity.
TASKER_LOG_LEVEL	Ops / Tasker	Background job/notification logger.
SCALER_LOG_LEVEL	Ops / Scaler	Scaling/reconciliation logger.
TELPRO_LOG_LEVEL	TelPro / Kamailio	Delphi structured threshold plus native core verbosity mapping.
RTPENGINE_LOG_LEVEL	TelPro / RTPEngine	Maps to syslog severity.
WEBRTC_LOG_LEVEL	TelPro / Janus, when WebRTC is enabled	Maps to Janus verbosity.

Native mappings are not numerically interchangeable:

Delphi label	Kamailio core debug	RTPEngine syslog	Janus verbosity
fatal	0	3	1
error	0	3	2
warn	1	4	3
info	2	6	4
debug	2	7	7
trace	3	7	7

Kamailio's structured debug events can be enabled while core verbosity stays at 2. Explicit native Janus settings can override the mapping. Prefer the Delphi labels for portable deployment configuration.

Infrastructure and supporting containers

These components do not all implement the Delphi *_LOG_LEVEL interface. Inspect the configuration shipped with the selected release:

Component	Control / inspection point
PostgreSQL	Native database logging settings and database/managed-service logs; Database operations.
Redis / PgBouncer	Their native configuration and container logs; Database operations.
Caddy (Web and Media)	Generated Caddyfile log/debug configuration; Web and Media.
Media upload	Container stdout; the generated service does not expose a Delphi log-level variable.
Squid	Native squid.conf access/cache/debug controls; Squid operations.
SigNoz / ClickHouse / ZooKeeper	Native service configuration and container logs; SigNoz operations.
OpenTelemetry collector	Collector service telemetry configuration and processing/filter rules; Instance debugging.
log-to-span	Generated parser/export configuration; capture and telemetry conversion are independent of application severity.
coturn, when enabled	Generated TURN configuration and native logs; TelPro operations.
Migration/setup container	One-shot command output; retain exit status and logs with the change record.

Change and verify a level

1. Record the affected component, current value, investigation window, and its source in vars.yaml.
2. Set the component variable in that source. Follow Environment resolution and the restart matrix; drain call-serving capacity before disruptive restarts.
3. Run ./update.sh --restart-only in the affected service directory to refresh runtime configuration and restart consumers.
4. Reproduce the issue and verify the expected component/severity in SigNoz and container output. An application threshold cannot recover records already dropped by a collector, and a raw SIP capture does not imply an error event.
5. Restore the prior value and restart affected consumers again. Keep PII controls in place; do not disable redaction merely to raise verbosity.

At RTPENGINE_LOG_LEVEL=info (default), SigNoz receives end-of-call rtpengine_call_stats logs, DTMF events, and session-end notices. Raise to debug only for short media troubleshooting — raw RTPEngine stderr is not ingested into SigNoz.

TelSys dialplan emits recordKind=trace for call lifecycle (channel_created, stasis_enter, hangup, …) and recordKind=log for operational events (entrypoint bootstrap, config). Event catalogs and level semantics: Application and call-flow logging. Ops deploy notes: Voice operations.

Useful SigNoz log queries

service.name = "telpro" AND attributes.event = "rtpengine_call_stats" AND attributes.trace_id = "<trace-id>"
service.name = "telpro" AND attributes.service.component = "rtpengine" AND attributes.event = "rtpengine_media_error"
service.name = "telsys" AND attributes.recordKind = "log"
service.name = "telsys" AND attributes.recordKind = "trace" AND attributes.event = "channel_created"

Full variable reference: Environment variable index.

Security events

Security-relevant records carry security_event = true plus a security_category. Filter on those attributes rather than on message text — an audit record's body is the bare action name, so a phrase search for login failed will not match LOGIN_FAILED.

attributes.security_event = true
attributes.security_category = "certificate"
service.name = "telpro" AND attributes.security_event = true

The services set the tag themselves. The export collector also has its own text heuristics, but those are only a fallback for sources that do not tag. Note that a deployment without SIEM credentials renders a disabled export pipeline: the events still show up here in SigNoz, but nothing leaves the platform. See Security event export.

PII redaction (v0.9.13)

v0.9.13 adds coordinated PII controls in two layers: the local OTel collector scrubs telemetry before export to SigNoz, and TelPhi / TelAPI redact data written to Postgres (call-details timeline). Both layers read the same env vars from each service host's vars.yaml.

PII redaction is not transcript retention. Whether full transcript text is kept long-term is controlled by the app Recording toggle (see Recording and transcript retention). When recording is on, full transcript text is written to S3 with the recording artefact regardless of ENABLE_PII_LOGGING. When recording is off, full transcript text is not retained. PII settings only control what appears in Postgres, SigNoz, and traces — not whether the recording path persists complete text.

Variable	Default	Meaning
ENABLE_PII_LOGGING	false	When not true, redaction is active. In production, set true only for short support incidents, then revert. Staging/dev may leave it true when debugging with test traffic.
MSISDN_UNMASKED_DIGITS	4	How many trailing digits to keep when masking phone numbers (for example +447911123456 → ***3456).
LOG_INTEGRATION_PAYLOADS	false	When true, transaction records also carry their request/response payload and headers on successful legs. Failed legs always carry them. Set per service. A volume control, not a privacy control — redaction still applies.

Set both on every host that runs an OTel collector and on Voice (voiceai-telphi + voiceai-otel-collector) and API (telapi + voiceai-otel-collector). Other service collectors (Web, TelPro, Ops, Database, Squid) also honour the vars for log export even when they do not run TelPhi.

Since v0.9.17-patch2, shared application logging also honors MSISDN_UNMASKED_DIGITS before collector export. Web and Ops application containers receive that setting through the deployment configuration, alongside Voice/API. Refresh configuration and recreate the affected containers when changing it. The supported installation value is a whole number from 0 to 15; invalid shared-logger settings fall back to 4.

When ENABLE_PII_LOGGING=true, DTMF values remain visible in application logs and collector output. When it is false or unset, DTMF protection covers current and legacy digit fields, nested forwarded digits and digit-bearing messages. SIP INFO Signal values are masked in every message copy before timestamp normalization, so an unredacted alias cannot restore the digits in the final body.

OTel collector (transform/pii)

To change these rules — the replacement text, or how many trailing digits stay visible — edit the collector's own YAML. See OTel collector configuration.

Each service ships a transform/pii processor in its collector config. When ENABLE_PII_LOGGING is not true, log pipelines apply:

Pattern	Redaction
Email addresses in log bodies	user@domain.tld → ****@domain.tld
E.164 MSISDNs in log bodies	Mask body digits; keep the last MSISDN_UNMASKED_DIGITS (for example ***3456)
Card-like dddd-dddd-dddd-dddd sequences	****-****-****-****
Conversation transcript logs (logType=conversation)	Turn text → ***; body → conversation (redacted)
callerNumber log attribute	Last N digits only, prefixed with ***
Transaction payload attribute	[REDACTED] — every transaction body, whether or not the app-level flag logged it; applied in all three envelopes a record can arrive in (flat attribute, metadata.payload, pino.payload)

SIP routing headers remain available in records with event=sip_message so operators can debug call routing. SIP INFO DTMF Signal values are masked when redaction is active, including copies in the final body and message attributes.

On the Voice host, a second processor (transform/pii_traces) scrubs trace span attributes before export:

• transcription.text → ***
• call.caller_number → masked with the same trailing-digit rule

Restart the local voiceai-otel-collector (or re-run init.sh on that host) after changing these vars so the collector process picks up new values.

TelPhi (Voice)

When ENABLE_PII_LOGGING is not true, TelPhi redacts before persisting to the Conversation row:

• Transcript — every turn's text is stored as *** (node/role/timing metadata is kept).
• Caller number — masked with maskMsisdn() using MSISDN_UNMASKED_DIGITS (same semantics as the collector).

TelWeb Timeline and conversation log dialogs therefore show redacted transcript text unless PII logging is enabled.

Recording and transcript retention: when the app has Recording on and the platform allows recording (RECORDING_ENABLED), TelPhi captures a full in-memory snapshot before Postgres redaction and writes complete transcript text to the S3 recording sidecar. That happens whether or not ENABLE_PII_LOGGING is true. Operators or tenant admins with bucket access can retrieve full content from recordings even while the DB timeline and SigNoz remain scrubbed. To avoid retaining transcript text entirely, leave Recording off on the app — PII redaction alone does not disable retention when recording is enabled. See Conversations → Recording and transcript retention.

TelAPI (browser actions)

TelAPI applies the same transcript redaction on persist for browser-action sessions when ENABLE_PII_LOGGING is not true. Browser-action flows are not recorded to S3, so there is no sidecar exception.

Operator guidance

• Production — keep ENABLE_PII_LOGGING=false. Enable it only for short support or incident-debug sessions, then turn it off again and restart affected Voice/API hosts and collectors.
• Staging / dev — ENABLE_PII_LOGGING=true is acceptable when you need full transcripts in SigNoz and TelWeb for integration testing. Use synthetic or consented test data; do not treat non-production as a reason to log real customer PII indefinitely.
• Set MSISDN_UNMASKED_DIGITS consistently on Voice, API, and every collector host — mismatched values make SigNoz and the DB timeline disagree.
• To verify scrubbing, place a test call with PII logging off, then check SigNoz logs/traces and the TelWeb conversation Timeline — transcript text should be ***, caller numbers partially masked.
• Full variable reference: Environment variable index.

Dashboards

Custom Delphi dashboards ship with the deployment bundle under the SigNoz service's signoz/dashboards/ directory. Import them in SigNoz via Settings > Dashboards > Import.

Dashboard	File	Answers
Delphi Overview	voiceai-overview.json	Is the platform healthy? Are calls flowing? Are errors rising?
Call Statistics	voiceai-call-statistics.json	Call volume, duration, success / failure, traffic trends.
Containers	voiceai-containers.json	CPU, memory, restarts, and container-level saturation.
Infrastructure	voiceai-infrastructure.json	Disk, network, CPU per host.

The file names keep the legacy voiceai-* prefix that ships in the bundle. Dashboard titles should appear as Delphi in the SigNoz UI.

Useful upstream references:

• Manage dashboards in SigNoz
• Import dashboards in SigNoz
• SigNoz query builder

Alerts

Delphi ships five starter rules on the SigNoz host at /opt/services/signoz/alerts/. init.sh applies them with apply-alert-rules.sh when SIGNOZ_API_KEY is set (Editor or Admin service-account key in ${NAMESPACE}/signoz/secrets). How to create the key, attach a notification channel, and re-sync after a SigNoz upgrade is in SigNoz operations → Upgrade.

Alert	What it watches
CallErrorRateSpike	Voice pipeline error rate over 5 minutes
ConversationRuntimeCrash	TelPhi fatal runtime error logs
SipTrunkDown	Failed kamailio.trunk_probe spans
SttTtsProviderFailure	Failed STT/TTS spans
TelphiNoTraces	No TelPhi traces (process or collector down)

You can add more in the SigNoz UI. Start with a small set of noisy-but-actionable alerts, then tune thresholds per deployment:

Area	Alert idea	Why it matters
Telemetry intake	No spans or logs received from a service for 5-10 minutes.	The service may be down, or the local OTel collector cannot reach SigNoz.
Call failures	Error rate or failed-call count exceeds a threshold.	Catches carrier, TelPro, Voice, and provider regressions.
Web / API	5xx responses or request latency spike.	User-visible outage signal.
Voice	TelPhi provider latency or error spans spike.	AI provider or proxy path degraded.
Containers	Restart count increases, CPU / memory stays high.	Service instability before users report symptoms.
Storage	ClickHouse or data volume above 80%.	SigNoz retention will fail or queries will degrade.
Collectors	Collector export errors to 10.0.1.10:4317.	Telemetry gap: local debugging may be needed.

Useful upstream references:

• SigNoz alerts documentation
• SigNoz notification channels

Investigating a call

Every call surfaces three correlated identifiers:

Identifier	Where it lives	Used to find
Trace ID	SigNoz, TelPhi logs, TelWeb conversation row	Full distributed trace across TelAPI / TelPhi / TelSys / Kamailio.
Call ID	Kamailio / Asterisk / TelPhi	SIP ladder, RTPEngine sessions, ARI events.
Conversation ID	Postgres Conversation row, TelWeb URL	Persisted timeline, transcripts, QA scoring.

These identifiers are available on the TelWeb conversation detail page.

TelWeb call detail tabs

Tab	Shows
Debug	Separate Logs, Spans, and SIP Ladder panes for the trace. The SIP Ladder pane includes capture details, DTMF evidence, and text export for support tickets.
Timeline	Channel events ordered by ChannelMessage.timestamp. Chat, browser actions, audio segments, control.
Flow run	Per-node execution of the Flow Builder graph with inputs / outputs at each step.
QA	QA scoring results (enqueued on flow finalisation / hangup).
Token	Token usage breakdown by provider.
Action	Browser action invocations and their results.

Superadmin trace search

v0.9.14 adds a SUPERUSER trace-debug search surface in TelWeb for retained SigNoz data. Use it when you have a trace ID, call ID, or SIP/header value but do not yet have a conversation row open. The search uses the same SigNoz URL and API key configured under Platform settings. It accepts pasted SIP header lines (From:, To:, Call-ID:, CSeq:, User-Agent:, X-Call-ID:) and a raw ClickHouse SQL mode (sql: prefix). See Trace debug search for syntax and examples.

Query patterns

Use these as starting filters in SigNoz Logs / Traces:

service.name = "telpro" AND call_id = "<id>"      # Kamailio SIP captures (TelPro)
service.name = "telsys" AND call_id = "<id>"      # Asterisk PJSIP captures (Voice)
attributes.recordKind = "sip_message" AND call_id = "<id>"
service.name = "rtpengine" AND call_id = "<id>"   # Media session
service.name = "telphi" AND trace_id = "<id>"     # Full conversation trace
service.name = "telapi" AND trace_id = "<id>"     # Channel WebSocket lifecycle

Some deployments still index legacy service.name = "kamailio" — see SIP signaling reference.

Common investigations

Question	Start in SigNoz / TelWeb	Then check
Why did this call hang up?	TelWeb conversation > Debug > span tree and SIP ladder.	SIP signaling reference for BYE/CANCEL/481/487; TelPro/Voice ops for routing.
Why is there no audio?	Debug > SIP ladder for the call_id; RTPEngine spans.	TelPro no-audio matrix and RTP firewall ranges.
Did keypad / DTMF input arrive?	Debug > SIP ladder DTMF chips and timeline events.	Compare SIP INFO, RFC4733 SDP/media, ARI, and in-band evidence before debugging the flow node.
Which provider was slow?	Token + Debug logs filtered to service.name = telphi.	Voice logs if provider spans are missing.
Did billing or subscription checks fire?	Flow/API spans and TelAPI logs.	Ops / Tasker logs for background reconciliation.
Was QA scored?	QA tab and Tasker spans.	Ops if qaScoring jobs are absent.
Why is telemetry missing?	Check whether all services stopped reporting at once or only one host is missing.	Instance and container debugging and SigNoz operations.

When to debug on the instance

Use SigNoz first when telemetry is present. Drop to the host or container when:

• a service or host has stopped sending telemetry entirely;
• Docker health checks are failing but no fresh logs appear in SigNoz;
• the local OTel collector cannot export to 10.0.1.10:4317;
• startup fails before the collector is running;
• you need to validate local files, mounts, ports, or docker compose state.

The shared command set lives in Instance and container debugging. Service-specific symptoms stay in each Operations page's Troubleshooting tab.

Stitching a retained trace into a conversation

When a known-good trace has aged out of a tenant's view but is still retained in SigNoz, you can re-link a Conversation row to that trace ID to make the Debug tab usable for support sessions:

UPDATE "Conversation"
SET    "traceId" = '<retained-trace-id>'
WHERE  "id"      = '<conversation-id>';

The mapping is informational; nothing else moves. Use during outage post-mortems or paired support sessions.

See also

• SigNoz operations — run and repair the SigNoz backend.
• Instance and container debugging — SSH / Docker checks when telemetry is missing.
• SIP signaling reference — Kamailio/TelSys captures, OPTIONS/CANCEL/PRACK, response codes.
• TelPro operations — SIP, RTP, Janus, TURN.
• Voice operations — TelPhi and Asterisk call handling.