Monitoring in SigNoz
SigNoz is the central place to investigate a Delphi deployment. Every service host runs an OpenTelemetry collector that forwards logs, traces, metrics, and health signals to the SigNoz service at 10.0.1.10:4317 (OTLP gRPC) / 10.0.1.10:4318 (OTLP HTTP). The SigNoz UI is exposed through the Web service Caddy proxy at https://${DOMAIN_SIGNOZ}.
For running, upgrading, or migrating the SigNoz backend itself, use SigNoz operations. This page is about using SigNoz once telemetry is flowing.
What lands in SigNoz
| Signal | Examples | Typical use |
|---|---|---|
| Traces | TelAPI WebSocket lifecycle, TelPhi conversation spans, provider calls, flow execution | Follow a request or call end-to-end. |
| Logs | Application logs, structured TelPro/TelSys JSON, RTPEngine QoS logs (rtpengine_call_stats), tasker jobs | Search for failures by service, trace ID, call ID, or media metrics. |
| Metrics | Container CPU / memory, request rates, error rates, collector health, ClickHouse storage | Dashboards and alerts. |
| SIP / call spans | Kamailio SIP ladder, RTPEngine rtpengine.call_stats spans, Asterisk / TelSys channel handling, DTMF | Debug call setup, teardown, routing, keypad input, and no-audio cases. |
Kamailio and TelSys SIP captures, method/response-code reference, and capture-point glossary: SIP signaling reference.
Kamailio, RTPEngine, and Janus logs are converted into OTLP spans by the log-to-span sidecar on TelPro. Voice-side call handling lands through Voice collectors.
Structured logging schema
Delphi voice services emit one JSON object per line with aligned keys so SigNoz filters work across TelPro, TelSys, RTPEngine, and Node apps.
For what each log level means and which call-flow events TelSys / TelPro / RTPEngine emit, see Application and call-flow logging.
For the SIP trunk health monitor and TelAPI trunk-health log catalog, see SIP trunk log reference.
| Key | Meaning |
|---|---|
timestamp | Milliseconds since epoch |
level | Pino label: trace, debug, info, warn, error, fatal |
logType | Domain: telephony (TelPro), pbx (TelSys), media (RTPEngine), sip_capture |
recordKind | trace → spans only; log → SigNoz Logs tab; sip_message → SIP ladder |
service | Logical service (telpro, telsys, rtpengine, …) |
service.component | Implementation (kamailio, asterisk, rtpengine, entrypoint, …) |
trace_id, span_id, parent_span_id | Cross-service correlation (32- / 16-char hex) |
event, component, msg | Queryable semantics and human-readable text |
| Call context | sipCallId, channelId, trunk/customer fields as applicable |
RTPEngine call quality appears in both SigNoz Traces and Logs for the same call:
- Trace span:
rtpengine.call_stats(flame graph child ofkamailio.call) - Log entry:
event = "rtpengine_call_stats"withrtp.codec,rtp.mos.*,rtp.jitter.*,rtp.packet_loss.*, packet/byte counts
RTPEngine control plane (offer/answer/delete) is logged from Kamailio (rtpengine_offer, rtpengine_answer, …). The RTPEngine container contributes data plane QoS, DTMF (dtmf_media_event), and media errors (rtpengine_media_error).
Severity casing and historical records
The standard SigNoz severity labels are TRACE, DEBUG, INFO, WARN, ERROR and FATAL. Direct shared-logger OpenTelemetry export uses uppercase severityText; raw Pino JSON still uses lowercase level values. Collector normalization produces the canonical severity, while a retained source level attribute may remain lowercase. Missing severity defaults to INFO in the configured normalization path; this fallback does not establish that the event succeeded.
Historical records are not rewritten, so a time range spanning older records can show both info and INFO in the filters. If new records still show lowercase normalized severity, verify the actual collector path and deployed configuration, then query a period starting after its update. Distinguish normalized severity from the raw level attribute. A processor that only fills empty severity does not normalize existing lowercase values.
Configuration values below describe verbosity controls, not the casing of the displayed severity.
Per-service log levels
Delphi application/telephony log controls accept trace, debug, info, warn, error, fatal in increasing severity. A threshold includes that level and more severe events. Use info for routine operation; use debug/trace temporarily for a specific investigation. Structured application logging and raw packet/capture logs are separate surfaces.
Generated production Compose files pass the service variable into each container's LOG_LEVEL. Change the service-specific variable in its configured source, usually SSM via vars.yaml; a host-level LOG_LEVEL alone is not a universal production override. Development bundles can provide a global fallback.
| Variable | Host / component | Level handling |
|---|---|---|
TELWEB_LOG_LEVEL | Web / TelWeb | Node structured logger. |
TELAPI_LOG_LEVEL | API / TelAPI | Node structured logger. |
TELPHI_LOG_LEVEL | Voice / TelPhi | Node structured logger. |
AUDIOPROC_LOG_LEVEL | Voice / AudioProc | Audio-processing service logger. |
TELSYS_LOG_LEVEL | Voice / Asterisk and its telemetry sidecars | Asterisk channels and collector filtering; SIP capture is separate from operational severity. |
TASKER_LOG_LEVEL | Ops / Tasker | Background job/notification logger. |
SCALER_LOG_LEVEL | Ops / Scaler | Scaling/reconciliation logger. |
TELPRO_LOG_LEVEL | TelPro / Kamailio | Delphi structured threshold plus native core verbosity mapping. |
RTPENGINE_LOG_LEVEL | TelPro / RTPEngine | Maps to syslog severity. |
WEBRTC_LOG_LEVEL | TelPro / Janus, when WebRTC is enabled | Maps to Janus verbosity. |
Native mappings are not numerically interchangeable:
| Delphi label | Kamailio core debug | RTPEngine syslog | Janus verbosity |
|---|---|---|---|
fatal | 0 | 3 | 1 |
error | 0 | 3 | 2 |
warn | 1 | 4 | 3 |
info | 2 | 6 | 4 |
debug | 2 | 7 | 7 |
trace | 3 | 7 | 7 |
Kamailio's structured debug events can be enabled while core verbosity stays at 2. Explicit native Janus settings can override the mapping. Prefer the Delphi labels for portable deployment configuration.
Infrastructure and supporting containers
These components do not all implement the Delphi *_LOG_LEVEL interface. Inspect the configuration shipped with the selected release:
| Component | Control / inspection point |
|---|---|
| PostgreSQL | Native database logging settings and database/managed-service logs; Database operations. |
| Redis / PgBouncer | Their native configuration and container logs; Database operations. |
| Caddy (Web and Media) | Generated Caddyfile log/debug configuration; Web and Media. |
| Media upload | Container stdout; the generated service does not expose a Delphi log-level variable. |
| Squid | Native squid.conf access/cache/debug controls; Squid operations. |
| SigNoz / ClickHouse / ZooKeeper | Native service configuration and container logs; SigNoz operations. |
| OpenTelemetry collector | Collector service telemetry configuration and processing/filter rules; Instance debugging. |
| log-to-span | Generated parser/export configuration; capture and telemetry conversion are independent of application severity. |
| coturn, when enabled | Generated TURN configuration and native logs; TelPro operations. |
| Migration/setup container | One-shot command output; retain exit status and logs with the change record. |
Change and verify a level
- Record the affected component, current value, investigation window, and its source in
vars.yaml. - Set the component variable in that source. Follow Environment resolution and the restart matrix; drain call-serving capacity before disruptive restarts.
- Run
./update.sh --restart-onlyin the affected service directory to refresh runtime configuration and restart consumers. - Reproduce the issue and verify the expected component/severity in SigNoz and container output. An application threshold cannot recover records already dropped by a collector, and a raw SIP capture does not imply an
errorevent. - Restore the prior value and restart affected consumers again. Keep PII controls in place; do not disable redaction merely to raise verbosity.
At RTPENGINE_LOG_LEVEL=info (default), SigNoz receives end-of-call rtpengine_call_stats logs, DTMF events, and session-end notices. Raise to debug only for short media troubleshooting — raw RTPEngine stderr is not ingested into SigNoz.
TelSys dialplan emits recordKind=trace for call lifecycle (channel_created, stasis_enter, hangup, …) and recordKind=log for operational events (entrypoint bootstrap, config). Event catalogs and level semantics: Application and call-flow logging. Ops deploy notes: Voice operations.
Useful SigNoz log queries
service.name = "telpro" AND attributes.event = "rtpengine_call_stats" AND attributes.trace_id = "<trace-id>"
service.name = "telpro" AND attributes.service.component = "rtpengine" AND attributes.event = "rtpengine_media_error"
service.name = "telsys" AND attributes.recordKind = "log"
service.name = "telsys" AND attributes.recordKind = "trace" AND attributes.event = "channel_created"
Full variable reference: Environment variable index.
PII redaction (v0.9.13)
v0.9.13 adds coordinated PII controls in two layers: the local OTel collector scrubs telemetry before export to SigNoz, and TelPhi / TelAPI redact data written to Postgres (call-details timeline). Both layers read the same env vars from each service host's vars.yaml.
PII redaction is not transcript retention. Whether full transcript text is kept long-term is controlled by the app Recording toggle (see Recording and transcript retention). When recording is on, full transcript text is written to S3 with the recording artefact regardless of ENABLE_PII_LOGGING. When recording is off, full transcript text is not retained. PII settings only control what appears in Postgres, SigNoz, and traces — not whether the recording path persists complete text.
| Variable | Default | Meaning |
|---|---|---|
ENABLE_PII_LOGGING | false | When not true, redaction is active. In production, set true only for short support incidents, then revert. Staging/dev may leave it true when debugging with test traffic. |
MSISDN_UNMASKED_DIGITS | 4 | How many trailing digits to keep when masking phone numbers (for example +447911123456 → ***3456). |
Set both on every host that runs an OTel collector and on Voice (voiceai-telphi + voiceai-otel-collector) and API (telapi + voiceai-otel-collector). Other service collectors (Web, TelPro, Ops, Database, Squid) also honour the vars for log export even when they do not run TelPhi.
Since v0.9.17-patch2, shared application logging also honors MSISDN_UNMASKED_DIGITS before collector export. Web and Ops application containers receive that setting through the deployment configuration, alongside Voice/API. Refresh configuration and recreate the affected containers when changing it. The supported installation value is a whole number from 0 to 15; invalid shared-logger settings fall back to 4.
When ENABLE_PII_LOGGING=true, DTMF values remain visible in application logs and collector output. When it is false or unset, DTMF protection covers current and legacy digit fields, nested forwarded digits and digit-bearing messages. SIP INFO Signal values are masked in every message copy before timestamp normalization, so an unredacted alias cannot restore the digits in the final body.
OTel collector (transform/pii)
To change these rules — the replacement text, or how many trailing digits stay visible — edit the collector's own YAML. See OTel collector configuration.
Each service ships a transform/pii processor in its collector config. When ENABLE_PII_LOGGING is not true, log pipelines apply:
| Pattern | Redaction |
|---|---|
| Email addresses in log bodies | user@domain.tld → ****@domain.tld |
| E.164 MSISDNs in log bodies | Mask body digits; keep the last MSISDN_UNMASKED_DIGITS (for example ***3456) |
Card-like dddd-dddd-dddd-dddd sequences | ****-****-****-**** |
Conversation transcript logs (logType=conversation) | Turn text → ***; body → conversation (redacted) |
callerNumber log attribute | Last N digits only, prefixed with *** |
SIP routing headers remain available in records with event=sip_message so operators can debug call routing. SIP INFO DTMF Signal values are masked when redaction is active, including copies in the final body and message attributes.
On the Voice host, a second processor (transform/pii_traces) scrubs trace span attributes before export:
transcription.text→***call.caller_number→ masked with the same trailing-digit rule
Restart the local voiceai-otel-collector (or re-run init.sh on that host) after changing these vars so the collector process picks up new values.
TelPhi (Voice)
When ENABLE_PII_LOGGING is not true, TelPhi redacts before persisting to the Conversation row:
- Transcript — every turn's
textis stored as***(node/role/timing metadata is kept). - Caller number — masked with
maskMsisdn()usingMSISDN_UNMASKED_DIGITS(same semantics as the collector).
TelWeb Timeline and conversation log dialogs therefore show redacted transcript text unless PII logging is enabled.
Recording and transcript retention: when the app has Recording on and the platform allows recording (RECORDING_ENABLED), TelPhi captures a full in-memory snapshot before Postgres redaction and writes complete transcript text to the S3 recording sidecar. That happens whether or not ENABLE_PII_LOGGING is true. Operators or tenant admins with bucket access can retrieve full content from recordings even while the DB timeline and SigNoz remain scrubbed. To avoid retaining transcript text entirely, leave Recording off on the app — PII redaction alone does not disable retention when recording is enabled. See Conversations → Recording and transcript retention.
TelAPI (browser actions)
TelAPI applies the same transcript redaction on persist for browser-action sessions when ENABLE_PII_LOGGING is not true. Browser-action flows are not recorded to S3, so there is no sidecar exception.
Operator guidance
- Production — keep
ENABLE_PII_LOGGING=false. Enable it only for short support or incident-debug sessions, then turn it off again and restart affected Voice/API hosts and collectors. - Staging / dev —
ENABLE_PII_LOGGING=trueis acceptable when you need full transcripts in SigNoz and TelWeb for integration testing. Use synthetic or consented test data; do not treat non-production as a reason to log real customer PII indefinitely. - Set
MSISDN_UNMASKED_DIGITSconsistently on Voice, API, and every collector host — mismatched values make SigNoz and the DB timeline disagree. - To verify scrubbing, place a test call with PII logging off, then check SigNoz logs/traces and the TelWeb conversation Timeline — transcript text should be
***, caller numbers partially masked. - Full variable reference: Environment variable index.
Dashboards
Custom Delphi dashboards ship with the deployment bundle under the SigNoz service's signoz/dashboards/ directory. Import them in SigNoz via Settings > Dashboards > Import.
| Dashboard | File | Answers |
|---|---|---|
| Delphi Overview | voiceai-overview.json | Is the platform healthy? Are calls flowing? Are errors rising? |
| Call Statistics | voiceai-call-statistics.json | Call volume, duration, success / failure, traffic trends. |
| Containers | voiceai-containers.json | CPU, memory, restarts, and container-level saturation. |
| Infrastructure | voiceai-infrastructure.json | Disk, network, CPU per host. |
The file names keep the legacy voiceai-* prefix that ships in the bundle. Dashboard titles should appear as Delphi in the SigNoz UI.
Useful upstream references:
Alerts
Delphi ships five starter rules on the SigNoz host at /opt/services/signoz/alerts/. init.sh applies them with apply-alert-rules.sh when SIGNOZ_API_KEY is set (Editor or Admin service-account key in ${NAMESPACE}/signoz/secrets). How to create the key, attach a notification channel, and re-sync after a SigNoz upgrade is in SigNoz operations → Upgrade.
| Alert | What it watches |
|---|---|
| CallErrorRateSpike | Voice pipeline error rate over 5 minutes |
| ConversationRuntimeCrash | TelPhi fatal runtime error logs |
| SipTrunkDown | Failed kamailio.trunk_probe spans |
| SttTtsProviderFailure | Failed STT/TTS spans |
| TelphiNoTraces | No TelPhi traces (process or collector down) |
You can add more in the SigNoz UI. Start with a small set of noisy-but-actionable alerts, then tune thresholds per deployment:
| Area | Alert idea | Why it matters |
|---|---|---|
| Telemetry intake | No spans or logs received from a service for 5-10 minutes. | The service may be down, or the local OTel collector cannot reach SigNoz. |
| Call failures | Error rate or failed-call count exceeds a threshold. | Catches carrier, TelPro, Voice, and provider regressions. |
| Web / API | 5xx responses or request latency spike. | User-visible outage signal. |
| Voice | TelPhi provider latency or error spans spike. | AI provider or proxy path degraded. |
| Containers | Restart count increases, CPU / memory stays high. | Service instability before users report symptoms. |
| Storage | ClickHouse or data volume above 80%. | SigNoz retention will fail or queries will degrade. |
| Collectors | Collector export errors to 10.0.1.10:4317. | Telemetry gap: local debugging may be needed. |
Useful upstream references:
Investigating a call
Every call surfaces three correlated identifiers:
| Identifier | Where it lives | Used to find |
|---|---|---|
| Trace ID | SigNoz, TelPhi logs, TelWeb conversation row | Full distributed trace across TelAPI / TelPhi / TelSys / Kamailio. |
| Call ID | Kamailio / Asterisk / TelPhi | SIP ladder, RTPEngine sessions, ARI events. |
| Conversation ID | Postgres Conversation row, TelWeb URL | Persisted timeline, transcripts, QA scoring. |
These identifiers are available on the TelWeb conversation detail page.
TelWeb call detail tabs
| Tab | Shows |
|---|---|
| Debug | Separate Logs, Spans, and SIP Ladder panes for the trace. The SIP Ladder pane includes capture details, DTMF evidence, and text export for support tickets. |
| Timeline | Channel events ordered by ChannelMessage.timestamp. Chat, browser actions, audio segments, control. |
| Flow run | Per-node execution of the Flow Builder graph with inputs / outputs at each step. |
| QA | QA scoring results (enqueued on flow finalisation / hangup). |
| Token | Token usage breakdown by provider. |
| Action | Browser action invocations and their results. |
Superadmin trace search
v0.9.14 adds a SUPERUSER trace-debug search surface in TelWeb for retained SigNoz data. Use it when you have a trace ID, call ID, or SIP/header value but do not yet have a conversation row open. The search uses the same SigNoz URL and API key configured under Platform settings. It accepts pasted SIP header lines (From:, To:, Call-ID:, CSeq:, User-Agent:, X-Call-ID:) and a raw ClickHouse SQL mode (sql: prefix). See Trace debug search for syntax and examples.
Query patterns
Use these as starting filters in SigNoz Logs / Traces:
service.name = "telpro" AND call_id = "<id>" # Kamailio SIP captures (TelPro)
service.name = "telsys" AND call_id = "<id>" # Asterisk PJSIP captures (Voice)
attributes.recordKind = "sip_message" AND call_id = "<id>"
service.name = "rtpengine" AND call_id = "<id>" # Media session
service.name = "telphi" AND trace_id = "<id>" # Full conversation trace
service.name = "telapi" AND trace_id = "<id>" # Channel WebSocket lifecycle
Some deployments still index legacy service.name = "kamailio" — see SIP signaling reference.
Common investigations
| Question | Start in SigNoz / TelWeb | Then check |
|---|---|---|
| Why did this call hang up? | TelWeb conversation > Debug > span tree and SIP ladder. | SIP signaling reference for BYE/CANCEL/481/487; TelPro/Voice ops for routing. |
| Why is there no audio? | Debug > SIP ladder for the call_id; RTPEngine spans. | TelPro no-audio matrix and RTP firewall ranges. |
| Did keypad / DTMF input arrive? | Debug > SIP ladder DTMF chips and timeline events. | Compare SIP INFO, RFC4733 SDP/media, ARI, and in-band evidence before debugging the flow node. |
| Which provider was slow? | Token + Debug logs filtered to service.name = telphi. | Voice logs if provider spans are missing. |
| Did billing or subscription checks fire? | Flow/API spans and TelAPI logs. | Ops / Tasker logs for background reconciliation. |
| Was QA scored? | QA tab and Tasker spans. | Ops if qaScoring jobs are absent. |
| Why is telemetry missing? | Check whether all services stopped reporting at once or only one host is missing. | Instance and container debugging and SigNoz operations. |
When to debug on the instance
Use SigNoz first when telemetry is present. Drop to the host or container when:
- a service or host has stopped sending telemetry entirely;
- Docker health checks are failing but no fresh logs appear in SigNoz;
- the local OTel collector cannot export to
10.0.1.10:4317; - startup fails before the collector is running;
- you need to validate local files, mounts, ports, or
docker composestate.
The shared command set lives in Instance and container debugging. Service-specific symptoms stay in each Operations page's Troubleshooting tab.
Stitching a retained trace into a conversation
When a known-good trace has aged out of a tenant's view but is still retained in SigNoz, you can re-link a Conversation row to that trace ID to make the Debug tab usable for support sessions:
UPDATE "Conversation"
SET "traceId" = '<retained-trace-id>'
WHERE "id" = '<conversation-id>';
The mapping is informational; nothing else moves. Use during outage post-mortems or paired support sessions.
See also
- SigNoz operations — run and repair the SigNoz backend.
- Instance and container debugging — SSH / Docker checks when telemetry is missing.
- SIP signaling reference — Kamailio/TelSys captures, OPTIONS/CANCEL/PRACK, response codes.
- TelPro operations — SIP, RTP, Janus, TURN.
- Voice operations — TelPhi and Asterisk call handling.