Monitoring in SigNoz
SigNoz is the central place to investigate a Delphi deployment. Every service host runs an OpenTelemetry collector that forwards logs, traces, metrics, and health signals to the SigNoz service at 10.0.1.10:4317 (OTLP gRPC) / 10.0.1.10:4318 (OTLP HTTP). The SigNoz UI is exposed through the Web service Caddy proxy at https://${DOMAIN_SIGNOZ}.
For running the SigNoz backend itself, use SigNoz operations. This page is about using SigNoz once telemetry is flowing.
What lands in SigNoz
| Signal | Examples | Typical use |
|---|---|---|
| Traces | TelAPI WebSocket lifecycle, TelPhi conversation spans, provider calls, flow execution | Follow a request or call end-to-end. |
| Logs | Application logs, tasker jobs, API errors, multiline Kamailio / RTPEngine / Janus output | Search for failures by service, trace ID, or call ID. |
| Metrics | Container CPU / memory, request rates, error rates, collector health, ClickHouse storage | Dashboards and alerts. |
| SIP / call spans | Kamailio SIP ladder, RTPEngine media session events, Asterisk / TelSys channel handling, DTMF evidence | Debug call setup, teardown, routing, keypad input, and no-audio cases. |
Kamailio and TelSys SIP captures, method/response-code reference, and capture-point glossary: SIP signaling reference.
Kamailio, RTPEngine, and Janus logs are converted into OTLP spans by the log-to-span sidecar on TelPro. Voice-side call handling lands through Voice collectors.
PII redaction (v0.9.13)
v0.9.13 adds coordinated PII controls in two layers: the local OTel collector scrubs telemetry before export to SigNoz, and TelPhi / TelAPI redact data written to Postgres (call-details timeline). Both layers read the same env vars from each service host's vars.yaml.
PII redaction is not transcript retention. Whether full transcript text is kept long-term is controlled by the app Recording toggle (see Recording and transcript retention). When recording is on, full transcript text is written to S3 with the recording artefact regardless of ENABLE_PII_LOGGING. When recording is off, full transcript text is not retained. PII settings only control what appears in Postgres, SigNoz, and traces — not whether the recording path persists complete text.
| Variable | Default | Meaning |
|---|---|---|
ENABLE_PII_LOGGING | false | When not true, redaction is active. In production, set true only for short support incidents, then revert. Staging/dev may leave it true when debugging with test traffic. |
MSISDN_UNMASKED_DIGITS | 4 | How many trailing digits to keep when masking phone numbers (for example +447911123456 → ***3456). |
Set both on every host that runs an OTel collector and on Voice (voiceai-telphi + voiceai-otel-collector) and API (telapi + voiceai-otel-collector). Other service collectors (Web, TelPro, Ops, Database, Squid) also honour the vars for log export even when they do not run TelPhi.
OTel collector (transform/pii)
Each service ships a transform/pii processor in its collector config. When ENABLE_PII_LOGGING is not true, log pipelines apply:
| Pattern | Redaction |
|---|---|
| Email addresses in log bodies | user@domain.tld → ****@domain.tld |
| E.164 MSISDNs in log bodies | Mask body digits; keep the last MSISDN_UNMASKED_DIGITS (for example ***3456) |
Card-like dddd-dddd-dddd-dddd sequences | ****-****-****-**** |
Conversation transcript logs (logType=conversation) | Turn text → ***; body → conversation (redacted) |
callerNumber log attribute | Last N digits only, prefixed with *** |
SIP ladder logs are not masked — records with event=sip_message keep raw SIP text so operators can debug call routing. That is intentional: scrubbing would break ladder analysis.
On the Voice host, a second processor (transform/pii_traces) scrubs trace span attributes before export:
transcription.text→***call.caller_number→ masked with the same trailing-digit rule
Restart the local voiceai-otel-collector (or re-run init.sh on that host) after changing these vars so the collector process picks up new values.
TelPhi (Voice)
When ENABLE_PII_LOGGING is not true, TelPhi redacts before persisting to the Conversation row:
- Transcript — every turn's
textis stored as***(node/role/timing metadata is kept). - Caller number — masked with
maskMsisdn()usingMSISDN_UNMASKED_DIGITS(same semantics as the collector).
TelWeb Timeline and conversation log dialogs therefore show redacted transcript text unless PII logging is enabled.
Recording and transcript retention: when the app has Recording on and the platform allows recording (RECORDING_ENABLED), TelPhi captures a full in-memory snapshot before Postgres redaction and writes complete transcript text to the S3 recording sidecar. That happens whether or not ENABLE_PII_LOGGING is true. Operators or tenant admins with bucket access can retrieve full content from recordings even while the DB timeline and SigNoz remain scrubbed. To avoid retaining transcript text entirely, leave Recording off on the app — PII redaction alone does not disable retention when recording is enabled. See Conversations → Recording and transcript retention.
TelAPI (browser actions)
TelAPI applies the same transcript redaction on persist for browser-action sessions when ENABLE_PII_LOGGING is not true. Browser-action flows are not recorded to S3, so there is no sidecar exception.
Operator guidance
- Production — keep
ENABLE_PII_LOGGING=false. Enable it only for short support or incident-debug sessions, then turn it off again and restart affected Voice/API hosts and collectors. - Staging / dev —
ENABLE_PII_LOGGING=trueis acceptable when you need full transcripts in SigNoz and TelWeb for integration testing. Use synthetic or consented test data; do not treat non-production as a reason to log real customer PII indefinitely. - Set
MSISDN_UNMASKED_DIGITSconsistently on Voice, API, and every collector host — mismatched values make SigNoz and the DB timeline disagree. - To verify scrubbing, place a test call with PII logging off, then check SigNoz logs/traces and the TelWeb conversation Timeline — transcript text should be
***, caller numbers partially masked. - Full variable reference: Environment variable index.
Dashboards
Custom Delphi dashboards ship with the deployment bundle under the SigNoz service's signoz/dashboards/ directory. Import them in SigNoz via Settings > Dashboards > Import.
| Dashboard | File | Answers |
|---|---|---|
| Delphi Overview | voiceai-overview.json | Is the platform healthy? Are calls flowing? Are errors rising? |
| Call Statistics | voiceai-call-statistics.json | Call volume, duration, success / failure, traffic trends. |
| Containers | voiceai-containers.json | CPU, memory, restarts, and container-level saturation. |
| Infrastructure | voiceai-infrastructure.json | Disk, network, CPU per host. |
The file names keep the legacy voiceai-* prefix that ships in the bundle. Dashboard titles should appear as Delphi in the SigNoz UI.
Useful upstream references:
Alerts worth creating
Start with a small set of noisy-but-actionable alerts, then tune thresholds per deployment:
| Area | Alert idea | Why it matters |
|---|---|---|
| Telemetry intake | No spans or logs received from a service for 5-10 minutes. | The service may be down, or the local OTel collector cannot reach SigNoz. |
| Call failures | Error rate or failed-call count exceeds a threshold. | Catches carrier, TelPro, Voice, and provider regressions. |
| Web / API | 5xx responses or request latency spike. | User-visible outage signal. |
| Voice | TelPhi provider latency or error spans spike. | AI provider or proxy path degraded. |
| Containers | Restart count increases, CPU / memory stays high. | Service instability before users report symptoms. |
| Storage | ClickHouse or data volume above 80%. | SigNoz retention will fail or queries will degrade. |
| Collectors | Collector export errors to 10.0.1.10:4317. | Telemetry gap: local debugging may be needed. |
Useful upstream references:
Investigating a call
Every call surfaces three correlated identifiers:
| Identifier | Where it lives | Used to find |
|---|---|---|
| Trace ID | SigNoz, TelPhi logs, TelWeb conversation row | Full distributed trace across TelAPI / TelPhi / TelSys / Kamailio. |
| Call ID | Kamailio / Asterisk / TelPhi | SIP ladder, RTPEngine sessions, ARI events. |
| Conversation ID | Postgres Conversation row, TelWeb URL | Persisted timeline, transcripts, QA scoring. |
These identifiers are available on the TelWeb conversation detail page.
TelWeb call detail tabs
| Tab | Shows |
|---|---|
| Debug | Separate Logs, Spans, and SIP Ladder panes for the trace. The SIP Ladder pane includes capture details, DTMF evidence, and text export for support tickets. |
| Timeline | Channel events ordered by ChannelMessage.timestamp. Chat, browser actions, audio segments, control. |
| Flow run | Per-node execution of the Flow Builder graph with inputs / outputs at each step. |
| QA | QA scoring results (enqueued on flow finalisation / hangup). |
| Token | Token usage breakdown by provider. |
| Action | Browser action invocations and their results. |
Superadmin trace search
v0.9.14 adds a SUPERUSER trace-debug search surface in TelWeb for retained SigNoz data. Use it when you have a trace ID, call ID, or SIP/header value but do not yet have a conversation row open. The search uses the same SigNoz URL and API key configured under Platform settings.
Query patterns
Use these as starting filters in SigNoz Logs / Traces:
service.name = "telpro" AND call_id = "<id>" # Kamailio SIP captures (TelPro)
service.name = "telsys" AND call_id = "<id>" # Asterisk PJSIP captures (Voice)
attributes.recordKind = "sip_message" AND call_id = "<id>"
service.name = "rtpengine" AND call_id = "<id>" # Media session
service.name = "telphi" AND trace_id = "<id>" # Full conversation trace
service.name = "telapi" AND trace_id = "<id>" # Channel WebSocket lifecycle
Some deployments still index legacy service.name = "kamailio" — see SIP signaling reference.
Common investigations
| Question | Start in SigNoz / TelWeb | Then check |
|---|---|---|
| Why did this call hang up? | TelWeb conversation > Debug > span tree and SIP ladder. | SIP signaling reference for BYE/CANCEL/481/487; TelPro/Voice ops for routing. |
| Why is there no audio? | Debug > SIP ladder for the call_id; RTPEngine spans. | TelPro no-audio matrix and RTP firewall ranges. |
| Did keypad / DTMF input arrive? | Debug > SIP ladder DTMF chips and timeline events. | Compare SIP INFO, RFC4733 SDP/media, ARI, and in-band evidence before debugging the flow node. |
| Which provider was slow? | Token + Debug logs filtered to service.name = telphi. | Voice logs if provider spans are missing. |
| Did billing or subscription checks fire? | Flow/API spans and TelAPI logs. | Ops / Tasker logs for background reconciliation. |
| Was QA scored? | QA tab and Tasker spans. | Ops if qaScoring jobs are absent. |
| Why is telemetry missing? | Check whether all services stopped reporting at once or only one host is missing. | Instance and container debugging and SigNoz operations. |
When to debug on the instance
Use SigNoz first when telemetry is present. Drop to the host or container when:
- a service or host has stopped sending telemetry entirely;
- Docker health checks are failing but no fresh logs appear in SigNoz;
- the local OTel collector cannot export to
10.0.1.10:4317; - startup fails before the collector is running;
- you need to validate local files, mounts, ports, or
docker composestate.
The shared command set lives in Instance and container debugging. Service-specific symptoms stay in each Operations page's Troubleshooting tab.
Stitching a retained trace into a conversation
When a known-good trace has aged out of a tenant's view but is still retained in SigNoz, you can re-link a Conversation row to that trace ID to make the Debug tab usable for support sessions:
UPDATE "Conversation"
SET "traceId" = '<retained-trace-id>'
WHERE "id" = '<conversation-id>';
The mapping is informational; nothing else moves. Use during outage post-mortems or paired support sessions.
See also
- SigNoz operations — run and repair the SigNoz backend.
- Instance and container debugging — SSH / Docker checks when telemetry is missing.
- SIP signaling reference — Kamailio/TelSys captures, OPTIONS/CANCEL/PRACK, response codes.
- TelPro operations — SIP, RTP, Janus, TURN.
- Voice operations — TelPhi and Asterisk call handling.