Monitoring in SigNoz
SigNoz is the central place to investigate a Delphi deployment. Every service host runs an OpenTelemetry collector that forwards logs, traces, metrics, and health signals to the SigNoz service at 10.0.1.10:4317 (OTLP gRPC) / 10.0.1.10:4318 (OTLP HTTP). The SigNoz UI is exposed through the Web service Caddy proxy at https://${DOMAIN_SIGNOZ}.
For running the SigNoz backend itself, use SigNoz operations. This page is about using SigNoz once telemetry is flowing.
What lands in SigNoz
| Signal | Examples | Typical use |
|---|---|---|
| Traces | TelAPI WebSocket lifecycle, TelPhi conversation spans, provider calls, flow execution | Follow a request or call end-to-end. |
| Logs | Application logs, tasker jobs, API errors, multiline Kamailio / RTPEngine / Janus output | Search for failures by service, trace ID, or call ID. |
| Metrics | Container CPU / memory, request rates, error rates, collector health, ClickHouse storage | Dashboards and alerts. |
| SIP / call spans | Kamailio SIP ladder, RTPEngine media session events, Asterisk / TelSys channel handling, DTMF evidence | Debug call setup, teardown, routing, keypad input, and no-audio cases. |
Kamailio, RTPEngine, and Janus logs are converted into OTLP spans by the log-to-span sidecar on TelPro. Voice-side call handling lands through Voice collectors.
PII redaction (v0.9.13)
v0.9.13 adds coordinated PII controls in two layers: the local OTel collector scrubs telemetry before export to SigNoz, and TelPhi / TelAPI redact data written to Postgres (call-details timeline). Both layers read the same env vars from each service host's vars.yaml.
| Variable | Default | Meaning |
|---|---|---|
ENABLE_PII_LOGGING | false | When not true, redaction is active. In production, set true only for short support incidents, then revert. Staging/dev may leave it true when debugging with test traffic. |
MSISDN_UNMASKED_DIGITS | 4 | How many trailing digits to keep when masking phone numbers (for example +447911123456 → ***3456). |
Set both on every host that runs an OTel collector and on Voice (voiceai-telphi + voiceai-otel-collector) and API (telapi + voiceai-otel-collector). Other service collectors (Web, TelPro, Ops, Database, Squid) also honour the vars for log export even when they do not run TelPhi.
OTel collector (transform/pii)
Each service ships a transform/pii processor in its collector config. When ENABLE_PII_LOGGING is not true, log pipelines apply:
| Pattern | Redaction |
|---|---|
| Email addresses in log bodies | user@domain.tld → ****@domain.tld |
| E.164 MSISDNs in log bodies | Mask body digits; keep the last MSISDN_UNMASKED_DIGITS (for example ***3456) |
Card-like dddd-dddd-dddd-dddd sequences | ****-****-****-**** |
Conversation transcript logs (logType=conversation) | Turn text → ***; body → conversation (redacted) |
callerNumber log attribute | Last N digits only, prefixed with *** |
SIP ladder logs are not masked — records with event=sip_message keep raw SIP text so operators can debug call routing. That is intentional: scrubbing would break ladder analysis.
On the Voice host, a second processor (transform/pii_traces) scrubs trace span attributes before export:
transcription.text→***call.caller_number→ masked with the same trailing-digit rule
Restart the local voiceai-otel-collector (or re-run init.sh on that host) after changing these vars so the collector process picks up new values.
TelPhi (Voice)
When ENABLE_PII_LOGGING is not true, TelPhi redacts before persisting to the Conversation row:
- Transcript — every turn's
textis stored as***(node/role/timing metadata is kept). - Caller number — masked with
maskMsisdn()usingMSISDN_UNMASKED_DIGITS(same semantics as the collector).
TelWeb Timeline and conversation log dialogs therefore show redacted transcript text unless PII logging is enabled.
Call recording exception: when recording is enabled and the call is sampled, TelPhi captures a full in-memory snapshot before redaction. The S3 recording sidecar keeps the complete transcript and caller number even while the DB row stays redacted. Operators with bucket access can still retrieve full content from recordings; SigNoz and the DB timeline remain scrubbed by default.
TelAPI (browser actions)
TelAPI applies the same transcript redaction on persist for browser-action sessions when ENABLE_PII_LOGGING is not true. Browser-action flows are not recorded to S3, so there is no sidecar exception.
Operator guidance
- Production — keep
ENABLE_PII_LOGGING=false. Enable it only for short support or incident-debug sessions, then turn it off again and restart affected Voice/API hosts and collectors. - Staging / dev —
ENABLE_PII_LOGGING=trueis acceptable when you need full transcripts in SigNoz and TelWeb for integration testing. Use synthetic or consented test data; do not treat non-production as a reason to log real customer PII indefinitely. - Set
MSISDN_UNMASKED_DIGITSconsistently on Voice, API, and every collector host — mismatched values make SigNoz and the DB timeline disagree. - To verify scrubbing, place a test call with PII logging off, then check SigNoz logs/traces and the TelWeb conversation Timeline — transcript text should be
***, caller numbers partially masked. - Full variable reference: Environment variable index.
Dashboards
Custom Delphi dashboards ship with the deployment bundle under the SigNoz service's signoz/dashboards/ directory. Import them in SigNoz via Settings > Dashboards > Import.
| Dashboard | File | Answers |
|---|---|---|
| Delphi Overview | voiceai-overview.json | Is the platform healthy? Are calls flowing? Are errors rising? |
| Call Statistics | voiceai-call-statistics.json | Call volume, duration, success / failure, traffic trends. |
| Containers | voiceai-containers.json | CPU, memory, restarts, and container-level saturation. |
| Infrastructure | voiceai-infrastructure.json | Disk, network, CPU per host. |
The file names keep the legacy voiceai-* prefix that ships in the bundle. Dashboard titles should appear as Delphi in the SigNoz UI.
Useful upstream references:
Alerts worth creating
Start with a small set of noisy-but-actionable alerts, then tune thresholds per deployment:
| Area | Alert idea | Why it matters |
|---|---|---|
| Telemetry intake | No spans or logs received from a service for 5-10 minutes. | The service may be down, or the local OTel collector cannot reach SigNoz. |
| Call failures | Error rate or failed-call count exceeds a threshold. | Catches carrier, TelPro, Voice, and provider regressions. |
| Web / API | 5xx responses or request latency spike. | User-visible outage signal. |
| Voice | TelPhi provider latency or error spans spike. | AI provider or proxy path degraded. |
| Containers | Restart count increases, CPU / memory stays high. | Service instability before users report symptoms. |
| Storage | ClickHouse or data volume above 80%. | SigNoz retention will fail or queries will degrade. |
| Collectors | Collector export errors to 10.0.1.10:4317. | Telemetry gap: local debugging may be needed. |
Useful upstream references:
Investigating a call
Every call surfaces three correlated identifiers:
| Identifier | Where it lives | Used to find |
|---|---|---|
| Trace ID | SigNoz, TelPhi logs, TelWeb conversation row | Full distributed trace across TelAPI / TelPhi / TelSys / Kamailio. |
| Call ID | Kamailio / Asterisk / TelPhi | SIP ladder, RTPEngine sessions, ARI events. |
| Conversation ID | Postgres Conversation row, TelWeb URL | Persisted timeline, transcripts, QA scoring. |
These identifiers are available on the TelWeb conversation detail page.
TelWeb call detail tabs
| Tab | Shows |
|---|---|
| Debug | Logs, span tree, SIP ladder, and DTMF evidence for the trace. Logs query the SigNoz Logs API filtered to trace_id = <traceId>. |
| Timeline | Channel events ordered by ChannelMessage.timestamp. Chat, browser actions, audio segments, control. |
| Flow run | Per-node execution of the Flow Builder graph with inputs / outputs at each step. |
| QA | QA scoring results (enqueued on flow finalisation / hangup). |
| Token | Token usage breakdown by provider. |
| Action | Browser action invocations and their results. |
Query patterns
Use these as starting filters in SigNoz Logs / Traces:
service.name = "telphi" AND trace_id = "<id>" # Full conversation trace
service.name = "telapi" AND trace_id = "<id>" # Channel WebSocket lifecycle
service.name = "kamailio" AND call_id = "<id>" # SIP ladder for the call
service.name = "rtpengine" AND call_id = "<id>" # Media session
service.name = "telsys" AND call_id = "<id>" # Asterisk channel handling
Common investigations
| Question | Start in SigNoz / TelWeb | Then check |
|---|---|---|
| Why did this call hang up? | TelWeb conversation > Debug > span tree. | TelPro / Voice troubleshooting if the terminating span points at SIP or Asterisk. |
| Why is there no audio? | Debug > SIP ladder for the call_id; RTPEngine spans. | TelPro no-audio matrix and RTP firewall ranges. |
| Did keypad / DTMF input arrive? | Debug > SIP ladder DTMF chips and timeline events. | Compare SIP INFO, RFC4733 SDP/media, ARI, and in-band evidence before debugging the flow node. |
| Which provider was slow? | Token + Debug logs filtered to service.name = telphi. | Voice logs if provider spans are missing. |
| Did billing or subscription checks fire? | Flow/API spans and TelAPI logs. | Ops / Tasker logs for background reconciliation. |
| Was QA scored? | QA tab and Tasker spans. | Ops if qaScoring jobs are absent. |
| Why is telemetry missing? | Check whether all services stopped reporting at once or only one host is missing. | Instance and container debugging and SigNoz operations. |
When to debug on the instance
Use SigNoz first when telemetry is present. Drop to the host or container when:
- a service or host has stopped sending telemetry entirely;
- Docker health checks are failing but no fresh logs appear in SigNoz;
- the local OTel collector cannot export to
10.0.1.10:4317; - startup fails before the collector is running;
- you need to validate local files, mounts, ports, or
docker composestate.
The shared command set lives in Instance and container debugging. Service-specific symptoms stay in each Operations page's Troubleshooting tab.
Stitching a retained trace into a conversation
When a known-good trace has aged out of a tenant's view but is still retained in SigNoz, you can re-link a Conversation row to that trace ID to make the Debug tab usable for support sessions:
UPDATE "Conversation"
SET "traceId" = '<retained-trace-id>'
WHERE "id" = '<conversation-id>';
The mapping is informational; nothing else moves. Use during outage post-mortems or paired support sessions.
See also
- SigNoz operations — run and repair the SigNoz backend.
- Instance and container debugging — SSH / Docker checks when telemetry is missing.
- TelPro operations — SIP, RTP, Janus, TURN.
- Voice operations — TelPhi and Asterisk call handling.