Telemetry, SigNoz and alerts

Use this page to understand where Delphi telemetry goes, how to open it and which alerts watch it. Each host role runs its own OpenTelemetry collector. All collectors send to one SigNoz backend, which stores logs, traces and metrics and provides the query views and alerts.

Availability: this page is for the deployment and observability operator. You need access to the SigNoz user interface and, for alert setup, a SigNoz service-account key. A Delphi account or team role does not grant SigNoz access. The telphi.heartbeat span, the revised TelphiNoTraces alert and the export of web audit records to SigNoz are available from 0.9.18.

How telemetry reaches SigNoz

On a small screen, scroll the diagram horizontally to read every branch.

1. Applications send OpenTelemetry logs and traces to the collector on their own host. Container output is read from the Docker log files on the same host.
2. Each role collector forwards everything to OTEL_BACKEND_ENDPOINT, which the Compose files set to OTEL_BACKEND_HOST on port 4317.
3. The SigNoz ingester writes the records to ClickHouse. The SigNoz app reads ClickHouse for its views, alerts and dashboards.
4. An optional export collector forwards selected security events to external destinations. See Verify security event export and retention.

Collectors on each role

Role	Application OTLP	Log files read	Metrics	Traces
Web	Yes	Docker container logs	Host, containers and application OTLP metrics	Yes
API	Yes	Docker container logs	Host and containers	Yes
Voice	Yes	Docker container logs	Host and containers	Yes
TelPro	Yes	Docker container logs	Host and containers	Yes
Ops	No	Docker container logs	Host and containers	No
Database	No	Docker container logs, PostgreSQL log	Host, containers, PostgreSQL and Redis	No
Squid	No	Docker container logs, Squid access log	Host and containers	No

Every collector exposes a health check on port 13133. A healthy endpoint shows that the collector process responds; it does not show that records reach SigNoz.

Behavior that applies to all roles:

• Service filter: log pipelines keep only records whose service is telpro, telweb, telapi, tasker, scaler, telsys or telphi. Records without one of these names are dropped.
• Personal data: every log pipeline applies a transform/pii step. The Voice collector also masks trace attributes.
• Audit records: the Web, API and Ops collectors mark audit records so you can filter them. See Audit records in SigNoz.
• TelPro: the collector promotes the SIP proxy's structured fields and drops its unstructured native lines. See the SIP signaling and trunk log reference.
• Log-to-span sidecar: on the TelPro and Voice hosts, log-to-span turns telephony container logs and captured SIP messages into trace spans and sends them to the local collector.

The Ops collector has no OTLP receiver. To find out why a record from a role is missing, use Check the delivery path.

Change a collector configuration

Use this procedure to change a receiver, processor, export destination or masking rule. The role's deployment bundle contains otel-collector-config.yaml; the host copy is /opt/services/<role>/otel-collector-config.yaml. Change the bundle that owns the file so a later configuration sync does not overwrite the correction.

1. Save the previous bundle reference and effective collector configuration. Identify the affected host role and signal: logs, traces or metrics. Read the ordered chain in service.pipelines; declaring a processor does not activate it unless that chain includes it.
2. Change the owning bundle. For a substituted ${env:NAME} value, update its effective configuration source instead. The configuration layers determine which value wins; changing a fallback in vars.yaml does not override an existing parameter or secret.
3. For masking changes, edit transform/pii and its log_statements for logs. Voice trace masking uses transform/pii_traces with trace_statements; API, Web and TelPro include trace statements in transform/pii. Check each relevant signal separately. Ops, Database and Squid have no trace pipeline to update.
4. Validate the changed YAML and transformation expressions with the collector image selected for the deployment in a test environment. error_mode: ignore allows processing to continue after a statement error; it does not prove that the value was masked. Inspect startup errors and processing errors separately.
5. Apply the bundle through the deployment update procedure. A configuration sync alone changes files, not the running process. From the affected role directory, ./update.sh --restart-only resolves configuration and recreates containers using cached images. This can restart the role's application containers too; schedule the service impact before running it.
6. Inspect the collector's startup logs and send synthetic examples through every changed signal. Query the exported records in SigNoz and inspect both the body and attributes. Confirm expected masking, retained non-sensitive fields and continued delivery. A healthy collector endpoint alone is insufficient. If verification fails, restore the previous bundle and configuration values, repeat the controlled restart and verify again.

Adjust a masking rule

A replace_pattern statement names a target field, a regular expression and replacement text. The where condition determines which records it changes. Preserve the shipped type checks and the ENABLE_PII_LOGGING condition when changing a rule. The diagnostic protection settings explain the effect and scope of that setting and the trailing-digit allowance.

Use the existing rule for the same field as the starting point. Changing a log-body rule does not change a separately exported attribute or span. Keep capture-group references that retain surrounding text when changing the mask characters. Validate replacement escaping after configuration substitution with the selected collector image; YAML parsing alone cannot establish the resulting transformation.

For a different trailing-digit allowance, change MSISDN_UNMASKED_DIGITS in its effective configuration source and restart the affected role. The collector substitutes the value into the phone-number expressions. Test short numbers, formatted numbers and full international numbers with synthetic data; do not assume every numeric string matches the same expression.

Statements run in order. A later assignment can copy an unmasked attribute back into a body that an earlier statement masked. Inspect the whole active chain, then test the exported value. Collector masking only covers records traversing that collector; it cannot remove copies already written or exported elsewhere.

SigNoz app, ingester and ClickHouse

The SigNoz host runs several separate components:

Component	What it does
Ingester	An OpenTelemetry collector that accepts OTLP on 4317 (gRPC) and 4318 (HTTP) and writes to ClickHouse.
ClickHouse	Stores logs, traces and metrics. ZooKeeper runs alongside it for coordination.
SigNoz app	Serves the user interface and API on port 8080, evaluates alert rules and runs dashboards.
Export collector	Optional. Listens on 4327 and forwards selected security events; its health check is on 13134.

Each component has its own version. The declared versions are listed in Declared upstream versions. Do not set the ingester to the app version just to make them match.

Upgrade or migrate the SigNoz host

Identify the installed layout before changing versions. The managed Foundry layout uses /opt/services/signoz/pours/deployment/compose.yaml; the older layout uses /opt/signoz-repo/deploy/docker. The SigNoz application, ingester, ClickHouse and export collector have separate pins. Record their effective versions and the configuration-bundle reference before the change.

For an existing Foundry installation:

1. Establish a recoverable backup of the SigNoz data and SQLite metastore, which holds accounts, dashboards and alert configuration. Check the selected SigNoz release's migration requirements and intermediate-version instructions. A rollback of an image does not reverse a metastore or ClickHouse schema migration.
2. From /opt/services/signoz, apply the reviewed application version with ./update.sh --signoz-version <version>. Do not add --restart-only to an image upgrade: it uses cached images. The update changes the bootstrap version and attempts to update the corresponding SSM parameter; check both results. The bootstrap pin takes precedence over SSM, so reconcile a failed parameter write before rebuilding the host.
3. Initialization re-fetches the selected configuration bundle and regenerates the Foundry deployment. Recheck the effective ingester, ClickHouse and export-collector images afterward; the application version does not select matching versions for those components.
4. Run ./verify-signoz-usable.sh, confirm /api/v1/version, and inspect the generated Compose service state. This probe tests that OTLP reaches writable ClickHouse storage. Also verify existing dashboards, saved views, ingestion and the alert rules described below before declaring the upgrade complete.

For a legacy installation, use ./init.sh --migrate-from-legacy from the SigNoz service directory during a planned telemetry maintenance window. Its preflight saves the old Compose/configuration under /opt/deployment/signoz-legacy-backup-*, preserves data volumes and stops the old stack. This is not a substitute for a consistent data/metastore backup. Initialization writes /opt/deployment/signoz-foundry-migrated only after the usability probe passes. During migration, initialization also attempts replica recovery for read-only ClickHouse tables; include that database change in the maintenance plan and verify the backup first.

If migration fails, retain the saved configuration and data and stop before repeating destructive recovery actions. Compare the ClickHouse replica/shard macros with the preserved legacy values; a mismatch can leave tables read-only. Coordinate repair or restoration with the database/observability owner. Reusing the old Compose file alone is not evidence that changed schemas are compatible. Do not remove volumes to resolve an initialization error.

Open SigNoz and its views

The Web host's Caddy server publishes the SigNoz user interface at the domain in DOMAIN_SIGNOZ. It forwards requests to OTEL_BACKEND_HOST on SIGNOZ_PORT (default 8080). Caddy adds no Delphi sign-in in front of SigNoz, so SigNoz's own accounts control access.

1. Open https://signoz.example.com, using the domain from your DOMAIN_SIGNOZ setting, and sign in with your SigNoz account.
2. For log records, open the SigNoz logs view. Filter by service name, for example telphi or telapi, and narrow the time range to the incident.
3. For traces, open the SigNoz traces view and filter by the same service name or a trace ID.
4. Open one record and read its attributes before adding more filters. Field meanings are listed in Log levels and object structure.

You can also search SigNoz without opening it. Admin Settings → Log Search (Trace Debug) queries SigNoz by trace ID, SIP Call-ID or header; see Find a trace without an open conversation. It needs the SigNoz URL and API key in Platform Settings.

Search the logs of one conversation

In a conversation's Debug tab, the Logs view lists the call's records. From 0.9.18, its search also matches attribute values, up to 20,000 characters of attribute text per record. See Search the logs of a call.

TelPhi heartbeat and the no-traces alert

TelPhi emits a telphi.heartbeat span on every server heartbeat, by default every 10 seconds. It emits the span even when no call is active, so an idle server still produces traces. The span is emitted only when OpenTelemetry is enabled for TelPhi.

Span attribute	Meaning
heartbeat.server_id	Server registration ID
heartbeat.server_address	Address the server registers
heartbeat.status	Reported server status
heartbeat.active_calls	Active calls at the time of the heartbeat
heartbeat.used_rtp_ports	RTP ports in use
heartbeat.interval_ms	Heartbeat interval in milliseconds

The TelphiNoTraces alert fires when no trace with service name telphi arrives for 10 minutes. Because the heartbeat always produces traces, silence means one of these:

• The TelPhi process is down or unhealthy.
• The collector on the Voice host is down or unreachable.
• The SigNoz ingester is unreachable from that host.

If the span cannot be emitted, TelPhi logs the warning Failed to emit heartbeat span. Check container health with service health before restarting anything.

Alert rules

Delphi provides these SigNoz alert rules:

Alert	Signal	Severity	Window	Fires when
CallErrorRateSpike	Traces	critical	5 minutes	More than 10% of voice pipeline spans fail, per service
SipTrunkDown	Traces	critical	5 minutes	Three or more SIP OPTIONS probe spans fail for the same trunk
TelphiNoTraces	Traces	critical	10 minutes	No telphi traces arrive
SttTtsProviderFailure	Traces	warning	5 minutes	More than 10 speech-to-text or text-to-speech spans fail
ConversationRuntimeCrash	Logs	warning	10 minutes	More than 5 conversation runtime errors of selected crash types from telphi appear

Apply or update the alert rules

1. In SigNoz, create a service account with the Editor or Admin role and store its key as SIGNOZ_API_KEY for the SigNoz host. If you do not manage SigNoz accounts, ask your SigNoz administrator.
2. On the SigNoz host, run ./apply-alert-rules.sh in /opt/services/signoz/, or let the host initialization run it. Without SIGNOZ_API_KEY, the script skips the rule sync.
3. The script creates missing rules and updates existing rules with the same name. It does not delete rules.
4. A rule without its own notification channel gets the channels listed in SIGNOZ_ALERT_CHANNELS (comma-separated). If that is empty, it uses channels named Error or Teams, otherwise the first available channel.

To verify, open the alert list in SigNoz and confirm that all five rules exist with a notification channel.

Dashboards

Delphi provides these dashboards as JSON files in /opt/services/signoz/ on the SigNoz host. Import them in SigNoz through Dashboards → New Dashboard → Import JSON.

• VoiceAI Call Statistics
• VoiceAI Container Metrics
• VoiceAI Error Budget
• VoiceAI Infrastructure Health
• VoiceAI Platform Overview
• VoiceAI SIP Trunk Health
• VoiceAI Transaction Logs

Audit records in SigNoz

From 0.9.18, audit records that the web application writes to the database are also sent to SigNoz after the database change commits. If the change is rolled back, no record is sent. Delivery is best effort: a failed export does not undo the change, and the application does not send the record again.

In SigNoz, audit records carry logType audit and recordKind Audit. A record exported from the database also carries auditLogId, the ID of the database row. Security-relevant actions carry security_event and security_category; see Audit action catalogue.

When an application writes an audit record both directly and to its console, the Web, API and Ops collectors drop the console copy that is marked audit.delivery_role: mirror. The markers are explained in Audit transport markers.

Personal data in telemetry

Collectors mask selected personal data before export unless ENABLE_PII_LOGGING is true; read Protect diagnostic logs before you share any record.

Verify monitoring

1. In SigNoz, filter traces by service telphi for the last 15 minutes. You see telphi.heartbeat spans from every running TelPhi server.
2. Filter logs by each service you operate and confirm that recent records arrive.
3. Confirm that the five alert rules exist and that TelphiNoTraces is not firing.

Collector health and visible records show that the tested path works now; they do not prove that older records were kept.

Related

• Logs and diagnostics: to find the first missing stage when records do not arrive.
• SIP signaling and trunk log reference: to read SIP proxy, capture and trunk health records.
• Verify security event export and retention: to forward security events to an external destination.
• Release compatibility: to check SigNoz component versions.
• Check service health and intervene safely: to check a collector or service container.