Telemetry, SigNoz and alerts

Use this page to understand where Delphi telemetry goes, how to open it and which alerts watch it. Each host role runs its own OpenTelemetry collector. All collectors send to one SigNoz backend, which stores logs, traces and metrics and provides the query views and alerts.

Availability: this page is for the deployment and observability operator. You need access to the SigNoz user interface and, for alert setup, a SigNoz service-account key. A Delphi account or team role does not grant SigNoz access. The telphi.heartbeat span, the revised TelphiNoTraces alert and the export of web audit records to SigNoz are available from 0.9.18.

How telemetry reaches SigNoz

On a small screen, scroll the diagram horizontally to read every branch.

1. Applications send OpenTelemetry logs and traces to the collector on their own host. Container output is read from the Docker log files on the same host.
2. Each role collector forwards everything to OTEL_BACKEND_ENDPOINT, which the Compose files set to OTEL_BACKEND_HOST on port 4317.
3. The SigNoz ingester writes the records to ClickHouse. The SigNoz app reads ClickHouse for its views, alerts and dashboards.
4. An optional export collector forwards selected security events to external destinations. See Verify security event export and retention.

Collectors on each role

Role	Application OTLP	Log files read	Metrics	Traces
Web	Yes	Docker container logs	Host, containers and application OTLP metrics	Yes
API	Yes	Docker container logs	Host and containers	Yes
Voice	Yes	Docker container logs	Host and containers	Yes
TelPro	Yes	Docker logs and operation transition file	Host and containers	Yes
Ops	No	Docker container logs	Host and containers	No
Database	No	Docker container logs, PostgreSQL log	Host, containers, PostgreSQL and Redis	No
Squid	No	Docker container logs, Squid access log	Host and containers	No

Every collector exposes a health check on port 13133. A healthy endpoint shows that the collector process responds; it does not show that records reach SigNoz.

Behavior that applies to all roles:

• Service filter: log pipelines keep only records whose service is telpro, telweb, telapi, tasker, scaler, telsys or telphi. Records without one of these names are dropped.
• Personal data: every log pipeline applies a transform/pii step. The Voice collector also masks trace attributes.
• Audit records: the Web, API and Ops collectors mark audit records so you can filter them. See Audit records in SigNoz.
• TelPro: the collector promotes the SIP proxy's structured fields and drops its unstructured native lines. See the SIP signaling and trunk log reference.
• Log-to-span sidecar: on the TelPro and Voice hosts, log-to-span turns telephony container logs and captured SIP messages into trace spans and sends them to the local collector.

Availability — Ops log fields: The following structure is available from 0.9.19 and requires the matching Ops collector configuration. It does not change Ops transport support.

The Ops collector reads Tasker and Scaler's structured container output. It promotes the parsed Pino fields to top-level log attributes, so filters use jobId, attempt or operation.phase directly, without a pino. prefix. Numeric fields remain numeric; the readable message stays in the body and the log level becomes severity. Existing collector attributes take precedence if names collide. A valid application trace ID remains attached to the log record for correlation; this does not mean Ops exports spans. The nested Pino duplicate is removed before the normal personal-data redaction step.

Deploy the matching Ops collector configuration, then inspect a newly emitted Tasker or Scaler record in SigNoz: confirm its service, severity and top-level job or operation fields before enabling alerts. Old stored records retain their previous nested shape; this change does not rewrite history. Ordinary logs do not automatically become operation events—use the explicit envelope below for lifecycle dashboards.

The Ops collector has no OTLP receiver. To find out why a record from a role is missing, use Check the delivery path.

Monitor maintenance and scaling operations

Availability: The operation envelope below is available from 0.9.19. Deploy the matching application images and role configuration bundles before building alerts around it. Existing installations may have only the earlier free-text messages. Infrastructure operators need access to SigNoz; changing trunk or edge maintenance still requires the product's Super Admin permission. This contract does not install dashboards or alert rules.

Search logs for event.name = delphi.operation and event.version = 1. These records describe state changes in trunk maintenance, TelPro maintenance/certificate updates, Voice/API configuration refresh and autoscaling. Keep the emitting service.name filter separate from operation.service: for example, Tasker can report an operation affecting TelPro.

Shared event fields

Attribute	Meaning
event.name, event.version	delphi.operation, integer 1. Filter by both when using this contract.
event.id	Identifier of one emitted record. Deduplicate repeated delivery of that record by this value.
operation.kind	service_refresh, trunk_maintenance, autoscale or instance_lifecycle.
operation.phase	Observed step or outcome; interpret it using the catalogue below.
operation.service	Affected service: telpro, telphi or telapi.
operation.component	Reporting component: tasker, scaler, telpro-reconciler or control-plane.
operation.id, tasker.job.id	Available operation/command and parent job correlation. Not every event has both.
resource.id, resource.hostname	Available target identifier and hostname. For trunk maintenance, the resource ID is the technical trunk ID.
operation.strategy	Available action, such as drain-only, resume, drain-restart, reload-only, scale_up or scale_down.
operation.target_component	TelPro reconciler target, such as server-tls, kamailio, rtpengine or host.
operation.duration_ms	Duration supplied by the reporting step, when available; not automatically the whole operation duration.

Optional fields are absent or null when unavailable. The envelope excludes configuration payloads, certificates, private keys, caller numbers, SIP headers and free-text failure details. Use the correlated diagnostic records for investigation under the diagnostic protection settings.

Interpret operation phases

Operation	Events and interpretation
Trunk maintenance	requested precedes the configuration write. applied means the database update and shared Redis publication returned successfully; it is not an acknowledgement from every edge. failed records a write/publication failure. rolled_back means the previous database values and Redis flag were restored; rollback_failed requires investigation and reconciliation.
TelPro refresh/maintenance	Tasker reports job progress; the host reconciler reports steps such as acknowledged, draining, waiting, fetching_env, restarting, verifying, completed, failed and skipped. Certificate paths additionally report validation_failed, fallback_verified, rolled_back or rollback_failed. A verified runtime rollback still requires authoritative source recovery before Resume. Read the strategy and target component alongside the phase: completing drain-only intentionally leaves admission stopped, while completing resume restores admission.
Voice/API refresh	Tasker and Scaler report progress including queued, scaling_up, waiting_healthy, draining, waiting_drain, decommissioning, completed, failed and skipped. A failed attempt can be followed by another attempt. Use the job's final result and current readiness before declaring recovery.
Autoscaling	requested records an action selected by the decision loop. dispatch_returned only means its adapter returned; a drain may still be queued. The separate instance_lifecycle kind distinguishes provisioned from ready, and removed from inventory_removed when only local inventory was cleaned up. Instance lifecycle records also occur for configuration-refresh replacements; do not count them all as autoscaling. Decision-loop correlation is not a durable provisioning transaction ID; correlate instance events by target and time as well.

Events are observations, not an exactly-once audit trail. Repeated phase polls are suppressed in a bounded process-local cache; restarts or eviction can produce another observation. Different components can legitimately report the same phase. Group by operation, resource and reporting component before deriving durations; do not sum all completed records as completed jobs. A missing completion is not proof that an operation failed: check current state and telemetry delivery.

For safe operating procedures, use edge maintenance and the deployment update procedure. A successful configuration refresh does not prove that a new application image was deployed.

Delivery and verification

Tasker, Scaler and the control plane use their existing structured application logging path. TelPro's host reconciler also writes allowlisted JSON lines to /var/log/delphi/telpro-operations.jsonl. The TelPro collector reads that file and one rotated predecessor through filelog/operations, with service.name = telpro and recordKind = operation. The files are root-only; the collector mounts the log directory read-only. Rotation occurs at approximately 1 MiB per file. Persistent reader offsets live separately under /var/lib/delphi/otel-file-offsets.

This closes the host-journal ingestion gap for operation transitions; it does not ingest all systemd journal messages. Logging failure does not abort a maintenance operation. Bounded files, export retries and queues are best-effort delivery, not guaranteed retention during a long outage. Configure SigNoz retention independently and alert on telemetry silence separately from operation failure.

After rollout, perform an authorized synthetic maintenance action and query its exact job/operation ID. Verify the expected control-plane, worker and host observations, timestamps, target and severity in SigNoz. Inspect the exported attributes and body for sensitive data. Check failure and restoration paths as well as success. Collector health or the existence of a JSON file alone does not prove delivery.

Dashboard and alarm design

Build a recent-operations table grouped by kind, target and operation ID, showing the latest phase, first/last observation and reporting component. Use failed and rollback_failed as investigation signals; distinguish retries and later recovery. Derive a stuck-operation alarm from an agreed duration budget plus current runtime state and telemetry freshness. Do not use absence of a completion record as the only alarm condition.

Keep operation IDs, job IDs and host/trunk identifiers as log dimensions. Do not promote unbounded IDs into metric labels. No new per-trunk time-series gauge is exported by this feature: the trunk list reads lifecycle-maintained inbound call totals with freshness and integrity checks. Counting call-related log events is not a reliable active-call gauge.

Existing trunk health signals remain compatible: event.name = alarm with alarm.name = sip_trunk_health records the health monitor's debounced state changes, while the SIP proxy's event = trunk_call_rejected records rejected new calls. Planned maintenance rejections are expected; correlate them with admission state before paging. Fresh OPTIONS reachability, health-monitor alarm state and configured admission have different meanings and must not be collapsed into one red/green status.

Change a collector configuration

Use this procedure to change a receiver, processor, export destination or masking rule. The role's deployment bundle contains otel-collector-config.yaml; the host copy is /opt/services/<role>/otel-collector-config.yaml. Change the bundle that owns the file so a later configuration sync does not overwrite the correction.

1. Save the previous bundle reference and effective collector configuration. Identify the affected host role and signal: logs, traces or metrics. Read the ordered chain in service.pipelines; declaring a processor does not activate it unless that chain includes it.
2. Change the owning bundle. For a substituted ${env:NAME} value, update its effective configuration source instead. The configuration layers determine which value wins; changing a fallback in vars.yaml does not override an existing parameter or secret.
3. For masking changes, edit transform/pii and its log_statements for logs. Voice trace masking uses transform/pii_traces with trace_statements; API, Web and TelPro include trace statements in transform/pii. Check each relevant signal separately. Ops, Database and Squid have no trace pipeline to update.
4. Validate the changed YAML and transformation expressions with the collector image selected for the deployment in a test environment. error_mode: ignore allows processing to continue after a statement error; it does not prove that the value was masked. Inspect startup errors and processing errors separately.
5. Apply the bundle through the deployment update procedure. A configuration sync alone changes files, not the running process. From the affected role directory, ./update.sh --restart-only resolves configuration and recreates containers using cached images. This can restart the role's application containers too; schedule the service impact before running it.
6. Inspect the collector's startup logs and send synthetic examples through every changed signal. Query the exported records in SigNoz and inspect both the body and attributes. Confirm expected masking, retained non-sensitive fields and continued delivery. A healthy collector endpoint alone is insufficient. If verification fails, restore the previous bundle and configuration values, repeat the controlled restart and verify again.

Adjust a masking rule

A replace_pattern statement names a target field, a regular expression and replacement text. The where condition determines which records it changes. Preserve the shipped type checks and the ENABLE_PII_LOGGING condition when changing a rule. The diagnostic protection settings explain the effect and scope of that setting and the trailing-digit allowance.

Use the existing rule for the same field as the starting point. Changing a log-body rule does not change a separately exported attribute or span. Keep capture-group references that retain surrounding text when changing the mask characters. Validate replacement escaping after configuration substitution with the selected collector image; YAML parsing alone cannot establish the resulting transformation.

For a different trailing-digit allowance, change MSISDN_UNMASKED_DIGITS in its effective configuration source and restart the affected role. The collector substitutes the value into the phone-number expressions. Test short numbers, formatted numbers and full international numbers with synthetic data; do not assume every numeric string matches the same expression.

Statements run in order. A later assignment can copy an unmasked attribute back into a body that an earlier statement masked. Inspect the whole active chain, then test the exported value. Collector masking only covers records traversing that collector; it cannot remove copies already written or exported elsewhere.

SigNoz app, ingester and ClickHouse

The SigNoz host runs several separate components:

Component	What it does
Ingester	An OpenTelemetry collector that accepts OTLP on 4317 (gRPC) and 4318 (HTTP) and writes to ClickHouse.
ClickHouse	Stores logs, traces and metrics. ZooKeeper runs alongside it for coordination.
SigNoz app	Serves the user interface and API on port 8080, evaluates alert rules and runs dashboards.
Export collector	Optional. Listens on 4327 and forwards selected security events; its health check is on 13134.

Each component has its own version. The declared versions are listed in Declared upstream versions. Do not set the ingester to the app version just to make them match.

Upgrade or migrate the SigNoz host

Identify the installed layout before changing versions. The managed Foundry layout uses /opt/services/signoz/pours/deployment/compose.yaml; the older layout uses /opt/signoz-repo/deploy/docker. The SigNoz application, ingester, ClickHouse and export collector have separate pins. Record their effective versions and the configuration-bundle reference before the change.

For an existing Foundry installation:

1. Establish a recoverable backup of the SigNoz data and SQLite metastore, which holds accounts, dashboards and alert configuration. Check the selected SigNoz release's migration requirements and intermediate-version instructions. A rollback of an image does not reverse a metastore or ClickHouse schema migration.
2. From /opt/services/signoz, apply the reviewed application version with ./update.sh --signoz-version <version>. Do not add --restart-only to an image upgrade: it uses cached images. The update changes the bootstrap version and attempts to update the corresponding SSM parameter; check both results. The bootstrap pin takes precedence over SSM, so reconcile a failed parameter write before rebuilding the host.
3. Initialization re-fetches the selected configuration bundle and regenerates the Foundry deployment. Recheck the effective ingester, ClickHouse and export-collector images afterward; the application version does not select matching versions for those components.
4. Run ./verify-signoz-usable.sh, confirm /api/v1/version, and inspect the generated Compose service state. This probe tests that OTLP reaches writable ClickHouse storage. Also verify existing dashboards, saved views, ingestion and the alert rules described below before declaring the upgrade complete.

For a legacy installation, use ./init.sh --migrate-from-legacy from the SigNoz service directory during a planned telemetry maintenance window. Its preflight saves the old Compose/configuration under /opt/deployment/signoz-legacy-backup-*, preserves data volumes and stops the old stack. This is not a substitute for a consistent data/metastore backup. Initialization writes /opt/deployment/signoz-foundry-migrated only after the usability probe passes. During migration, initialization also attempts replica recovery for read-only ClickHouse tables; include that database change in the maintenance plan and verify the backup first.

If migration fails, retain the saved configuration and data and stop before repeating destructive recovery actions. Compare the ClickHouse replica/shard macros with the preserved legacy values; a mismatch can leave tables read-only. Coordinate repair or restoration with the database/observability owner. Reusing the old Compose file alone is not evidence that changed schemas are compatible. Do not remove volumes to resolve an initialization error.

Open SigNoz and its views

The Web host's Caddy server publishes the SigNoz user interface at the domain in DOMAIN_SIGNOZ. It forwards requests to OTEL_BACKEND_HOST on SIGNOZ_PORT (default 8080). Caddy adds no Delphi sign-in in front of SigNoz, so SigNoz's own accounts control access.

1. Open https://signoz.example.com, using the domain from your DOMAIN_SIGNOZ setting, and sign in with your SigNoz account.
2. For log records, open the SigNoz logs view. Filter by service name, for example telphi or telapi, and narrow the time range to the incident.
3. For traces, open the SigNoz traces view and filter by the same service name or a trace ID.
4. Open one record and read its attributes before adding more filters. Field meanings are listed in Log levels and object structure.

You can also search SigNoz without opening it. Admin Settings → Log Search (Trace Debug) queries SigNoz by trace ID, SIP Call-ID or header; see Find a trace without an open conversation. It needs the SigNoz URL and API key in Platform Settings.

Search the logs of one conversation

In a conversation's Debug tab, the Logs view lists the call's records. From 0.9.18, its search also matches attribute values, up to 20,000 characters of attribute text per record. See Search the logs of a call.

TelPhi heartbeat and the no-traces alert

TelPhi emits a telphi.heartbeat span on every server heartbeat, by default every 10 seconds. It emits the span even when no call is active, so an idle server still produces traces. The span is emitted only when OpenTelemetry is enabled for TelPhi.

Span attribute	Meaning
heartbeat.server_id	Server registration ID
heartbeat.server_address	Address the server registers
heartbeat.status	Reported server status
heartbeat.active_calls	Active calls at the time of the heartbeat
heartbeat.used_rtp_ports	RTP ports in use
heartbeat.interval_ms	Heartbeat interval in milliseconds

The TelphiNoTraces alert fires when no trace with service name telphi arrives for 10 minutes. Because the heartbeat always produces traces, silence means one of these:

• The TelPhi process is down or unhealthy.
• The collector on the Voice host is down or unreachable.
• The SigNoz ingester is unreachable from that host.

If the span cannot be emitted, TelPhi logs the warning Failed to emit heartbeat span. Check container health with service health before restarting anything.

Alert rules

Delphi provides these SigNoz alert rules:

Alert	Signal	Severity	Window	Fires when
CallErrorRateSpike	Traces	critical	5 minutes	More than 10% of voice pipeline spans fail, per service
SipTrunkDown	Traces	critical	5 minutes	Three or more SIP OPTIONS probe spans fail for the same trunk
TelphiNoTraces	Traces	critical	10 minutes	No telphi traces arrive
SttTtsProviderFailure	Traces	warning	5 minutes	More than 10 speech-to-text or text-to-speech spans fail
ConversationRuntimeCrash	Logs	warning	10 minutes	More than 5 conversation runtime errors of selected crash types from telphi appear

Apply or update the alert rules

1. In SigNoz, create a service account with the Editor or Admin role and store its key as SIGNOZ_API_KEY for the SigNoz host. If you do not manage SigNoz accounts, ask your SigNoz administrator.
2. On the SigNoz host, run ./apply-alert-rules.sh in /opt/services/signoz/, or let the host initialization run it. Without SIGNOZ_API_KEY, the script skips the rule sync.
3. The script creates missing rules and updates existing rules with the same name. It does not delete rules.
4. A rule without its own notification channel gets the channels listed in SIGNOZ_ALERT_CHANNELS (comma-separated). If that is empty, it uses channels named Error or Teams, otherwise the first available channel.

To verify, open the alert list in SigNoz and confirm that all five rules exist with a notification channel.

Dashboards

Delphi provides these dashboards as JSON files in /opt/services/signoz/ on the SigNoz host. Import them in SigNoz through Dashboards → New Dashboard → Import JSON.

• VoiceAI Call Statistics
• VoiceAI Container Metrics
• VoiceAI Error Budget
• VoiceAI Infrastructure Health
• VoiceAI Platform Overview
• VoiceAI SIP Trunk Health
• VoiceAI Transaction Logs

Audit records in SigNoz

From 0.9.18, audit records that the web application writes to the database are also sent to SigNoz after the database change commits. If the change is rolled back, no record is sent. Delivery is best effort: a failed export does not undo the change, and the application does not send the record again.

In SigNoz, audit records carry logType audit and recordKind Audit. A record exported from the database also carries auditLogId, the ID of the database row. Security-relevant actions carry security_event and security_category; see Audit action catalogue.

When an application writes an audit record both directly and to its console, the Web, API and Ops collectors drop the console copy that is marked audit.delivery_role: mirror. The markers are explained in Audit transport markers.

Personal data in telemetry

Collectors mask selected personal data before export unless ENABLE_PII_LOGGING is true; read Protect diagnostic logs before you share any record.

Verify monitoring

1. In SigNoz, filter traces by service telphi for the last 15 minutes. You see telphi.heartbeat spans from every running TelPhi server.
2. Filter logs by each service you operate and confirm that recent records arrive.
3. Confirm that the five alert rules exist and that TelphiNoTraces is not firing.

Collector health and visible records show that the tested path works now; they do not prove that older records were kept.

Related

• Logs and diagnostics: to find the first missing stage when records do not arrive.
• SIP signaling and trunk log reference: to read SIP proxy, capture and trunk health records.
• Verify security event export and retention: to forward security events to an external destination.
• Release compatibility: to check SigNoz component versions.
• Check service health and intervene safely: to check a collector or service container.