OTel collector configuration

The reference configurations for Voice, API, Web, TelPro, Ops, Database and Squid define a host OpenTelemetry collector (voiceai-otel-collector, image otel/opentelemetry-collector-contrib). It receives telemetry from the containers on that host, normalizes and filters it, scrubs PII, and exports it over OTLP to the configured OTEL_BACKEND_ENDPOINT. This page documents the file that configures it, every receiver/processor/pipeline it runs, and — in detail — how to change the PII obfuscation rules.

Where the configuration lives

The collector is configured by otel-collector-config.yaml, which ships inside the deployment's configuration bundle next to docker-compose.yaml, init.sh, and vars.yaml:

deployments/<deployment-slug>/<config-ref>/
  common/
  <role>/          docker-compose.yaml · init.sh · vars.yaml · otel-collector-config.yaml

fetch-config on the host syncs that directory into /opt/services/<role>/, so the live file is:

/opt/services/<role>/otel-collector-config.yaml

Edit the file in the bundle (the source of truth) rather than only on the host — a re-provision re-runs fetch-config and overwrites host-local edits. See Bootstrap and init for the delivery mechanism.

Two different changes are made in two different ways:

Change	Where you make it	What makes it live
A PII rule, pipeline, receiver or exporter	otel-collector-config.yaml in the bundle	Sync the file, then recreate the collector container
A value the YAML substitutes (${env:…})	SSM Parameter Store / Secrets Manager	Restart the service so the values are resolved again

Applying a YAML change

fetch-config only syncs files. It runs aws s3 sync from the config bundle into /opt/services/<role>/, validates the tree against the config manifest's checksums, keeps the host-local .env, and prints Configuration synced to …. Nothing is recreated, so the collector keeps running the configuration it read at startup — the edited file is on disk, but not live.

Recreate the container to load it, because the collector reads its configuration once at startup. On the affected host:

cd /opt/services/<role>
./update.sh --restart-only

update.sh --restart-only delegates to that role's initialization with cached images. This is a service-wide operation, not a collector-only restart: it can recreate application containers and perform role-specific setup. Review the exact role script and plan the application impact before running it. Initialization resolves the environment again and recreates containers using the synchronized configuration. Plain ./init.sh is broader and may pull images or run additional setup; --skip-fetch only suppresses the bundle sync, not every other operation. Check the actual image, effective configuration and docker logs voiceai-otel-collector afterward. Configuration or processor errors can prevent startup; collector health alone does not prove export or redaction succeeded.

Where the substituted values come from

vars.yaml is a manifest, not the runtime source of the values. It declares which variables a service needs, which containers they are injected into, and a default. fetch-env.sh resolves the real value at init time, last match winning:

1. local variables in the bootstrap /opt/deployment/.env
2. /${NAMESPACE}/NAME in SSM Parameter Store — shared by every service
3. /${NAMESPACE}/${SERVICE}/… in SSM Parameter Store — per-service, overriding the shared value
4. Secrets Manager, then external ARN secrets
5. the manifest default — used only when none of the above supplied a value

In a deployed environment, a default edited in vars.yaml therefore does not change the value while an SSM parameter of the same name exists: the SSM value wins. To change ENABLE_PII_LOGGING or MSISDN_UNMASKED_DIGITS, update the parameter under /<namespace>/<service>/ and restart the service (update.sh --restart-only) so the resolution runs again and the container is recreated. vars.yaml remains where the fallback defaults (false and 4) and the container list are defined.

See How the environment is resolved and the vars.yaml schema for the full model — the namespace conventions, secret discovery, and composed variables.

The configuration file

An OTel collector config has five top-level blocks; the ones Delphi sets are:

Block	Purpose
receivers	Where telemetry comes from — OTLP, container log files, host/container metrics
processors	Ordered transformations applied to records (filters, normalizers, PII scrubbing, batching)
exporters	Where records go — the SigNoz OTLP endpoint (plus debug on Voice/TelPro)
extensions	health_check on 0.0.0.0:13133
service	Which receivers, processors, and exporters make up each pipeline (logs, traces, metrics)

Receivers

Receiver	Used by	Purpose
otlp	all roles with apps	OTLP over gRPC (0.0.0.0:4317) and HTTP (0.0.0.0:4318) from the applications
filelog	all roles	Container stdout JSON logs (filelog/docker, filelog/postgresql, filelog/squid where relevant)
hostmetrics	all roles	Host CPU, memory, disk, network metrics
docker_stats	all roles	Per-container resource metrics
postgresql, redis	database only	Database and cache metrics (uses the monitoring credentials in vars.yaml)

Processors

Processor order is significant — it is the order listed in the pipeline, and every processor sees what the previous one left behind. The processors in the shipped configs:

Processor	Roles	Purpose
memory_limiter	all	Bounds collector memory; drops data rather than OOM-killing the collector
resourcedetection	all (logs/metrics only)	Adds host attributes to log and metric records. Never on the traces pipeline — varying resource attributes break flame graphs
resource	all	Adds the static service/deployment attributes that identify the host
filter/drop_*	all	Drop rules: collector's own logs, records with no service name, non-canonical services, and role-specific noise (PJSIP console envelopes, non-JSON TelSys lines, mirrored stdout copies, non-structured Kamailio lines)
transform/log_level	several roles	Normalizes the log level field so SigNoz filters match across services
transform/telsys_normalize, transform/kamailio_promote	voice, telpro	Normalize TelSys/Kamailio log shapes and promote fields to attributes
transform/canonical_service	api	Rewrites the service name to its canonical form
transform/pii	all	PII scrubbing for log records on every role, and for spans on api, web and telpro (see below)
transform/pii_traces	voice only	PII scrubbing for spans on voice — kept in a processor of its own rather than in transform/pii (see below)
batch, batch/traces	all	Batches records before export

Chains differ by role and build

Read the file on the host you are changing before editing it. Newer builds add further transforms — severity normalization (transform/normalize_severity, transform/audit) and readable ISO-8601 timestamps (transform/add_iso_timestamp, transform/add_iso_timestamp_traces) — inserted into the same chains.

Pipelines

The service.pipelines block names the chain for each signal. The shipped Voice logs pipeline for this version is:

service:
  pipelines:
    logs:
      receivers: [filelog, otlp]
      processors:
        - memory_limiter
        - resourcedetection
        - resource
        - filter/drop_collector_self_logs
        - filter/drop_no_service_logs
        - filter/drop_pjsip_console_envelope
        - filter/drop_telsys_non_json
        - filter/drop_telsys_pjsip_envelope
        - transform/telsys_normalize
        - transform/log_level
        - transform/normalize_severity
        - filter/drop_telsys_by_level
        - filter/drop_non_canonical_service_logs
        - transform/pii
        - transform/add_iso_timestamp
        - batch
      exporters: [otlp]

transform/pii sits late in the chain on purpose: it runs after the drop filters, so it only pays the cost on records that survive to export. Anything you add to the chain must appear in both the processor block and the pipeline's processors: list — a processor that exists but is not listed in a pipeline never runs.

Environment variables substituted into the collector config

The YAML uses ${env:...} substitution, so the values below are resolved on the host at init time from SSM Parameter Store and Secrets Manager, with the vars.yaml default used only as a fallback — see Where the substituted values come from. Keep PII policy values consistent across their intended consumers, and check role-specific variables separately — see the environment variable index for the canonical definitions.

Name	Source	Scope	Default	Description
ENABLE_PII_LOGGING	env	all	false	Master switch for PII redaction. Every PII statement in the collector is guarded on this value, so redaction applies unless it is exactly true. Production: keep false except for short support sessions.
MSISDN_UNMASKED_DIGITS	env	all	4	Trailing phone-number digits kept when masking MSISDNs, substituted into the OTTL quantifier that captures the digits to preserve (for example +447911123456 → ***3456).
LOG_LEVEL	env	voice	—	Used by Voice filter/drop_telsys_by_level for selected TelSys operational records. This is not a threshold for every collector pipeline; SIP capture records are excluded by that filter.

PII obfuscation in the collector

This collector layer can scrub records before its configured export. It does not protect copies already emitted through another path. Its mounted configuration can be changed without rebuilding the application image. It is documented end-to-end on Log and telemetry obfuscation; this section is the operator's how-to for the YAML itself.

How a rule is written

Every PII rule is one OTTL statement of the form:

- replace_pattern(<field>, "<pattern>", "<replacement>") where <condition>

• <field> is the attribute or record field to rewrite — the record body, or an attribute such as attributes["callerNumber"]. Map-valued attributes can be reached by adding another key: attributes["metadata"]["callerNumber"].
• "<pattern>" is a regular expression matched inside that field.
• "<replacement>" is the text written back. It may contain $1, $2, … back-references to capture groups, which is how the surrounding text is preserved.
• where <condition> gates the statement. The PII statements use the ENABLE_PII_LOGGING condition together with field/type and event checks. For example, a body rule can use:

where IsString(body) and "${env:ENABLE_PII_LOGGING}" != "true"

In a YAML plain scalar containing an OTTL quoted regex, retain the doubled backslashes needed by the OTTL string (for example \\s, not \s). YAML parsing alone does not validate OTTL. See the Collector replacement syntax; validate against the image actually selected for this deployment.

Two more conventions in the shipped block:

• error_mode: ignore logs statement evaluation errors and continues processing; it does not guarantee the affected value was masked. silent is a different mode. Invalid configuration, OTTL syntax or a regular expression can fail collector initialization before any records are processed. Validate the configuration and inspect startup and processing errors separately.
• Statements run top to bottom inside one statements: list, and a later statement can overwrite the work of an earlier one. In the shipped block the body-based masks (replace_pattern(body, …)) sit after the set(body, attributes[…]) statements that populate the body, so ordering must be checked in the exact role and release configuration; a later copy can otherwise overwrite an earlier mask. When you add a rule, put it with the other rules for the same field rather than at the end of the list.

Log records vs spans: which processor to edit

Processor	Statement key	Context	Applies to
transform/pii	log_statements	context: log	Log records, on every role
transform/pii	trace_statements	context: span	Spans, on api, web and telpro
transform/pii_traces	trace_statements	context: span	Spans, on voice — the same statements, in a processor of its own

transform/pii:
  error_mode: ignore
  log_statements:
    - context: log
      statements:
        - replace_pattern(...) where ...

A rule in the wrong processor never runs

log_statements and trace_statements apply to different signals. A valid statement runs only when the matching signal passes through a pipeline that includes its processor. Adding a log rule to a processor used only by a traces pipeline does not add log masking. Invalid statements or configuration can also prevent startup: validate the effective configuration instead of relying on silent behavior.

Which keys exist on which role in current builds:

Role	Log masking	Span masking
voice	transform/pii	transform/pii_traces
api, web, telpro	transform/pii	transform/pii (trace_statements)
ops, database, squid	transform/pii	none — these roles export no traces

All four span-masking roles apply the same five statements — transcription.text, textPreview, responsePreview, call.caller_number and call.called_number — under the ENABLE_PII_LOGGING guard; only the processor that holds them differs.

Changing the replacement text

The replacement argument combines literal text and capture-group references. The table shows the replacement seen by the regular-expression engine; when writing a Collector configuration file, escape a capture reference such as $1 as $$1 so configuration environment substitution does not consume it. Keep ${env:NAME} only where environment substitution is intended. The replacement forms in the source templates are:

Rule	Shipped replacement	Example result
DTMF Signal= line in SIP bodies	$1$2***	Signal=5 → Signal=***
Phone-number attributes	***$1	+447911123456 → ***3456
E.164 numbers inside log bodies	$1***$3$4	Call from +447911123456. → Call from ***3456.
Email addresses in bodies	****@$2.$3	alice@example.com → ****@example.com
Card-shaped numbers	****-****-****-****	1234-5678-9012-3456 → ****-****-****-****
Conversation transcript fields	***	attributes["text"] → ***
Redacted conversation body	conversation (redacted)	whole body replaced

To use a different mask, edit the third argument and keep the back-references that preserve the rest of the value. For example, to emit {} markers instead of asterisks for the DTMF signal in captured SIP frames:

# asterisk mask (Collector configuration syntax)
- replace_pattern(attributes["rawMessage"], "(?im)(^|\\r?\\n)(Signal\\s*=\\s*)[0-9A-D#*]+", "$$1$$2***")
  where IsString(attributes["rawMessage"]) and attributes["event"] == "sip_message" and "${env:ENABLE_PII_LOGGING}" != "true"

# custom mask — same pattern, replacement emits {}{}{} after "Signal="
- replace_pattern(attributes["rawMessage"], "(?im)(^|\\r?\\n)(Signal\\s*=\\s*)[0-9A-D#*]+", "$$1$$2{}{}{}")
  where IsString(attributes["rawMessage"]) and attributes["event"] == "sip_message" and "${env:ENABLE_PII_LOGGING}" != "true"

$$1 and $$2 in the configuration become the first and second capture references: they retain the line break and Signal=. The literal becomes {}{}{}. The same edit on the phone-number rules replaces the *** prefix — "***$$1" becomes "{}$$1", "[redacted]$$1", and so on. Keep the escaped back-references ($$1, $$2, …) or you will lose the surrounding text, and apply the same mask to every statement that redacts the same kind of value so operators can recognise it.

Changing how many trailing digits stay unmasked

The trailing-digit rules capture the last N digits and put them back after the mask:

- replace_pattern(attributes["callerNumber"], "^.*([0-9]{${env:MSISDN_UNMASKED_DIGITS}})$", "***$$1")
  where IsString(attributes["callerNumber"]) and "${env:ENABLE_PII_LOGGING}" != "true"

There are two ways to change N:

1. Everywhere (recommended) — set MSISDN_UNMASKED_DIGITS for every service host, in SSM Parameter Store under /<namespace>/<service>/ (see Where the substituted values come from); the vars.yaml default is only the fallback when no SSM value exists. Some application and capture helpers also use this variable, but they have separate validation and matching rules. A shared value does not guarantee identical masking across SigNoz, raw SIP logs and stored conversation data. For example MSISDN_UNMASKED_DIGITS=3 turns +447911123456 into ***456.
2. One rule only — replace the ${env:...} reference with a literal quantifier, for example [0-9]{3} to keep three digits on that field while the rest of the deployment keeps four.

The collector substitutes this value into its regular expression; it does not inherit the shared logger's 0–15 validation and fallback behavior. Use a reviewed numeric value and validate every affected rule against representative synthetic inputs. An invalid expression can prevent startup, and an evaluation error can leave a field unchanged. A large retained suffix can expose the entire matched number, while a nonmatching value can remain untouched. Do not treat 0 as a platform-wide guarantee that every digit is hidden. Recreate the intended consumer and verify both startup and resulting records after a change.

Verifying a change

1. Validate the edited configuration against the selected collector image before activation. Apply it through the reviewed role-specific procedure above, accounting for service-wide impact, and check the collector's configured health endpoint and startup logs. A successful health response is only the first check.
2. Exercise an approved synthetic record or test scenario through the intended pipeline, using no real credentials or private conversation content. Avoid enabling PII merely to compare results.
3. In SigNoz, open the log or span and check the field — a masked caller number should read ***3456, transcript text ***, and Signal= lines Signal=***.
4. Switch ENABLE_PII_LOGGING back to false if you enabled it to compare.

What the collector masks

Field	Result
Signal= DTMF lines in SIP capture records	Signal=*** on sip.raw and rawMessage on Voice and TelPro, plus body on Voice; no DTMF statements on the other roles in this version
callerNumber, calledNumber, phoneNumber, transferTarget	Last MSISDN_UNMASKED_DIGITS digits kept, *** prefixed
E.164 numbers inside log bodies	Same trailing-digit rule
Email local parts in bodies	****@domain
Card-shaped numbers in bodies	****-****-****-****
Conversation turns (logType=conversation, text, textPreview, responsePreview)	***; body becomes conversation (redacted)
Spans (on voice, api, web, telpro): transcription.text, call.caller_number, call.called_number, transcript previews	*** / trailing-digit mask

What is not masked

• The body of a raw SIP frame. When ENABLE_PII_LOGGING is false, phone-like SIP/TEL URI users and phone-number display names in the emitted assembled frame are masked by the log-to-span capture parser, which runs ahead of the collector, to *** plus the last MSISDN_UNMASKED_DIGITS digits; the headers listed in SIP_CAPTURE_REDACT_HEADERS become [REDACTED] there too. Other frame content is retained; this is not a guarantee that every SIP field, derived attribute or original container log is anonymized, because scrubbing the frame whole would break call-routing analysis. The collector adds the role-specific Signal= rules listed above. See SIP signaling reference.
• SIP number attributes on spans. Kamailio's structured logs carry fromUri, toUri and destinationUri straight from $fu, $ru and $du, and the TelSys dialplan carries callerNum and calledNum from ${CALLERID(num)} and ${EXTEN}. The log-to-span attribute mapping copies those onto spans as sip.from_uri, sip.to_uri, sip.caller_num, sip.called_num and kamailio.called_number, and no collector has a statement for those key names — each collector masks callerNumber, calledNumber and call.*_number instead. If you need them masked, add statements to the trace side of the collector that emits them (transform/pii on the telpro host for Kamailio, transform/pii_traces on the voice host for TelSys) and keep the ENABLE_PII_LOGGING guard.
• Everything a downstream store keeps. The collector scrubs telemetry on its way to SigNoz; it does not touch data written to PostgreSQL or S3. See Log and telemetry obfuscation for the other layers.

Upstream references

• OpenTelemetry Collector — what the collector is and how it runs
• Collector configuration — receivers, processors, exporters, pipelines, ${env:} substitution
• Transforming telemetry — the transform processor
• transform processor reference — statement syntax and error handling
• OTTL functions — replace_pattern, set, delete_key, IsString, IsMatch, …
• filelog receiver — container stdout collection
• SigNoz: PII scrubbing — additional SigNoz ingestion-side processing; not retroactive query-time protection
• Troubleshooting the collector — when telemetry stops arriving

See also

• Log and telemetry obfuscation — every redaction layer and what each one masks
• Monitoring in SigNoz — dashboards, queries, alerts, and the PII operator guidance
• Service logs — what each service logs and how the pipelines deliver it
• Bootstrap and init — how the configuration bundle reaches a host
• Environment variable index — canonical definitions and defaults