Service logs

Use this page as the per-service index of what each Delphi service logs and where those logs land. It complements Monitoring in SigNoz (telemetry plumbing, JSON schema, per-service *_LOG_LEVEL) and Application and call-flow logging (call-path logs for TelSys / TelPro / RTPEngine).

This session documents TelWeb and TelPhi in full. The other services are listed in the hierarchy below with cross-references and will be expanded in later sessions.

Where logs land

Every service host runs an OpenTelemetry collector. Two transports carry application logs into SigNoz:

Transport	Source	Notes
OTLP (direct)	The service's LoggerProvider (@delphi/logger pino → OTLPLogExporter → ${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/logs)	Carries the full structured record: service.name, service.namespace, deployment.environment, service.instance.id, traceId, component, and audit fields.
stdout filelog	The collector's filelog receiver tails /var/lib/docker/containers/*/*.log and parses pino JSON	Backstop for containerized services: logs reach SigNoz even if the in-app OTLP exporter is off. service.name is recovered from the pino service field. Host-run processes (e.g. pnpm dev) are not scraped — their logs only reach SigNoz via OTLP.

Audit records (recordKind: Audit) are delivered once: the canonical copy via OTLP, and the stdout mirror is dropped by the collector's filter/drop_audit_filelog_mirror to avoid duplication. See Security event export.

Log levels

All structured logs use lowercase Pino-style labels: fatal, error, warn, info, debug, trace. Control verbosity with the global LOG_LEVEL or the service override TELWEB_LOG_LEVEL (resolved by resolveServiceLogLevel('TELWEB')). See Per-service log levels.

Services hierarchy

Service	Role	Log sources	Documented
telweb	Next.js web app; hosts the @delphi/api tRPC router and @delphi/auth in-process	Server pino, audit/auth, route handlers, OTel init, client console	This page
telapi	NestJS API (also runs @delphi/api / @delphi/auth)	Server pino, manual spans, audit/auth	Later session — see Monitoring in SigNoz
telphi	Voice application server (ARI + ExternalMedia)	Call/telephony events, CDR, AI/DB spans, trunk-health alarms, route handlers	This page
telsys	Voice PBX (Asterisk)	Channel/call lifecycle	Application flow logging
telpro	SIP edge (Kamailio / RTPEngine / Janus)	SIP ladder, media QoS, log-to-span sidecar	Application flow logging, SIP signaling reference
tasker	Background jobs	Job lifecycle	Later session
scaler	Autoscaling	Scale events	Later session

---

TelWeb (telweb)

TelWeb is a Next.js 16 standalone app (apps/telweb, port 5050). Its Node runtime hosts the @delphi/api appRouter (mounted at /api/trpc) and @delphi/auth, so auth and audit events fired by tRPC procedures execute inside the TelWeb process and are tagged service.name=telweb.

Identity and transport

• Server logger: apps/telweb/src/utils/logger.ts builds a pino service logger via createServiceLogger('telweb', …) from @delphi/logger, with service.instance.id from HOSTNAME. Child loggers add a component field via createLogger(component).
• OTLP: the instrumentation hook (apps/telweb/src/instrumentation.node.ts) registers a LoggerProvider whose resource carries service.name=telweb, service.namespace=voiceai, deployment.environment (from OTEL_DEPLOYMENT_ENVIRONMENT), and service.instance.id. Logs go to ${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/logs.
• stdout: anything written to stdout (pino, console.* in server code) is scraped by the collector filelog receiver when TelWeb runs in a container.
• Client (browser): client-side console.* is not shipped to SigNoz — it stays in the user's browser devtools.

What TelWeb logs

Auth events (via @delphi/auth → logAuth)

Emitted in-process by @delphi/auth (packages/auth/src, e.g. utils/loginAttempts.ts) using logAuth / logSecurityEvent from @delphi/logger. Actions are the AuthActions set:

Action	When	Typical level
LOGIN	Successful sign-in	info
LOGIN_FAILED	Rejected sign-in (details.reason = LoginFailureReason: bad_password, no_user, inactive, account_not_linked, identity_claim_mismatch, invalid_identity_claims, max_sessions_reached, session_store_unavailable, sso_processing_failed, locked, account_locked)	warn
LOGOUT / SESSION_END	Sign-out / session end	info
PASSWORD_CHANGE / PASSWORD_RESET_REQUESTED / PASSWORD_RESET_SUCCEEDED	Password lifecycle	info
ACCOUNT_LOCKED / ACCOUNT_UNLOCKED	Auto-lock after failed-attempt threshold (VGWK-90) / unlock after reset	warn
PRIVILEGE_ESCALATION_ATTEMPT / ACCESS_DENIED / SYSTEM_MANIPULATION_ATTEMPT	Blocked unauthorized access	warn/error
PLATFORM_ROLE_CHANGE_STARTED / PLATFORM_ROLE_CHANGE_SUCCEEDED / PLATFORM_ROLE_CHANGE_DENIED / PLATFORM_ROLE_SESSION_CLEANUP_FAILED	Platform role change flow	info/warn
SUPER_ADMIN_RECOVERY_ELEVATED / SUPER_ADMIN_RECOVERY_DENIED / PASSWORD_RESET_BY_SETUP_CLI	Break-glass setup/recovery CLI	warn
SECURITY_LOGS_ACCESSED / SECURITY_POLICY_MODIFIED / REJECTED_CONNECTION	Security-relevant access	warn

Record fields: userId, username, entityId, appId, details.old / details.new, details.ip, plus traceId when a span is active. Two failure paths also emit plain console.error from packages/auth/src/index.ts: [Auth] LOGOUT audit log failed: and [Auth] Failed to drop Redis session entry on logout:.

Audit logs (via @delphi/api → logAudit)

tRPC procedures call logAudit(...) (packages/api/src/services/logging/logAudit.ts), which builds the record via AuditLogFactory and writes it through AuditLoggingService with the DatabaseLoggingStrategy — i.e. persisted to the audit store (DB) and mirrored to OTLP/stdout with recordKind: Audit. The stdout mirror is deduped by the collector.

• Actions: CREATE, UPDATE, DELETE, plus the auth actions above (LOGIN, LOGOUT, PASSWORD_CHANGE, …).
• Categories (AuditCategories): ROLE_MANAGEMENT, FLOW, NUMBER, SECURITY, SYSTEM.
• Fields: same AuditLogEntry shape as auth — userId, username, entityId, appId, details.old/new, ip.

TelWeb surfaces these in the UI via api.auditLog.list:

• src/components/Admin/AccessLogs.tsx — platform access/audit log viewer (mirrors AuditActionTypes from @delphi/logger).
• src/components/Apps/ActivityHistory.tsx + src/hooks/auditLog/useAuditLogs.ts — per-app activity feed.

tRPC server errors

The /api/trpc fetch adapter (src/app/api/trpc/[trpc]/route.ts) logs every server-side procedure error:

>>> tRPC Error on '${path}'  <error>

via console.error → stdout → filelog → SigNoz.

Route-handler logs

Route	Log	Level
POST /api/webhooks/stripe	[Webhook] … entries for billing activation, resubscribe activation, custom-deal checkout, and their failure cases (missing Stripe IDs, plan not found, missing metadata)	console.log / console.error / console.warn
POST /api/webrtc/session-token/*	[WebRTC] Team is not entitled to WebRTC: <message> (in session-proxy.ts)	console.warn
GET /api/health, GET /api/config, GET /api/i18n/*	No structured logs (health/config/i18n are side-effect-free)	—

OpenTelemetry bootstrap (stdout)

src/instrumentation.node.ts prints lifecycle lines to stdout (scraped by filelog):

Initializing OpenTelemetry for telweb
Sending traces to: <otlpEndpoint>/v1/traces
Sending logs to: <otlpEndpoint>/v1/logs
OpenTelemetry initialized for telweb (traces: HTTP+Redis auto-instrumentation, logs: OTLP)

When OTel is disabled or no endpoint is configured:

OpenTelemetry not configured, skipping initialization

These are the first signal that TelWeb's OTLP export is (or is not) wired up. See TelWeb traces missing in SigNoz below.

Client-side logs (browser only — not in SigNoz)

Source	What	Where
tRPC loggerLink (src/trpc/react.tsx, src/utils/api.ts)	tRPC request/response logging. In development it logs every operation; in production it logs only responses where result instanceof Error. Sensitive procedures (e.g. user.confirmPlatformRoleChange) are redacted by shouldLogTrpcOperation (src/trpc/logger.ts).	Browser console
WebRTCPhone (src/components/WebRTC/WebRTCPhone.tsx)	logDebug → console.debug('[WebRTCPhone]', …): pending call/text chat received, channel status/connection state, chat/text chat received, startCall/reconnect/upgrade/downgrade failures, auto-enabling text chat	Browser console
Other client components	~110 console.error/warn/log call sites across client code	Browser console

Correlation

When investigating from TelWeb, carry these identifiers into SigNoz:

Identifier	Use
trace_id	Distributed trace across web, API, voice, and edge logs.
sipCallId	SIP Call-ID (shown as "call id" in TelWeb Debug).
Conversation id	TelWeb conversation id from the call detail page.

Start from Conversations → call detail → Debug (Logs / Spans / SIP ladder), then search SigNoz with the same trace_id or Call-ID. See Trace debug search.

TelWeb traces missing in SigNoz

If TelWeb logs appear in SigNoz but traces do not, check the bootstrap lines above in the container stdout. The most common causes:

• OTel init skipped — OTEL_ENABLED is not true, or OTEL_EXPORTER_OTLP_ENDPOINT is unset (the guard in instrumentation.node.ts returns early). The filelog receiver still surfaces the "skipping initialization" line.
• Endpoint unreachable in dev — when running pnpm dev on the host, .env must set OTEL_EXPORTER_OTLP_ENDPOINT (e.g. http://localhost:4318); the compose-injected host.docker.internal:4318 value is not applied to a host-run process.
• Environment label mismatch — TelWeb traces are tagged deployment.environment from OTEL_DEPLOYMENT_ENVIRONMENT. Filtering SigNoz by the wrong environment hides them.

---

TelPhi (telphi)

TelPhi is the voice application server (apps/telphi) — a Node.js ARI + WebSocket-ExternalMedia service that bridges Asterisk (TelSys) media to the LLM/conversation engine. It is server-only (no browser/client), has no auth, no audit, and no tRPC, and its traces are manual spans + Redis only — auto-instrumentation is intentionally disabled for everything except Redis.

Identity and transport

• Server logger: apps/telphi/src/utils/logger.ts builds a StructuredLogger via createLogger('telphi', { defaultPiiLoggingEnabled: config.pii.enabled }) from @delphi/logger, with service.instance.id from TELPHI_INSTANCE_ID || HOSTNAME. It exposes domain loggers: logSystem, logDTMF, logError, logTelephony, logConversation, logModelUsage, logPerformance, logAudio, logVM, logSandbox, logRealtimeUsage, logRealtimeAggregatedUsage, logCDR, plus info/error/warn/debug. Pino level comes from TELPHI_LOG_LEVEL via resolveServiceLogLevel('TELPHI').
• Three sinks per entry (StructuredLogger.log, packages/logger/src/structured-logger.ts): (1) pino stdout — scraped by the collector filelog receiver; (2) emitOTelLog → OTLP when isOtelExportEnabled() (OTEL_ENABLED=true + endpoint set), tagged log_transport: 'otlp_direct'; (3) an optional DB sink (ENABLE_DB_LOGGING !== 'false'). Audit-style entries are stamped log_transport: 'stdout_filelog' + audit.delivery_role: 'mirror' on the pino path.
• OTel SDK: apps/telphi/src/utils/otel-instrumentation.ts — a hand-rolled NodeSDK (not @vercel/otel), started first in main.ts. LoggerProvider + BatchLogRecordProcessor + OTLPLogExporter → ${OTLP}/v1/logs; NodeSDK + OTLPTraceExporter → ${OTLP}/v1/traces. Auto-instrumentation: only ioredis and redis enabled — http, express, grpc, pg, fs, net, dns are all off ("Traces: manual spans + Redis only").
• Resource attrs: service.name=telphi (env OTEL_SERVICE_NAME), service.namespace=voiceai, deployment.environment (legacy wire key, from OTEL_DEPLOYMENT_ENVIRONMENT), service.instance.id = TELPHI_INSTANCE_ID || HOSTNAME.
• docker-compose (.infrastructure/services/voice/docker-compose.yaml, service voiceai-telphi): network_mode: host, OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318, OTEL_SERVICE_NAME=telphi, OTEL_SERVICE_NAMESPACE=voiceai, OTEL_DEPLOYMENT_ENVIRONMENT=${ENVIRONMENT:-staging}, LOG_LEVEL=${TELPHI_LOG_LEVEL}, ENABLE_PII_LOGGING. No TELPHI_INSTANCE_ID is set, so service.instance.id falls back to HOSTNAME.

What TelPhi logs

Call / telephony logs (logTelephony)

TelephonyLogEntry (packages/logger/src/types.ts) carries event + channelType (sip | external_media | bridge) on every entry, plus optional ariChannelId, bridgeId, callerNumber, calledNumber, extension, rtpPort, mediaFormat, message, metadata. Base fields: channelId, callId, sessionId, conversationId, appId, turnId, and the injected trace_id/span_id.

event	When	message
channel_start	ExternalMedia leg / SIP leg (handlers/stasis-start.ts)	—
channel_end	handlers/stasis-end.ts	ExternalMedia channel ended / SIP channel ended (metadata callId, isResumeable)
call_ringing	handlers/inbound-readiness.ts	Sent SIP 180 Ringing via ARI (metadata sipResponse:'180')
call_rejected	inbound not-ready	Inbound readiness failed: <reason> (metadata onNotReady, phase)
call_answered	inbound-not-ready transfer / answer paths	Answered for inbound-not-ready transfer
call_hangup	setup-abort / various hangup paths	Call setup aborted — channel ended during setup / Caller hung up during setup
bridge_create / bridge_destroy	bridge lifecycle	—
media_ready	ari/helpers.ts	ExternalMedia channel added to bridge
dtmf_received	handlers/dtmf.ts	Starting extension input handling (metadata sourceLayer:'ari', digit, durationMs, collectedLength)

logDTMF is PII-aware: it redacts metadata.digit / metadata.digits to * when piiLoggingEnabled is false. Emitted at the aborted_during_collection, timeout, and extension_received events.

CDR (logCDR)

One CdrLogEntry per call at StasisEnd (handlers/stasis-end.ts via buildCDR from utils/cdr-builder.ts): logType: 'cdr', schemaVersion: '1.0'. It carries the correlation IDs (channelId, conversationId, callId, traceId, parentSpanId, appId, teamId, flowDefinitionId, flowVersion, sessionId), timing, participants, status (completed | error | transferred | abandoned), failure, botOutcome, provider (STT/LLM/TTS provider + model, chains), and latency (rtpMediaSeconds, llm tokens, ttsProviders). On build failure: logError with component:'cdr', errorType:'CdrBuildError'.

AI / DB tracing (apps/telphi/src/utils/tracing.ts)

Helper	Span kind	Name	Attributes
withAISpan(attrs, fn, parentCtx?)	CLIENT	${provider}.${operation} (e.g. openai.llm.generate)	ai.provider, ai.operation, ai.model, ai.input_size, ai.output_size, ai.duration_ms, peer.service
withDBSpan({ system, operation, statement? }, fn)	—	—	db.system, db.operation, db.statement, peer.service
recordTokenUsage(span, usage)	—	—	ai.tokens.prompt / completion / total
recordAudioMetrics(span, metrics)	—	—	ai.audio.duration_ms / sample_rate / channels / bytes

withAISpan is used throughout providers/modular-audio-pipeline.ts (transcript and AI-response-audio spans) and sandbox/sandbox-vm.ts.

SIP trunk health alarms (recordTrunkHealthAlarm)

Emits a span named alarm.sip_trunk_health with alarm.* semantic attributes (alarm.name, alarm.severity, alarm.state, alarm.previous_state, alarm.resource.type, alarm.resource.id, alarm.server.id, alarm.reason, alarm.rtt_ms) and an alarm.raised / alarm.recovered event.

Call span lifecycle (startCallSpan)

startCallSpan(name, attrs, externalTraceId?, parentSpanId?) creates a SpanKind.SERVER span. It is wired at handlers/stasis-start.ts with span name call.incoming and attributes call.channel_id, call.channel_name, call.caller_number, call.caller_name; the span is stored in activeCallSpans and ended at stasis-end.ts with call.end_reason, call.end_reason_category, call.end_reason_details.

Also: runInPersistedTraceContext(name, traceId, parentSpanId, fn) (cross-service TelAPI calls), startSpan / withSpan (generic child spans). TelPhi has no startRootSpan / startClientSpan.

Route-handler logs (Express)

A minimal Express server (transport/ws-server.ts) is mounted at /api and /api/resources. Logs use logSystem with component / action / status:

Route	Log
GET /api/health/live	no log unless dead
GET /api/health/ready	not-ready: component:'health_check', action:'readiness_check', status:'warn', Instance not ready (metadata healthStatus, utilization, dbHealthy, asteriskHealthy)
GET /api/health	unhealthy: Health check status: <status>
GET /api/resources/status	component:'resource_status_api', action:'get_status', Resource status requested
POST /api/resources/cleanup	action:'manual_cleanup' (warn → success); failure action:'manual_cleanup_failed'
POST /api/resources/cleanup-all	action:'manual_cleanup_all' (warn → success); failure action:'manual_cleanup_all_failed'
GET /api/resources/channels	no log (JSON only)

Background services log via logSystem / logError with component: matching the service: ari_server, ari_client, media_ws_server, heartbeat, notification, sip_trunk_health, usage_monitor, resource_monitor, sipmap_reconcile, resumable_session, channel_service, process (uncaughtException / unhandledRejection), startup, trace_correlation, call_id_resolver, trace_resolver, sip_header_resolver, external_media, cleanup, dtmf_handler, inbound_readiness, transfer_handler, flow_engine, cdr, modular_audio_pipeline, bridge_manager, call_setup. The media WS server additionally uses component: 'media_ws_server' for listening, stopped, reject_upgrade, early_media_start, session_established, and errors (UnexpectedBinaryFrame, InvalidControlJson, MissingMediaStart, WsError, UnexpectedChannel).

Not applicable to TelPhi: auth events (logAuth is re-exported but never called — there is no user login surface), audit logs (no logAudit / AuditActionTypes / AuditCategories), tRPC (not used; the API surface is Express plus @delphi/notification / @delphi/db clients), Stripe webhooks (none — the UsageMonitorService deliberately skips Stripe-linked subscriptions), WebRTC (media is Asterisk chan_websocket ExternalMedia over WS, not WebRTC).

OpenTelemetry bootstrap (stdout)

otel-instrumentation.ts prints lifecycle lines to stdout (raw console.*, not pino — scraped by filelog):

Initializing OpenTelemetry for telphi
OTLP Endpoint: <otlpEndpoint>
OpenTelemetry initialized (minimal instrumentation)
- Service: telphi
- OTLP Endpoint: <otlpEndpoint>
- Traces: manual spans + Redis only
- Logs: collected via Docker container stdout

When disabled or unconfigured: OpenTelemetry not configured (OTEL_EXPORTER_OTLP_ENDPOINT not set), skipping initialization. startCallSpan also emits unconditional [OTEL DEBUG] Created span with remote parent: / [OTEL DEBUG] Created root span: lines (not gated by a debug flag). TelPhi does not configure diag / DiagLogLevel (default silent).

Correlation

Carry these identifiers into SigNoz when investigating a TelPhi call:

Identifier	Source	Use
trace_id / span_id	injected by StructuredLogger.extractTraceContext from the active OTel span onto every pino + OTLP entry	join logs to the call trace
X_TRACE_ID	ARI channel var set by TelSys from Kamailio's X-Trace-ID SIP header (ari/helpers.ts:getTraceIdFromChannel)	upstream trace from the edge; if absent, derived as MD5(SIP Call-ID)
X_TELSYS_SPAN_ID	ARI channel var (getSpanIdFromChannel)	TelSys parent span — TelPhi's call.incoming span is its child
callId / SIP Call-ID	getCallIdFromChannel (X_CALL_ID → PJSIP_HEADER(Call-ID) → X-Call-ID → CHANNEL(pjsip,call-id))	call identity across services
conversationId / channelId / sessionId	base log fields	call / conversation correlation

startCallSpan parent-child logic (stasis-start.ts): externalTraceId = a resumed trace (resumable sessions carry their own) or the SIP-header trace; parentSpanId = X_TELSYS_SPAN_ID. When a resumed trace overrides, callParentSpanId = null — TelPhi becomes the new trace root. If a traceId arrives without a valid parent span, the span joins as a sibling with trace.parent_missing: true and a freshly-generated spanId. A successful link logs component:'trace_correlation', action:'linked'.

• Outbound to TelAPI: services/channel-service.ts reads activeCallSpans.get(channelId)?.spanContext() and wraps the TelAPI call in runInPersistedTraceContext(name, traceId, parentSpanId, fn) so TelAPI joins the same trace.
• Sandbox trace bridge: SandboxLogEntry.operation includes set_trace_id, set_span_context, set_flow_context_bridge, propagating trace context into bot HTTP fetches from the flow-engine sandbox.
• PII redaction: redactForLogs (packages/logger/src/log-redaction.ts) scrubs caller numbers / MSISDNs when ENABLE_PII_LOGGING !== 'true'; redacted entries get pii_redacted: true + redacted_fields_count. logDTMF separately redacts metadata.digit / digits.