Voice service operations

The Voice service runs the actual voice-AI processing: TelSys (Asterisk 22.9.0) provides PBX and media handling, TelPhi (Node 24) orchestrates AI conversations via realtime and modular speech providers, and AudioProc does optional VAD / noise reduction. SIP calls arrive from TelPro and are processed here. Multiple Voice instances run in parallel for horizontal scaling.

Overview

Instance

Instance vCPU and RAM are in Provision instances. The reference size (3 vCPU / 4 GB RAM) targets ~100 concurrent calls per Voice instance — provision additional instances (manually or via the Ops Scaler) for higher load.

Field	Value
Network	Private only — no public IP; traffic arrives via managed load balancer
Scalable	Yes — set initial count in your provisioning tool; the Ops Scaler can adjust count dynamically

Containers

Container	Base	Port(s)	Purpose
voiceai-telsys	Asterisk 22.9.0 (built from source on debian:bookworm-slim)	5080 SIP · 8088 ARI · 10000–11999 RTP	Asterisk PBX / media
voiceai-telphi	Node 24-alpine	12001 media WS	AI conversation engine
voiceai-audioproc	python:3.11-slim (CPU) or nvidia/cuda:12.4.1-runtime-ubuntu22.04 (GPU)	8790 WS	Audio preprocessing (optional)
log-to-span	alpine:3.19 + Go binary	—	Converts TelSys structured logs to OTLP spans
voiceai-otel-collector	otel/opentelemetry-collector-contrib:0.154.0	—	Telemetry collector

AudioProc is the optional audio-preprocessing sidecar in the Voice stack. When the audioPreprocessing feature flag is off, the generated Compose file drops voiceai-audioproc and TelPhi runs without that preprocessing hop. When it is on, TelPhi can send inbound audio frames through AudioProc before they reach the configured speech or realtime AI provider, mainly to improve noisy carrier audio with VAD / noise reduction.

Voice stack baseline

Asterisk upgraded to 22.9.0 with missing slin RTP codec entries fixed. chan_websocket enables WebSocket-oriented voice infrastructure. TelPhi media-session handling improved; WebRTC start-call no longer requires two clicks.

Call flow

1. TelPro routes an INVITE to TelSys (VAIPROXIESPIPED:SIPPORTTELPRO).
2. TelSys prepares the call and bridges to TelPhi via Asterisk ExternalMedia over the TelPhi media WebSocket (TELPHI_MEDIA_PORT, default 12001).
3. TelPhi prepares the configured AI provider and media path. In v0.9.13, inbound readiness policy can delay answer until the provider and media path are ready.
4. AudioProc (when AUDIOPROC_ENABLED=true) preprocesses inbound audio frames.
5. Managed actions such as transfer, hangup, and failover flow through TelPhi and TelSys, then route out through TelPro when they need SIP egress.

Managed telephony actions

v0.9.13 unifies transfer, hangup, failover, and inbound-not-ready behavior across Flow Builder nodes, managed LLM callbacks, and sandbox/runtime commands. Operationally this means:

• TelPhi validates and executes the call action.
• TelSys performs the Asterisk-side bridge, REFER, BYE, reinvite, or disconnect behavior.
• TelPro handles SIP egress, outbound routing, and failover toward the carrier or PBX.
• Transfer restrictions are checked before a managed transfer is attempted.

When debugging a managed action, correlate TelPhi application logs, TelSys PJSIP logs, and TelPro SIP ladder spans for the same call ID.

RFC 3262 (100rel / PRACK)

SIP_100REL_MODE on TelSys controls reliable provisionals and PRACK toward TelPro. Default off. Use peer_supported when a carrier expects PRACK.

Full method behaviour, ladder labels, and Kamailio [SIP_100REL] logging: SIP signaling reference → RFC 3262.

Value	Meaning
off	No PRACK / 100rel. Default.
peer_supported	Reliable 180/183 only when the peer advertises Supported: 100rel.
yes	Always use reliable provisionals on this endpoint.
required	Require 100rel on the endpoint (strictest).

Configure on the Voice service env bundle (vars.yaml / SSM).

SIP errors and conversation state

Voice turns accepted SIP calls into Delphi conversations, but it does not own every SIP failure. Use this split when reading errors:

Failure point	Typical user-visible result	Owner to inspect
Carrier/PBX rejects an inbound or outbound leg	SIP ladder shows the upstream response code and reason. A conversation may not exist if the call never reached Voice.	TelPro Kamailio / carrier routing.
TelPro cannot dispatch to Voice	INVITE fails before TelSys creates the channel.	TelPro Redis dispatcher state and Voice heartbeats.
TelSys accepts the call but cannot bridge media to TelPhi	Conversation may start, then fail or remain silent.	TelSys PJSIP/ARI logs and TelPhi media-session logs.
TelPhi rejects a managed action	Call can continue, fail the action, or hang up depending on the action policy.	TelPhi action logs and the TelWeb conversation timeline.
Provider or flow runtime fails after answer	SIP may stay connected while the AI path errors, then hang up or fall back according to the flow.	TelPhi trace, provider logs, and flow run.

For any SIP-facing incident, start with the Debug SIP ladder and the SIP signaling reference, then correlate the same trace to TelSys and TelPhi before changing flow or trunk config.

Runbook

Scaling

Set the initial Voice instance count in your provisioning tool. The Ops Scaler can adjust this dynamically based on active call count. Instances are named ${environment}-voice-01, -02, etc.

When the Scaler creates a new Voice instance, it appears in the Kamailio dispatcher set in Redis (voiceai:dispatcher:*) and starts receiving calls automatically.

Deploy and update

cd /opt/services/voice
./init.sh
./update.sh --ecr-tag v0.9.13
./update.sh --restart-only

init.sh copies the log-to-span binary into the right path, fetches config from S3, loads env vars from SSM / SM, logs into ECR, and starts Compose. Standard flags apply.

RTP port allocation

Component	Range
TelSys (carrier-facing)	10000–11999
TelPhi	media WebSocket only (no separate RTP range)
Janus (on TelPro)	10000–12000
RTPEngine (on TelPro)	20000–40000

RTP range

Adjust the carrier-facing range via RTPSTART / RTPEND in SSM if your upstream carrier needs a different window.

Recording

When RECORDING_ENABLED=true, TelPhi writes recordings to the S3 bucket RECORDING_BUCKET using AWS credentials available to the container. Recording is post-processed by Tasker on the Ops service.

v0.9.14-patch3 stores new recordings under a team/app/conversation-oriented prefix:

recordings/{teamId}/{appId}/{ISO-timestamp}_{conversationId}[_{tobiConversationId}]

Conversation download and cleanup paths still check legacy recording-id and call-id prefixes, so older recordings remain accessible. Patch3 Tasker processing also writes per-turn stt-input/ and tts-output/ segment artefacts plus transcript metadata for modular speech pipelines.

Each app also opts in via the Recording toggle in Flow Designer (see App overview). Transcript retention follows that app toggle: when recording is on for a call, full transcript text is persisted with the S3 recording artefact regardless of ENABLE_PII_LOGGING. When recording is off, full transcript text is not retained. PII redaction still applies to Postgres and SigNoz exports — see PII redaction and Conversations → Recording and transcript retention.

Configuration

TelSys (Asterisk)

Name	Source	Scope	Default	Description
VAIPROXIESPIPED	SSM	all	—	Kamailio SIP proxy (TelPro private IP).
SIP_PORT_INTERNAL	SSM	all	—	Internal SIP port (from TelPro).
SIPPORTTELPRO	SSM	all	—	TelPro SIP port TelSys dials out to.
RTPSTART	SSM	all	10000	Carrier-facing RTP start.
RTPEND	SSM	all	11999	Carrier-facing RTP end.

TelPhi (Voice AI)

Highlights — full list in vars.yaml / show-env.sh --service voice

Name	Source	Scope	Default	Description
DATABASE_URL	Secrets Manager	all	—	PostgreSQL connection.
DATABASE_POOL_SIZE	SSM	all	50	Connection pool size.
DATABASE_SSL_MODE	SSM	all	disable	Postgres client TLS: disable | require | verify-ca | verify-full.
DATABASE_SSL_CA_BUNDLE_B64	Secrets Manager	all	—	Base64 PEM CA for TelPhi Postgres TLS (same semantics as API).
DATABASE_SSL_CA_FILE	env	all	/etc/ssl/database/ca.crt	Decoded DB CA path in the TelPhi container.
REDIS_HOST	SSM	all	—	Redis server.
REDIS_PASSWORD	Secrets Manager	all	—	Redis auth.
REDIS_TLS_ENABLED	SSM	all	false	Enable TLS to Redis.
REDIS_TLS_CA_BUNDLE_B64	Secrets Manager	all	—	Base64 PEM CA for Redis TLS; optional for Amazon-signed Redis.
REDIS_TLS_CA_FILE	env	all	/etc/ssl/redis/ca.crt	Decoded Redis CA path in the TelPhi container.
TELPHI_MEDIA_PORT	SSM	all	12001	TCP port for ExternalMedia WS.
TELPHI_MEDIA_PATH	SSM	all	/media	URL path Asterisk dials.
TELPHI_MEDIA_SCHEME	SSM	all	ws	ws or wss.
TELPHI_LOG_LEVEL	env	all	LOG_LEVEL	Pino-aligned TelPhi log level.
RECORDING_ENABLED	SSM	all	false	Enable S3 call recording.
RECORDING_BUCKET	SSM	all	recordings	S3 recordings bucket.
AUDIOPROC_ENABLED	SSM	all	false	Toggle AudioProc preprocessor.
AUDIOPROC_LOG_LEVEL	env	all	LOG_LEVEL	Pino-aligned AudioProc log level. Values map to Python logging levels.
SIP_100REL_MODE	env	all	off	TelSys PJSIP 100rel mode: off (default) | peer_supported | yes | required. Controls PRACK / reliable provisional responses toward TelPro. See Voice operations — RFC 3262 (100rel / PRACK).
VAIASTERISKCONSOLELOGFORMAT	env	all	json	Asterisk console log format. json enables structured TelSys logging and log-to-span parsing.
TELSYS_LOG_LEVEL	env	all	info	Pino-aligned TelSys/Asterisk log level: trace | debug | info | warn | error | fatal.
SIP_CAPTURE_ENABLE_PJSIP_LOGGER	env	all	1	Enables detailed PJSIP message logging for SIP ladder and log-to-span diagnostics. Review log volume before enabling in high-traffic environments.
ENABLE_PII_LOGGING	env	all	false	When not true, TelPhi redacts transcript text and caller numbers before DB persist; voiceai-otel-collector scrubs exported logs/traces. Production: false except short support sessions; staging/dev may use true for debugging.
MSISDN_UNMASKED_DIGITS	env	all	4	Trailing digits kept when masking caller numbers (TelPhi + collector). Set on voiceai-telphi and voiceai-otel-collector.
TTS_MEDIA_CACHE_BASE_URL	SSM	all	—	HTTPS base URL for Media service.
TTS_MEDIA_CACHE_UPLOAD_TOKEN	Secrets Manager	all	—	Bearer for media writes.
TTS_MEDIA_CACHE_CA_BUNDLE	SSM	all	—	CA bundle for private media TLS.
HTTP_PROXY	SSM	all	—	Squid proxy for outbound HTTP (OpenAI, etc.).
HTTPS_PROXY	SSM	all	—	Squid proxy for outbound HTTPS.
VONAGE_SMS_API_KEY	Secrets Manager	all	—	Vonage SMS API key (when SMS feature enabled).

Troubleshooting

Call processing

Symptom	Likely cause	Check
Calls ring but no AI response	TelPhi not connected to Asterisk ARI	TelPhi logs for ARI connection errors.
TelSys won't start (90s timeout)	Asterisk config error	docker compose logs voiceai-telsys.
TelPhi can't reach OpenAI	Proxy misconfigured or Squid down	Verify HTTP_PROXY / HTTPS_PROXY; check Squid.
Call drops after ~30s	Realtime API timeout / WS drop	TelPhi logs for WebSocket close; proxy stability.
No SIP from TelPro	VAIPROXIESPIPED wrong	Verify matches TelPro private IP.
Caller hears no AI audio	TelSys can't reach TelPhi media WS	Tail TelPhi for media_session connect; check chan_websocket errors in TelSys.

Transfer to human

Symptom	Cause	Check
Transfer not initiating	transfer() tool not on agent, AI never calls it	Agent tools in TelWeb; TelPhi logs for tool invocations.
Outbound leg rings, no answer	Target unreachable / no egress trunk	TelPhi logs for SIP response codes; egress trunk linked to the base number.
Mini AI session silent	ExternalMedia bridge / RTP wiring failed	Look for Mini session audio wired.
Transfer blocked	Transfer restriction or egress policy denied it	Check flow/team transfer restrictions and TelPhi action logs.
Carrier rejects early media	100rel / PRACK mode mismatch	Set SIP_100REL_MODE to peer_supported (see Overview → RFC 3262); check TelSys PJSIP logs and the Debug SIP ladder.

See also

• TelPro operations — SIP / WebRTC entry point.
• Database operations — Redis keys used for routing.
• Monitoring in SigNoz — trace IDs, call-ID correlation, logs, and dashboards.