TelPro service operations
The TelPro service is the public-facing telephony gateway. It always runs Kamailio (SIP proxy) and RTPEngine (media proxy). When WebRTC is enabled, it also runs Janus (WebRTC gateway) and coturn (TURN server). Inbound SIP calls and WebRTC sessions arrive here and route to Voice instances for AI processing. Single instance, static IP.
- Overview
- Runbook
- Configuration
- Troubleshooting
Instance
Instance vCPU and RAM are in Provision instances. Delphi has validated the reference size (3 vCPU / 4 GB RAM) at ~1000 concurrent calls on a single TelPro instance.
| Field | Value |
|---|---|
| Network | Public static IP + private network |
| Scalable | No — single instance |
Containers
Rows marked webrtc are present only when the webrtc flag is enabled. SIP-only deployments can leave the Janus, TURN, and TURN-relay firewall ports closed.
| Container | Base | Port(s) | Purpose |
|---|---|---|---|
voiceai-telpro | Kamailio 6.1.1-bookworm (ghcr.io/kamailio/kamailio) | 5060 UDP/TCP · 5061 TLS | SIP signaling |
voiceai-rtpengine | RTPEngine 26.0 (dfx.at channel) | 22222/UDP ng · 20000–40000/UDP RTP | Media proxy |
voiceai-webrtc | Janus v1.3.3 (built from source) | JANUS_WS_PORT (default 443) WSS · 10000–12000/UDP RTP | WebRTC gateway (webrtc) |
voiceai-turn | coturn/coturn:4.6.2 | 3478 / 5349 · 49152–51152/UDP relay | TURN server (webrtc) |
log-to-span | alpine:3.19 + Go binary | — | Structured-log → OTLP spans for Kamailio / RTPEngine / Janus |
voiceai-otel-collector | otel/opentelemetry-collector-contrib:0.154.0 | — | Telemetry collector |
The Kamailio, Janus, and RTPEngine versions are pinned in the image source under .docker/voiceai-{telpro,webrtc,rtpengine}/Dockerfile; coturn and the OTel collector are pinned in .infrastructure/services/telpro/docker-compose.yml. See Version sources.
WebRTC image notes
voiceai-webrtc is the Janus Gateway container that ships with the TelPro service when the webrtc flag is on:
- Image — pulled from ECR as
${ECR_REGISTRY}/voiceai-webrtc:${ECR_TAG}like the rest of the Delphi images. - Runtime — Janus v1.3.3 built from source with the SIP plugin enabled.
- TLS material — mounts
/opt/services/telpro/tls/server.{crt,key}, shared with Kamailio. - Signaling path — exposes WSS on
JANUS_WS_PORTand connects outbound to Kamailio through the Janus SIP plugin.
Firewall
Base SIP deployments need TCP 22/80/5060/5061, UDP 5060, and UDP 20000–40000 for RTPEngine media. When webrtc is enabled, also open TCP 443 for Janus WSS, UDP 10000–12000 for Janus RTP, TCP/UDP 3478/5349 for TURN, and UDP 49152–51152 for TURN relay. These WebRTC-only TelPro rules are this release's only feature-flagged public port requirements.
SIP signaling
TelPro is the SIP edge — Kamailio terminates the carrier/WebRTC hop, anchors RTP through RTPEngine, and dispatches to Voice. Voice owns Asterisk channel state and TelPhi conversation logic; TelPro owns reachability, egress pools, and carrier-facing headers.
For which SIP methods and response codes are logged, capture points, PRACK/CANCEL/481 behaviour, and SigNoz / Debug ladder queries, use the dedicated SIP signaling reference.
Quick routing scenarios:
| Scenario | TelPro role | First debug step |
|---|---|---|
| Inbound to a base number | Match trunk, pick Voice from Redis, relay INVITE | Debug ladder + SMEMBERS voiceai:telsys:server_ips |
| No Voice target | Reject INVITE before TelSys | Kamailio dispatcher / Voice heartbeats |
| Outbound transfer | Egress pool + header allowlists | Ladder telpro_egress_to_outside + TelPhi action logs |
| Carrier error | Relay upstream code when possible | Ladder on telpro_ingress_from_outside |
TLS certificates
Three sources in priority order:
- Certbot (Let's Encrypt) — when
DOMAIN_TELPROis set andCERTBOT_ENABLED=true. Requires port 80 open for HTTP-01. - SSM / Secrets Manager — base64-encoded
TLS_SERVER_CRT,TLS_SERVER_KEY,TLS_CA_CRT. Generated viacat server.crt | base64 | tr -d '\n'. - Manual files — drop into
/opt/services/telpro/tls/; ensurechmod 600 server.key.
Self-signed certificates are generated automatically when TLS is enabled but no material is provided. Do not rely on self-signed in production.
SIP TLS certificate renewal
For SIP over TLS (:5061), Kamailio reads certificate files from the TelPro host at container start. If TelPro uses Certbot with DOMAIN_TELPRO and CERTBOT_ENABLED=true, renewal can update the files automatically, but the running container still needs a controlled reload/restart to present the renewed chain. If certificate material comes from SSM / Secrets Manager (TLS_SERVER_CRT, TLS_SERVER_KEY, TLS_CA_CRT), update the secret values first and then run:
cd /opt/services/telpro
./update.sh --restart-only
For deployments that cannot interrupt SIP traffic, use an active/passive or load-balancer cutover: keep the old TelPro endpoint serving active calls, move new SIP INVITEs to the renewed endpoint, then retire the old endpoint once calls drain. A single TelPro instance restart can drop in-flight signaling and media, so schedule it as maintenance unless the customer has a tested edge failover design.
After renewal, verify from outside the host that DOMAIN_TELPRO:5061 presents the new certificate chain and then place a test call over TLS.
Log level mapping
init.sh maps Pino-aligned log levels to the numeric values Kamailio, RTPEngine, and Janus expect. Prefer service-specific overrides when you only need to raise one component:
| Pino level | TELPRO_LOG_LEVEL → Kamailio | RTPENGINE_LOG_LEVEL → RTPEngine | WEBRTC_LOG_LEVEL → Janus |
|---|---|---|---|
error | 0 | 3 | 1 |
warn | 1 | 4 | 3 |
info | 2 | 6 | 4 |
debug | 3 | 7 | 5 |
trace | 4 | 7 | 7 |
If a component-specific variable is unset, TelPro falls back to LOG_LEVEL. Change values in SSM, then run ./update.sh --restart-only.
SIP ladder capture remains available when general log levels are reduced. Use SIP_CAPTURE_ENABLE_PJSIP_LOGGER on Voice for TelSys PJSIP captures, and raise TelPro component log levels only for short troubleshooting windows.
Deploy and update
cd /opt/services/telpro
./init.sh
./update.sh --ecr-tag v0.9.13
./update.sh --restart-only
Kamailio Redis keys
Kamailio stores routing state in Redis (on the Database instance):
docker exec -it voiceai-redis redis-cli -a "${REDIS_PASSWORD}"
KEYS *dialog* # SIP dialog state
KEYS *dispatcher* # Voice instance routing
KEYS *location* # registrations
GET voiceai:egress:<num> # egress trunk config for a base number
SMEMBERS voiceai:telsys:server_ips
LRANGE voiceai:egress:registrations 0 -1
v0.9.13 stores richer egress destination pools for transfer and outbound routing. Egress trunk and base-number edits in TelWeb now auto-sync the Redis payloads that Kamailio reads, but operators should still verify voiceai:egress:<num> when troubleshooting stale routing.
Outbound transfer and failover
Managed transfers now rely on TelPro for SIP egress failover and proxy routing. Kamailio selects the configured egress destination, forwards to the carrier/PBX, and records SIP replies for the ladder view. If one destination is unavailable, the egress pool can fail over to another configured member.
The v0.9.13 transfer path supports:
- routable
sip:/sips:Refer-To targets without forcing them back through a number-only trunk lookup; - number targets routed through the base-number egress pool selected by TelPhi / TelSys;
- callback-provided SIP headers and Refer-To query parameters, subject to outbound allowlists;
- BYE-based disconnect-reconnect transfers where the carrier reroutes from BYE headers.
For failed transfers, check:
- TelPhi action logs for validation or transfer-restriction failures.
- TelSys PJSIP logs for REFER, BYE, reinvite, or bridge failures.
- TelPro Kamailio logs and SIP ladder spans for outbound routing and carrier response codes.
Kamailio
| Name | Source | Scope | Default | Description |
|---|---|---|---|---|
PUBLIC_IP | SSM | all | — | Public IP for SIP contact headers. |
PRIVATE_IP | SSM | all | — | Private network IP. |
SIP_PORT | SSM | all | 5060 | SIP non-TLS port. |
SIPS_PORT | SSM | all | 5061 | SIP TLS port. |
TLS_ENABLED | SSM | all | false | Enable TLS on SIPS. |
REDIS_HOST | SSM | all | — | Redis address. |
REDIS_PASSWORD | Secrets Manager | all | — | Redis auth. |
REDIS_TLS_ENABLED | SSM | all | false | Requires a voiceai-telpro image built with hiredis_ssl. |
TELPRO_LOG_LEVEL | SSM | all | LOG_LEVEL | Pino-aligned Kamailio log level. Replaces generated DEBUG_LEVEL overrides. |
RELAY_OUTBOUND_URI | env | all | — | Optional outbound relay target used by Kamailio egress routing when a deployment needs a fixed next hop. |
REFER_TARGET_HOST | env | all | — | Kamailio REFER_SBC_HOST for loopback Refer-To detection. Defaults to DOMAIN_TELPRO in entrypoint.sh. Must match TelPhi REFER target configuration so carrier REFERs and legacy x-delphi-egress-route offload URIs rewrite correctly. |
LOG_TO_SPAN_OTLP_ENDPOINT | env | all | localhost:4318 | OTLP HTTP endpoint used by the log-to-span sidecar. Keep this pointed at the local collector. |
LOG_TO_SPAN_EXPORT_LOGS | env | all | false | When true, exports parsed SIP capture records as OTLP logs in addition to spans. Leave false unless the collector/log volume has been sized. |
Janus (WebRTC gateway)
JANUS_WS_PORTSSMproddefault: 443| Name | Source | Scope | Default | Description |
|---|---|---|---|---|
JANUS_WSS_ENABLED | SSM | all | true | Enable TLS on the WebSocket. |
JANUS_ADMIN_PORT | SSM | all | 7088 | Janus admin API port, bound for local/container administration only. |
JANUS_RTP_PORT_MIN | SSM | all | 10000 | Janus RTP start. |
JANUS_RTP_PORT_MAX | SSM | all | 12000 | Janus RTP end. |
KAMAILIO_SIP_HOST | SSM | all | ${PRIVATE_IP} | Kamailio host Janus dials via the SIP plugin. |
STUN_SERVER | SSM | all | stun.l.google.com | STUN host for ICE. |
STUN_PORT | SSM | all | 19302 | STUN port for ICE. |
TURN_SERVER | SSM | all | — | TURN server address (defaults to PUBLIC_IP). |
TURN_USER | SSM | all | telpro | TURN static user. |
TURN_PASSWORD | Secrets Manager | all | — | TURN static password. |
WEBRTC_LOG_LEVEL | SSM | all | LOG_LEVEL | Pino-aligned Janus WebRTC log level. Replaces generated JANUS_*LOG_LEVEL overrides. |
RTPEngine
| Name | Source | Scope | Default | Description |
|---|---|---|---|---|
RTP_PORT_MIN | SSM | all | 20000 | RTP start. |
RTP_PORT_MAX | SSM | all | 40000 | RTP end. |
RTPENGINE_INTERFACES | SSM | all | — | priv/PRIVATE_IP;pub/PUBLIC_IP (derived from above if unset). |
RTPENGINE_LOG_LEVEL | SSM | all | LOG_LEVEL | Pino-aligned RTPEngine log level. Replaces generated RTP_LOG_LEVEL overrides. |
TURN
| Name | Source | Scope | Default | Description |
|---|---|---|---|---|
TURN_REALM | SSM | all | telpro | TURN realm. |
TURN_PORT | SSM | all | 3478 | TURN standard port. |
TURN_TLS_PORT | SSM | all | 5349 | TURN TLS port. |
TURN_MIN_PORT | SSM | all | 49152 | Relay port start. |
TURN_MAX_PORT | SSM | all | 51152 | Relay port end. |
If you operate your own Janus deployment, set JANUS_WS_PORT (default 8188 locally, 443 in production) so TelWeb dials the correct WebSocket port.
SIP calls
| Symptom | Likely cause | Check |
|---|---|---|
| No incoming calls | Kamailio down or firewall blocking SIP | docker compose ps voiceai-telpro; verify :5060/:5061 reachable from carrier. |
| Calls connect, no audio | RTPEngine down or RTP UDP blocked | docker compose ps voiceai-rtpengine; verify 20000–40000/UDP. |
| One-way audio | NAT / IP misconfiguration | Verify PUBLIC_IP, PRIVATE_IP, and RTPENGINE_INTERFACES. |
| Calls not reaching Voice | Kamailio dispatcher empty | SMEMBERS voiceai:telsys:server_ips in Redis; check Voice instance heartbeats. |
WebRTC
| Symptom | Likely cause | Check |
|---|---|---|
| WebRTC won't connect | Janus down or TLS cert invalid | wss://${DOMAIN_TELPRO}:${JANUS_WS_PORT} reachability. |
| ICE fails | TURN not running or ports blocked | docker compose ps voiceai-turn; verify 3478/5349/UDP and 49152–51152/UDP. |
| WebRTC connects, no audio | Janus → Kamailio SIP bridge broken | Janus logs for SIP register failures; verify KAMAILIO_SIP_HOST. |
Outbound (transfer) calls
| Symptom | Likely cause | Check |
|---|---|---|
| Transfer not routing | Egress trunk not linked to base number | GET voiceai:egress:<num> in Redis should return JSON. |
| 403 from Kamailio | TelSys source IP not in allowlist | SMEMBERS voiceai:telsys:server_ips should include Voice instance IP. |
| Custom headers stripped | Header not in outbound allowlist | GET voiceai:config:outbound_header_allowlist. |
| Failover member skipped | Egress pool state is stale or invalid | Re-sync the trunk/base-number config and inspect voiceai:egress:<num>. |
| REFER target rejected | Target URI or outbound route invalid | Check the Refer-To URI, transfer routing pool, Kamailio egress logs, and REFER_TARGET_HOST. |
SIP TLS
| Symptom | Likely cause | Check |
|---|---|---|
| Carrier rejects TLS handshake | Expired cert, missing intermediate, wrong CA, or SAN mismatch | Inspect DOMAIN_TELPRO:5061 from outside and compare with the carrier trust requirements. |
| Renewed cert not visible | Cert files changed but Kamailio container has not restarted | Run ./update.sh --restart-only during the approved maintenance/cutover window. |
| WebRTC WSS cert changed but SIP did not | Janus/WebRTC and Kamailio may consume different listener config | Verify both JANUS_WS_PORT WSS and SIP TLS :5061 separately. |
See also
- Voice operations — SIP downstream consumer.
- SIP signaling reference — methods, response codes, Kamailio/TelSys captures, ladder pipeline.
- Database operations — Kamailio Redis state.