Operate the telephony edge (TelPro)

Use this page to keep the TelPro edge healthy. TelPro is the public telephony entry point: it accepts SIP from carriers and browser voice from Janus, and it forwards calls to the Voice hosts over the private network.

Availability: From 0.9.18, unknown-trunk rejections are logged at warn level and SIP trunks have a maintenance mode; neither needs a flag. Two-edge operation is controlled by TELPRO_HA_ENABLED (default false) on each TelPro host and also needs HCLOUD_TOKEN. Without it, each edge runs alone. To plan a pair, read Run two edges.

Edge components

All TelPro containers use host networking and run from /opt/services/telpro.

Container	Component	Role on the edge
voiceai-telpro	Kamailio	SIP proxy: trunk identification, routing, drain gate, TLS for SIP
voiceai-rtpengine	RTPEngine	Media relay for carrier calls
voiceai-webrtc	Janus	Browser voice gateway; sends SIP to Kamailio
voiceai-turn	coturn	TURN relay for browsers behind restrictive networks
voiceai-otel-collector	Telemetry collector	Exports edge telemetry to SigNoz
log-to-span	Log converter	Turns structured Kamailio log events into trace spans

Configuration sources

Source	Contents
Container images	Kamailio routing logic, RTPEngine and Janus builds, tied to ECR_TAG
TelPro configuration bundle	Compose file, collector settings, TURN start script, refresh reconciler
Parameter and secrets stores	Ports, addresses, TURN credentials, edge certificates, Redis connection
Redis	Trunk definitions, trunk order, maintenance settings, instance drain flags and heartbeats
Bootstrap file	PUBLIC_IP, PRIVATE_IP, floating addresses and the host identity

Trunk and routing changes that you save in the administration interface reach Kamailio through Redis and need no restart. Changes to parameters, secrets or the bundle need a refresh of the edge. The Environment variable reference lists every TelPro variable.

Initialization sets the instance identity from TELPRO_INSTANCE_ID, then KAMAILIO_INSTANCE_ID, then the host name, and passes the same value to Kamailio and the reconciler. Keep it unique per edge.

Restart order

The managed edge initializer creates containers and delivers certificate material through container stdin before waiting for dependent services to become healthy. A plain Compose start does not supply that material. The dependency order is:

1. RTPEngine.
2. Kamailio, after RTPEngine has started.
3. coturn.
4. Janus, after Kamailio is healthy and coturn has started.

A restart of Kamailio or RTPEngine interrupts calls on that edge. For a planned change, use the drain path in Drain an edge instance instead of restarting containers by hand.

Check edge health

Container	Health check
voiceai-rtpengine	RTPENGINE_LISTEN_PORT listens and rtpengine-ctl lists sessions
voiceai-telpro	kamcmd core.uptime answers
voiceai-webrtc	The container's main process is Janus
voiceai-turn	TURN_PORT accepts a TCP connection

Run this command on the edge to see the state of each container:

cd /opt/services/telpro
docker compose ps

A healthy container proves only that the process answers locally. To confirm call handling, follow Service health and place a test call.

From 0.9.19, a lost process can leave an answered-call record after its native dialog has disappeared. The reconciler releases that record and its counters only when both signaling and media ownership name this edge, the record predates the current Kamailio boot, and successful native probes show zero Kamailio dialogs and zero RTPEngine sessions. Current-boot calls, peer-owned media, unknown ownership, failed probes and malformed counter data are preserved. A late BYE does not release the same counters again. This repairs bookkeeping after process loss; it does not restore the lost call or establish successful handover. Compare the native workload, call list and conversation result before resuming an operation.

Read edge logs

cd /opt/services/telpro
docker compose logs -f telpro

Replace telpro with rtpengine, webrtc, turn or otel-collector for the other components.

• Kamailio writes structured events when STRUCTURED_LOGGING_ENABLED is true, which initialization sets by default. The log-to-span sidecar turns these events into trace spans in SigNoz.
• Docker keeps SIP_CAPTURE_TELPRO_LOG_MAX_FILES files of SIP_CAPTURE_TELPRO_LOG_MAX_SIZE each for Kamailio; see TelPro variables for the defaults.
• The refresh reconciler logs to the systemd journal. Its lines start with [config-refresh] component=telpro-reconciler and include a phase= field.

journalctl -u telpro-config-refresh.service

For log levels, field meanings and searching, see Logs and Telemetry and monitoring.

Recognize unknown-trunk rejections

When an INVITE arrives from an address that matches no configured trunk, Kamailio answers 403 Forbidden - Unknown Trunk. It logs Rejecting call from unknown trunk with the source address, From header and user agent.

From 0.9.18, the structured trunk_identify_failed event for this case has level warn instead of error. The telemetry collector marks it as an authorization security event. Kamailio treats these as expected rejections of unwanted traffic, so they no longer appear among error-level events.

Identification failures on a configured trunk, such as TLS or authentication problems, stay at error level. Investigate those first. For the full response table, see Routing responses.

Put an inbound trunk into maintenance

From 0.9.18, you can put an inbound trunk into maintenance so that Kamailio rejects its new calls with a response code you choose; see Maintain an inbound trunk to set it up.

Drain an edge instance

Each edge runs telpro-config-refresh.timer, which starts the reconciler every 30 seconds. The reconciler picks up refresh commands that the platform queues for this instance from Service Refresh.

The host requires Python 3 for the update lock. Ops updates and reconciliation share a host process lock, including the initializer called by an update. If another update holds the lock, a direct update exits with status 75; the reconciler logs phase=waiting detail=host_update_in_progress and leaves the queued command for a later tick. Let the active update finish, check its result and service health, then retry the direct command if needed. When a configuration fetch replaces the initializer, execution continues with the fetched version and the original requested options before changing services.

The reconciler stops new calls on the edge, waits for existing calls to end, restarts the edge with fresh configuration and reports the result.

While the drain flag is set, Kamailio answers new INVITEs with 503 Service Unavailable and logs Rejecting new INVITE during TelPro maintenance drain. Existing calls continue. The drain-restart strategy needs at least two ready edges, so that new calls have somewhere to go.

To follow a drain, read the reconciler journal. The phases are acknowledged, draining, waiting, fetching_env, restarting, verifying and completed, or failed with a detail= reason.

Run two edges

Two-edge maintenance separates new-call admission from the addresses and state used by answered calls. Before enabling handover, Ops must configure and verify these prerequisites on the existing machines:

• Admission address: the SIP floating address selects the edge receiving new calls. Moving it does not transfer an established TCP/TLS connection or Kamailio dialog state. The carrier must support reconnecting to the stable dialog route; verify this with the actual carrier before claiming continuity.
• Stable dialog addresses: each edge needs its own reachable SIP/TLS listener and matching Record-Route/contact address. Existing dialogs must continue to reach the edge that owns them after admission moves. Verify network access and certificate identity for both edges.
• Stable media addresses: each call remains anchored to its original RTPEngine and its reachable per-edge RTP address until it ends. Moving a shared media address cannot migrate engine state. Janus and coturn need separately verified stable media/ICE reachability; SIP/RTP tests alone do not validate browser media.
• Host-local identities: each host needs its own TELPRO_INSTANCE_ID and TELPRO_PROVIDER_SERVER_ID, the peer's identities, the configured floating address IDs and TELPRO_INITIAL_ACTIVE_INSTANCE_ID. See TelPro variables.

Ops configures the routing integration on fixed, existing machines. Hetzner floating-address assignment is topology-specific. The configured-http provider is an adaptable integration contract, not evidence of a deployed provider integration. AWS routing and actual carrier continuity are untested. Keep HA disabled until reachability, certificates, peer readiness and the relevant topology/carrier tests pass; a successful isolated simulation does not establish live acceptance.

Event	Safe behavior and boundary
Kamailio maintenance	New-call admission moves only to an accepting ready peer. Existing dialogs and media retain their stable per-edge routes until drained.
RTPEngine maintenance	Stop assigning new sessions to the target; restart only after its actual sessions reach zero.
Full-host maintenance	Drain both dialogs and media before restart; floating ownership does not transfer either state.
Active edge fails	A ready peer may restore admission, but calls whose dialog/media owner failed cannot be assumed recoverable. Measure recovery separately.
Edge certificate reload fails	Fallback requires an accepting ready peer, blocked target admission and a successful actual drain before Kamailio recreation. Failure leaves recovery blocked.

Renewable Redis leases serialize scoped maintenance. Durable operator-maintenance state keeps admission blocked across reconciler timers and restart; loss of lease does not authorize a restart or reopening. Carrier registrations remain off until Redis holds an elected owner.

Certificate/key/CA material is delivered through container stdin and stays in container-local TLS tmpfs and process memory. It is excluded from Docker creation-time environment variables, which Docker persists on the host. The Docker host retains only public fingerprint and recovery state, with no material cache or PEM bind mount. When TLS is required, startup fails if the authoritative cloud source is missing; it does not fall back to host PEM files or disable TLS. After activation, a planned Kamailio restart recreates the container from the exact verified authoritative source. If the container stops unexpectedly, the next 30-second reconciler tick may recreate it under the maintenance lease, only when the source still matches the pin and no TLS recovery is pending. This restores service after an outage; it does not preserve calls owned by a stopped edge. Janus and coturn also lose their TLS tmpfs bootstrap state after an independent container restart. The next leased reconciler tick redelivers material only to consumers that need it, and only when the authoritative source matches the verified public pin. Missing material, a mismatch or failed delivery keeps admission blocked until the source is repaired and an operator explicitly resumes the edge. A newer unactivated or rejected source is never adopted automatically. After rollback, repair the authoritative source and explicitly Resume; admission remains blocked through recreation until services and the served certificate are verified.

Certificate activation validates the staged identity, chain, key and validity, then verifies the served fingerprint through a new TLS connection to TELPRO_TLS_VERIFY_ENDPOINT. By default the connection goes to the edge's own stable address (TELPRO_DIALOG_IP, otherwise PUBLIC_IP) on SIPS_PORT and presents DOMAIN_TELPRO as the server name; a custom endpoint must actually reach the target edge. Failed activation retains or restores previous material where possible. Reopening admission requires verified service and routing recovery. After rollback, repair the authoritative certificate source and use explicit Resume; a matching live fingerprint alone does not establish safe restart material. See the maintenance procedure and Verify platform TLS.

Bind the floating media address

Availability: from 0.9.19, for edges that set TELPRO_MEDIA_IP. It matters most for deployments with browser voice (WebRTC), whose Janus and TURN media use that address.

When TELPRO_HA_ENABLED or TELPRO_FLOATING_IP_BIND_ENABLED is true, edge initialization binds TELPRO_MEDIA_IP on the interface of the IPv4 default route, or on TELPRO_MEDIA_INTERFACE when set; never on loopback. It persists the binding in /etc/netplan/61-telpro-media-floating-ip.yaml (mode 0600) and runs netplan generate, so a network service restart or reboot keeps the address. When both settings are off, initialization removes that file. Janus gathers ICE candidates only on the media address, and TURN binds its relay sockets to it.

Whenever TELPRO_MEDIA_IP is set, initialization checks that the address is bound to a non-loopback interface before it recreates any container. If it is not, it stops with TELPRO_MEDIA_IP … is not bound to a non-loopback interface; refusing to start WebRTC/TURN media followed by TelPro media address check failed; containers were not recreated, and the running containers stay as they were.

The default of TURN_LISTEN_IP is now empty. TURN then listens on each local address, so a reply leaves from the address the client contacted, including a floating address. An explicit comma- or space-separated list is still accepted; a wildcard 0.0.0.0 listener answers from the primary address, which clients behind NAT drop.

Common failures

Symptom	Likely cause	Action
Initialization stops with PUBLIC_IP not found	The bootstrap file lacks the public address	Add PUBLIC_IP to /opt/deployment/.env and rerun initialization.
All new calls get 503 from one edge	The drain flag is still set after a refresh	Read the reconciler journal for a failed phase, then rerun the refresh or recover the edge.
Many 403 Forbidden - Unknown Trunk responses	Scanners, or a carrier sending from an address you have not configured	Compare the source address with the trunk's configured addresses.
Calls connect without audio	Media ports blocked, or a wrong advertised address	Check the RTP port ranges in Network ports and NAT and media addresses.
Browser voice connects signaling but no media	TURN or Janus media ports blocked	Check TURN_PORT, the TURN relay range and the Janus RTP range.
voiceai-webrtc never starts	Kamailio is not healthy, so Janus waits	Fix Kamailio first; check docker compose logs telpro.
Initialization stops with TelPro media address check failed	TELPRO_MEDIA_IP is not bound to a non-loopback interface	Enable the binding or bind the address, check TELPRO_MEDIA_INTERFACE, then rerun initialization. See Bind the floating media address.
Browser voice loses caller audio after a network restart	The media address was bound only until the network service restarted	Rerun initialization so the binding is persisted, then confirm 61-telpro-media-floating-ip.yaml exists.
Reconciler reports peer_not_ready or peer_identity_not_configured	The two-edge peer is missing or not ready	Check the peer's health and its host-local identity values.

For caller-facing SIP responses and call traces, use SIP diagnostics.

Verify the edge

1. Run docker compose ps in /opt/services/telpro and confirm that all containers are running and healthy.
2. Confirm that the reconciler journal shows no failed phase since the last change.
3. Place an inbound test call through each configured trunk and confirm that it reaches the expected flow.
4. If browser voice is enabled, start a browser call and confirm audio in both directions.

Related

• SIP diagnostics: to interpret a SIP response that a caller or carrier reports.
• SIP routing: to change trunks, identification and trunk maintenance.
• Operate the voice services: to check the Voice hosts that receive calls from the edge.
• Deploy, verify and update Delphi: to queue an edge refresh after a configuration change.