Skip to main content
Version: 0.9.17 — archived documentation

Load testing

Use this page to investigate load-test behavior. Exact customer campaign inputs, measured concurrency, headroom and incident history are preserved in the assigned integration reference; they are not universal capacity guarantees.

What v0.9.16 / patch1 / v0.9.17 changed​

  • Hangup header sanitization so invalid SIP header names cannot drop BYE materialization
  • Scaler API metrics and drain/cleanup so replaced Voice instances leave Redis + Postgres
  • TelPhi max-call-age reconcile, full /api/resources/cleanup, idempotent Kamailio DECR on CANCEL/dialog:end (stuck-channel / admission-counter drift after abrupt remote hangup)
  • Azure STT/TTS WebSocket keepalive + mid-call reconnect (SCRUM-889)

Encoding matrix​

Choose and record the input/output codec, audioproc setting and bot path for each measured cell. The historical customer campaign matrix is preserved in the assigned integration reference. Do not infer certification for unmeasured encodings or providers.

STT/TTS WebSocket resilience under load (SCRUM-889)​

At 350 concurrent calls a small number of Azure-protocol STT WebSockets closed with code 1006 (abnormal closure — the TCP connection died without a close frame). Failures correlated with Voice instances running above ~90 % CPU; an instance at 76 % had none. 1006 under saturation means "no bytes flowed long enough that a peer, proxy, or the OS reset the socket", not a protocol error on either side.

The staging C200 campaign used the same Azure Speech WebSocket protocol against Pythia and completed 600/600 turns with no deaf-call signature. The client now reconnects instead of staying deaf (PR #446, on the current staging build).

What the log events mean​

All events are emitted by azure_stt / azure_tts (transport component in TelPhi):

ActionMeaning
ws_reconnectingAbnormal close detected; the configured reconnect policy fired. Metadata shows attempt, maxAttempts, delayMs, and the triggering close code. Audio is buffered and replayed on recovery.
ws_reconnectedMid-call recovery succeeded; the session resumed on a fresh socket.
ws_reconnect_exhaustedAll reconnect attempts failed. Metadata includes initialCloseCode (the close code that started the loop) and lastError. The provider now surfaces an error → STT fallback chain → failover.
ws_closedThe socket closed permanently without a reconnect. Metadata is unambiguous: expected: true + closeOrigin: client_close means Delphi itself tore the socket down (reset/stop) and any code — including 1006 — is benign; expected: false + reconnectSkipped: reconnect_disabled means an admin set reconnectAttempts: 0.
ws_keepalive_timeoutThe ping/pong keepalive declared the socket dead (no pong within the window) and terminated it so the failure flows into the reconnect path instead of lingering half-open.

If a load run shows ws_closed with expected: false, the resolved reconnect config in the same log line tells you exactly which resilience setting suppressed recovery.

Config knobs (per STT/TTS provider entry, resilience object)​

KeyDefaultEffect
reconnectAttempts5Mid-call WebSocket reconnects after an unexpected close. Independent of retryCount (initial-connect retries). 0 disables mid-call recovery.
keepalivetrueWebSocket ping/pong (20 s ping, 10 s pong window) that detects dead connections early.
retryCount0 STT / 1 TTSInitial connect/request retries only — no longer affects mid-call reconnect.

Recovery ladder when STT is lost mid-call: mid-call reconnect (reconnectAttempts) → sttFallbacks provider chain → one same-provider reopen → managed failover (stt_error). A call can no longer silently go deaf on STT loss.

Capacity note: choose the admission cap from a measured workload with agreed headroom; the original customer-specific measured bands remain in the scoped historical reference.