Load testing
Use this page when running or reviewing Vodafone / SIPp load tests against a Delphi staging (C2) deployment. Capacity planning for a single Voice instance is also noted in Voice operations.
The certified numbers below come from the delphi-load-testing capacity campaign (2026-08-06) against a single staging-voice-01 node: Hydra + SIPp deterministic-pcma, Pythia STT/TTS (Azure Speech WebSocket protocol, byte-clock finalization), and tobi-bot-mock. Success is exact SIPp + RTPEngine + STT + TOBi + TTS accounting, not “most calls OK”. Full write-up: scripts/capacity-campaign/CAPACITY_FINDINGS_staging_voice.md in that repo.
Certified single-voice capacity
| Role | Old instance size | After 2× voice resources |
|---|---|---|
| Certified (exact 3-turn accounting) | 100 concurrent (C100@60s: 300/300 turns) | 200 concurrent (C200×2: 600/600 turns, SIPp 200/200) |
| Stretch / intermittent | 110–120 with a 65 s hold (C110@65s can pass; C120 not repeatable) | 210 (one exact pass, then flake) |
| Hard fail | 140 — every call stuck at 2/3 turns | 220–240 — signaling may still pass while turns drop |
| Ops with ~25% headroom | ~75–80 | ~150 |
Customer ~90 OK / ~110 fail on real Azure STT + TOBi is a different cell. The campaign isolates platform capacity from provider variance. Do not treat those VF numbers as the mock-path cliff.
TelAPI is no longer the limiter: callback p95 is ~10–13 ms after the webhook hot-path fix (was ~10 s). Residual over-capacity failures are last-turn cut (packets_out=214 ≈ 2 of 3 TTS replies), not SIP setup.
Historical C2 errors above 100 concurrent (SCRUM-781)
Raunak’s original 110/128 C2 run reported three named errors. The staging campaign did not reproduce them: SIPp stayed 100% through C110 and C200; saturation failed as turn-loss, not dispatcher exhaustion or BYE/pacing exceptions.
If a C2 / real-Azure run still shows these, treat them as regressions — not as the expected 110+ shape:
| Symptom | Typical log | Component |
|---|---|---|
| 180 / “All servers failed for call” | Kamailio dispatcher exhausted | telpro |
| BYE / hangup “channel not found” | ARI channel already gone | telphi |
PacingLoopError | Media pacing under load | telphi |
Also ignore or file separately the always-on invalid character / HandleEntryErrorWithWrite lines that appear even when every call succeeds.
What v0.9.16 / patch1 / v0.9.17 changed
- TelAPI webhook pipelining and late-ack handling so TOBi callbacks do not pile up under concurrency
- Hangup header sanitization so invalid SIP header names cannot drop BYE materialization
- Scaler API metrics and drain/cleanup so replaced Voice instances leave Redis + Postgres
- TelPhi max-call-age reconcile, full
/api/resources/cleanup, idempotent Kamailio DECR on CANCEL/dialog:end(stuck-channel / admission-counter drift after abrupt remote hangup) - Azure STT/TTS WebSocket keepalive + mid-call reconnect (SCRUM-889)
Encoding matrix (SCRUM-715 / 573)
The campaign certified PCMA / 8 kHz / 20 ms with audioproc on the live TelPhi path and TOBi mock. Other encodings were not a separate cell. A useful follow-up matrix if VF still wants codec-specific numbers:
| Cell | Audio in | Audio out | Audioproc | Bot path |
|---|---|---|---|---|
| 1 | G.711 | G.711 | off | TOBi mock or AC mock |
| 2 | G.711 | G.711 | on | same |
| 3 | G.729 or market codec | G.711 | off | same |
| 4 | G.711 | market TTS codec | on | full TelAPI + Redis TOBi stream |
Do not treat mock-STT silence-as-speech packet growth (SCRUM-856) as media-path loss until the same shape appears with real STT or the deterministic Pythia mock.
STT/TTS WebSocket resilience under load (SCRUM-889)
At 350 concurrent calls a small number of Azure-protocol STT WebSockets closed with code 1006 (abnormal closure — the TCP connection died without a close frame). Failures correlated with Voice instances running above ~90 % CPU; an instance at 76 % had none. 1006 under saturation means "no bytes flowed long enough that a peer, proxy, or the OS reset the socket", not a protocol error on either side.
The staging C200 campaign used the same Azure Speech WebSocket protocol against Pythia and completed 600/600 turns with no deaf-call signature. The client now reconnects instead of staying deaf (PR #446, on the current staging build).
What the log events mean
All events are emitted by azure_stt / azure_tts (transport component in TelPhi):
| Action | Meaning |
|---|---|
ws_reconnecting | Abnormal close detected; the configured reconnect policy fired. Metadata shows attempt, maxAttempts, delayMs, and the triggering close code. Audio is buffered and replayed on recovery. |
ws_reconnected | Mid-call recovery succeeded; the session resumed on a fresh socket. |
ws_reconnect_exhausted | All reconnect attempts failed. Metadata includes initialCloseCode (the close code that started the loop) and lastError. The provider now surfaces an error → STT fallback chain → failover. |
ws_closed | The socket closed permanently without a reconnect. Metadata is unambiguous: expected: true + closeOrigin: client_close means Delphi itself tore the socket down (reset/stop) and any code — including 1006 — is benign; expected: false + reconnectSkipped: reconnect_disabled means an admin set reconnectAttempts: 0. |
ws_keepalive_timeout | The ping/pong keepalive declared the socket dead (no pong within the window) and terminated it so the failure flows into the reconnect path instead of lingering half-open. |
If a load run shows ws_closed with expected: false, the resolved reconnect config in the same log line tells you exactly which resilience setting suppressed recovery.
Config knobs (per STT/TTS provider entry, resilience object)
| Key | Default | Effect |
|---|---|---|
reconnectAttempts | 5 | Mid-call WebSocket reconnects after an unexpected close. Independent of retryCount (initial-connect retries). 0 disables mid-call recovery. |
keepalive | true | WebSocket ping/pong (20 s ping, 10 s pong window) that detects dead connections early. |
retryCount | 0 STT / 1 TTS | Initial connect/request retries only — no longer affects mid-call reconnect. |
Recovery ladder when STT is lost mid-call: mid-call reconnect (reconnectAttempts) → sttFallbacks provider chain → one same-provider reopen → managed failover (stt_error). A call can no longer silently go deaf on STT loss.
Capacity note: the 1006 closures clustered on instances above ~90 % CPU. Keep the per-instance admission cap (ServerGroup.maxConcurrentCalls) in the ops-headroom band above (~75–80 old size, ~150 after 2×) so sustained CPU stays below that band.
Related pages
- Voice operations — per-instance capacity
- Ops / Scaler — scale-out and stale-instance cleanup
- Application flow logging — which events to query
- SIP signaling reference — 180 / 487 / BYE