Plan capacity and maintain the fleet
Plan enough ready capacity for your workload, configure the fleet controls, and verify a maintenance operation before admitting normal traffic again. Increasing a limit does not add CPU, provision a working instance, or prove high availability.
Availability: Super Admin and Super Viewer accounts can open Admin Settings → Server Groups (/admin-settings). Changes to server groups and API rate-limit configuration require Super Admin; a visible editor is not proof that your account can save. Autoscaling additionally requires a running Scaler, Redis/database access, a configured provider and usable provisioning scripts. These controls are not enabled by a customer subscription flag.
For a failed instance, start with service health. For host layout and shared dependencies, use service topology. This page covers fleet sizing, admission controls, Voice runtime limits under load and maintenance decisions.
Plan ready capacity
- Record peak concurrent calls, calls started per second, call duration, dialogue turns, provider quotas/latency, codecs, recording and audio processing. For API traffic, include simultaneous connections, request rates and event-loop lag.
- Measure a sustainable per-instance workload with the intended release and actual CPU, memory and network allocation. Choose an operational allowance below the failure point. Completed conversations and acceptable latency matter; successful SIP setup alone is insufficient.
- Divide target peak concurrency by that allowance and round up. Add ready spare capacity for the intended maintenance/failure case. Include provisioning time: a larger Max Instances does not cover traffic that arrives before a replacement becomes ready.
- Validate the other constraints: TelPro signaling/RTP capacity, provider quotas, database/Redis, object storage and network. A spare Voice instance does not make a single shared dependency highly available.
For example, if your own validated allowance is 75 calls per Voice instance, a target of 500 calls needs seven serving instances. Eight provide one spare: seven remaining instances offer an allowance of 525. This is arithmetic, not a newly validated load-test result. Do not apply headroom twice if 75 already includes it. Keep the workload, release and measurement evidence with the deployment runbook.
Recorded single-Voice-instance measurements
These capacity figures are not certified; use them only as a starting point for your own measurement on one Voice instance with simulated speech and bot services.
What was tested. Each test call sent the same 53-second PCMA audio (8 kHz, 20 ms packets) and expected three complete dialogue turns: speech recognition, bot reply and spoken answer. Speech recognition, speech synthesis and the bot were deterministic test services, so provider latency and quotas did not limit the result. The speech-synthesis media cache was off. All runs used SIP over TCP and a fixed three-turn call of about 60 s; on 1 Sept, part of the turn loss was late turns cut off by the 65 s hangup. A run passed only if every call connected and every expected turn completed within the latency budget: speech end to first answer audio p95 ≤ 2 s and p99 ≤ 3 s.
How to read the numbers. Configured calls is the number of calls the load generator placed. Sampled peak is the highest active-call count read from the Voice instance about every 45 seconds. True concurrency was at least the sampled peak and at most the configured calls. In the August runs, calls were spread over the run window, so most runs never held their configured number at once.
| Voice instance | Date | Configured calls | Sampled peak | Result |
|---|---|---|---|---|
| 2 vCPU, 4 GiB | 6 Aug 2026 | 80 | 36–56 | Passed 5 of 8 runs; the first 3 runs failed |
| 2 vCPU, 4 GiB | 6 Aug 2026 | 100 | 39 | Passed its only run; one run is not a capacity claim |
| 2 vCPU, 4 GiB | 6 Aug 2026 | 110 | 61–95 | Passed 1 of 2 runs |
| 2 vCPU, 4 GiB | 6 Aug 2026 | 120 | 48–90 | Passed 4 of 11 runs; not repeatable |
| 2 vCPU, 4 GiB | 6 Aug 2026 | 140 | 99 | Failed: every call lost its third turn |
| 2 vCPU, 4 GiB | 6 Aug 2026 | 160 | 88 | Failed |
| 2 vCPU, 4 GiB | 1 Sept 2026 | 100 | 100 | Failed: 185 of 300 turns; speech end to first answer audio p95 4.7 s |
| 2 vCPU, 4 GiB | 1 Sept 2026 | 140 | 140 | Failed: 223 of 420 turns; 25 calls received no answer audio |
| Doubled resources, 8 GiB | 6 Aug 2026 | 140–180 | 77–132 | Passed 3 of 3 runs |
| Doubled resources, 8 GiB | 6 Aug 2026 | 200 | 114–153 | Passed 3 of 3 runs; p95 418 ms in the last run |
| Doubled resources, 8 GiB | 6 Aug 2026 | 210 | 136–153 | Passed 1 of 2 runs |
| Doubled resources, 8 GiB | 6 Aug 2026 | 220–240 | 182–192 | Failed: 219 of 220 and 235 of 240 calls connected; turns lost |
What these runs show:
- On 2 vCPU and 4 GiB, the only run that held 100 calls at once lost turns. Do not plan 100 calls on this size. Runs at or below a sampled peak of 56 passed most often; passes above that were not repeatable.
- On 8 GiB, passing runs held up to 153 calls at once (sampled peak). Every run configured above 210 calls failed, and 210 passed only once.
- None of these runs used real speech providers or several Voice instances together. Real providers, recording and audio processing can add load or limits.
Before you set Max Concurrent Calls or plan a fleet, repeat the measurement on your release, machine type, codecs, providers and recording settings. Hold the target number of calls at once and confirm it from the sampled peak. Then apply the arithmetic in plan ready capacity.
Voice runtime limits under load
Availability: from 0.9.18. The following fixes and settings apply to the Voice and TelPro roles. They remove failure modes found in sustained load and endurance runs; they do not raise the capacity figures above.
| Symptom before 0.9.18 | Behavior from 0.9.18 |
|---|---|
| New calls declined with SIP 603 after the ARI WebSocket dropped | TelSys waits up to ARIWEBSOCKETWRITETIMEOUT (default 1000 ms; Asterisk default 100 ms) for a slow ARI reader before it closes the socket. TelPhi logs websocket_reconnecting and websocket_reconnected; an error that reconnect attempts are exhausted means the instance no longer receives calls. |
| Calls rejected with SIP 503 because the TelPro heartbeat was stale | TelPhi writes its admission heartbeat to Redis before database work and refreshes it every 10 seconds by default (HEARTBEAT_INTERVAL). TelPro rejects a Voice server whose heartbeat is 60 seconds old or older. |
| First DTMF digit of an extension lost | Digit capture starts before the call is answered and early digits are buffered. Previously the call could end after 30 seconds with no_extension_received. |
| Answered calls left without a bot during call bursts | TelSys allows 1000 concurrent HTTP sessions on its ARI server by default (ARIHTTPSESSIONLIMIT; the Asterisk default is 100). The reference Voice Compose does not override it. |
| High TelPhi CPU from frequent speech-recognition audio pushes | Optional AZURE_STT_AUDIO_FLUSH_MS sets a larger audio batch per push to the speech-recognition service. Larger batches save CPU but add latency. |
| High log volume from speech recognition | TelPhi writes one audio_pushed_speech_summary record per recognition session instead of one record per audio push. |
Set AZURE_STT_AUDIO_FLUSH_MS in the Voice vars.yaml for the voiceai-telphi container, then recreate TelPhi. Change it only after you measure CPU and answer latency on your own workload. If calls still fail with 603 or 503, follow service health and correlate TelSys, TelPhi and TelPro logs with Logs and diagnostics.
Configure a server group
- Open Server Groups and inspect Service, Servers, Instances, Utilization, Refresh, Last Action and Last Error before changing an existing group. Inventory counts and runtime health are different views.
- Choose Create, or the row's edit icon. Set the fields below. For an existing deployment, preserve the established group identity until you have checked base-number assignments and provider instance discovery.
- Save with Create or Update. Reopen the group to verify persisted values, then check the runtime outcome. A database save can succeed while the best-effort Redis capacity sync logs an error.
- For Voice routing, assign the intended group through base-number administration. Verify a permitted test call reaches the intended group; do not assume creating a group moves existing calls or assignments.
| Control | Range and default | Effect |
|---|---|---|
| Group Name | Required and unique; 1–100 characters. | Group identity. Treat a rename as an operational change: it changes the Redis namespace; existing instance registrations and provisioning discovery need verification. |
| Enabled | On or off; default on. | Only enabled groups with autoscaling enabled enter normal autoscaling evaluation. For Voice, the value is also exported to routing state; disabling is not a standalone, verified maintenance drain. |
| Service | Voice or API, selected when creating. | Role of the group. The API rejects role changes afterwards and permits only one API group. |
| Description | Optional. | Operational notes. |
| Load Balancing Method | Round Robin, Least Calls or Random (round_robin, least_calls, random); default Round Robin. | Voice only. |
| Max Concurrent Calls | Integer 1–10,000; default 100. | Voice only, per-server admission setting. This is a configured cap, not measured capacity. On the measured 2 vCPU host this default exceeds sustained capacity; see recorded measurements. |
| Enable Auto-Scaling | On or off; default off. | Reveals the following controls; turning it on authorizes automatic infrastructure changes when the Scaler can execute them. |
| Min Instances / Max Instances | Integers 1–100, minimum no greater than maximum; defaults 1 / 10. | Decision bounds, not a promise that this many healthy instances already exist. |
| Scale Up Threshold / Scale Down Threshold | Values 0–1, down less than up; defaults 0.7 / 0.3. | Scale up uses at least the up threshold; scale down uses below the down threshold. |
| Cooldown (seconds) | Integer 0–3,600; default 300. | Scaling cooldown. Below-minimum recovery can bypass it. |
| Scaling Provider | Required when autoscaling is enabled. Options include Hetzner Cloud, AWS EC2, Local Docker and Custom. | Provider that executes scaling. A selectable provider is not proof that its scripts are installed. |
| Scaling Config (JSON) | JSON; whitespace parses as {}. Invalid JSON blocks submission. | Provider-specific configuration. JSON syntax validation does not validate credentials, machine availability or every provider-specific field. Use the approved configuration for your deployment. |
There is no editable Priority control in this form, and weighted_round_robin is not a load-balancing method. The editor normalizes the legacy stored method weighted to round_robin when loading the group.
Delphi ships executable scaling scripts for Hetzner Cloud and Local Docker. For AWS EC2 and Custom, no scaling scripts are included; you supply them yourself. The executor requires provider-specific scaleUp.sh and scaleDown.sh in its configured infrastructure directory. A missing scalingStatus.sh returns an unknown status rather than the same missing-script failure; verify the provider’s status/discovery behavior separately. Establish the installed provider implementation and credentials before enabling autoscaling; the Local Docker option does not demonstrate cloud HA.
The existing-group editor also shows a read-only instance summary and table: instance count/enabled count, active calls (Voice) or connections (API), Voice RTP ports, configured scaling range, operational status, server ID/address, status, provider/management origin and last heartbeat. These are observations, not per-instance stop/restart controls. Compare the instance identity and fresh heartbeat with the deployment inventory before acting.
Pass provider configuration to scripts
The Scaler converts each top-level field of Scaling Config (JSON) into a SCALING_CONFIG_ environment variable for its provider scripts. It inserts underscores before uppercase letters and uppercases the key: serverType becomes SCALING_CONFIG_SERVER_TYPE, and replicaCount becomes SCALING_CONFIG_REPLICA_COUNT. The provider field selects the script directory and is omitted from this conversion.
Values use JavaScript string conversion: 0 and false are passed as strings, while null and absent values are omitted. Nested objects are not flattened or JSON-encoded; use the field types expected by the installed script. These generated variables have no universal defaults: the provider script determines which fields it accepts and how it handles missing values.
For scale-up, the executor starts with the Scaler process environment and overlays secrets fetched for the group's secretsName. It then supplies the group name, server name, server type and generated configuration variables. SCALING_SERVER_TYPE is a separate executor input from SCALING_CONFIG_SERVER_TYPE; inspect the installed script to establish which it reads. The bundled Hetzner scale-up script reads SCALING_SERVER_TYPE and requires HETZNER_API_TOKEN; HCLOUD_TOKEN serves the separate TelPro floating-address integration. Verify a controlled provisioning attempt and the resulting instance before relying on a new configuration.
Interpret scaling and health
For the evidence behind a decision, follow Scaler authority, decision and provider logs. Separate a leader transition, an evaluation result and provider-script completion before concluding that healthy capacity changed.
The default evaluation interval is 30 seconds (SCALING_EVALUATION_INTERVAL_MS). The leader evaluates enabled, autoscaling-enabled groups. It counts healthy and provisioning capacity, recovers a deficit below minimum, then considers utilization and cooldown. Normal utilization growth adds one instance at a time; below-minimum recovery can request several Voice instances. Provisioning in progress can defer another decision. Lowering Max Instances is not an immediate force-shrink command.
| Service | Signal used for autoscaling |
|---|---|
| Voice | Active calls divided by configured capacity of healthy members. A cap that exceeds actual sustainable capacity makes this ratio misleading. |
| API | Connection utilization and event-loop lag contribute to scaling. JSON defaults include maxConnectionsPerServer: 1000 and eventLoopLagTargetMs: 100. These are targets/capacity assumptions, not CPU or RAM limits. |
The group list is an observation aid, not the exact scaling decision trace. Its API utilization is connection-based; the Scaler also considers event-loop pressure. Its Voice health window and the Scaler's admission-to-metrics window differ. A displayed healthy count therefore does not prove spare capacity or explain every scaling decision. Check Scaler logs, fresh instance heartbeats and the affected service's health checks together.
Voice capacity sync writes the per-member cap and computes a group capacity from Redis membership, using at least one member for the calculation. Membership is not a certified healthy capacity count. API group changes also mirror scaling fields, including Enabled, into an existing API configuration record; review API rate-limit behavior after such changes.
Adjust API rate limits
The API group's API rate limits tuning icon opens the existing API configuration record. The icon requires that record to exist; its absence is not a separate API-server provisioning control. Use the platform's deployment/bootstrap owner to investigate a missing configuration rather than creating duplicate records.
| Control | Behavior |
|---|---|
| Enabled | Whether this record can be selected as the active configuration. Disabling the record does not disable TelAPI; no active record falls back to enabled default limits. |
| Enable Rate Limiting | Defaults on. Turn off on an enabled record to disable this limiter. Other authentication, scope and subscription checks remain independent. |
| API Rate Limit Max / Rate Limit Window (ms) | Defaults 600 requests / 60,000 ms per client IP. Max 1–100,000; window 1,000–3,600,000 ms. |
The two numeric controls appear when rate limiting is enabled. Enter numbers; clearing the form is not a supported “inherit” action. The request limit applies to the selected window even where helper text says “per minute”. /health and /metrics bypass this limiter; the auth-path tier has a fixed 20 requests per 60 seconds preset rather than a field in this dialog. Verify client-IP forwarding for your deployment before interpreting per-IP counts.
Save, reopen the record, then verify the intended request class in an approved test. TelAPI uses Redis revision checks and a 60-second configuration cache; database/read failures can retain cached values or defaults. A save response or disabled record alone does not prove that protection is disabled on every instance. If behavior differs, inspect configuration revision/database errors and the selected enabled record before repeating a change.
Maintain capacity during a refresh
Use the background-job refresh workflow for supported service configuration refreshes. Before starting, record instance identities, images/configuration, healthy spare capacity, active workload, the strategy, timeout and recovery owner. Ensure the provider can provision replacement instances and that the target service is eligible. API and Voice refresh depend on provider scaling; TelPro drain-restart requires at least two ready edges.
This is the API/Voice drain-restart rolling replacement sequence. restart-only bypasses draining; it is not a call-preserving alternative. In the Service Refresh workflow, reload-only is unsupported for API and Voice: Tasker records those targets as skipped and does not queue their Scaler operations. Check each target result rather than interpreting a completed job as an applied reload. A refresh may temporarily need capacity above the steady-state fleet while a replacement becomes ready. With force: false, a timed-out drain does not authorize decommissioning in the rolling lifecycle; with force enabled, the drain helper can bypass the wait immediately despite remaining workload. Do not use force to make an unexplained drain appear successful.
Refresh is not the same operation as normal autoscale-down:
| Operation | Consequence to verify |
|---|---|
| Voice autoscale-down | Removes the selected instance from new-call routing and observes active application/SIP calls. The configured drainTimeoutMs defaults to one hour in this path; expiry can proceed to deletion even with active calls. Choose a timeout compatible with real call duration and monitor the drain. |
| API autoscale-down | Selects eligible Scaler-managed instances; Terraform-managed instances are protected from this automatic selection. Its default drain timeout is 60 seconds. If connections do not drain, it undrains the instance and defers deletion. |
| Refresh Stop/cancel | Requests cancellation at workflow boundaries. The current target may finish; completed changes are not rolled back. Inspect each host/instance result rather than assuming Stop restored the previous fleet. |
After a refresh, check both replacement readiness and the removal/state of the old instance. Verify expected healthy capacity, image/configuration identity, new-call/new-connection acceptance, completed conversations and error/latency behavior. A completed job is not evidence of interruption-free service or a representative HA test. The legacy Service Refresh controls do not provide standalone drain-only. From 0.9.19, use targeted Server Groups maintenance for TelPro drain-only and explicit Resume. Keep the old screen available while parity and live continuity acceptance are pending. An operator-maintenance block persists across timers and restart; Stop does not clear it.
If a replacement never becomes healthy, preserve the previous serving capacity and investigate provisioning/registration errors before retrying. If cancellation, timeout or failure leaves a mixed fleet, collect per-instance results and reconcile actual infrastructure, runtime membership and database inventory. Use deployment recovery and release compatibility for a deliberate rollback; do not force-delete state to clear a dashboard warning.
Remove a group safely
Check routing/base-number references and real instance ownership before removing a group. The row's delete action sends a normal delete request; the API rejects it while database server instances remain. A group with zero recorded instances is not proof that no orphaned infrastructure exists.
The separate API forceDelete option initiates destructive fleet removal. For populated groups it changes runtime routing/status and marks pending deletion for the Scaler; the response is not evidence that all infrastructure has been deleted. It is not exposed by this row's normal Delete action and is not a maintenance shortcut. Require an approved decommissioning plan, verify no required routing remains, and reconcile provider resources afterwards.
For the actual target checkboxes, strategy controls, job progress and cancellation limits, use Service Refresh. Capacity planning does not replace checking each target’s applied result.
Verify the fleet after a change
- Reopen the server group and confirm that the saved values match your change.
- Check that each expected instance shows a fresh heartbeat and passes its health checks.
- Place a permitted test call or API request and confirm that it completes on the intended group.
Related
- Service health: to confirm that a new or refreshed instance is ready before it takes traffic.
- Diagnose background jobs: to read the result of a Service Refresh job for each target.
- Voice services: to restart or drain a Voice host without losing calls.
- Log records: to follow a Scaler decision from leader to provider result.