Plan capacity and maintain the fleet
Plan enough ready capacity for your workload, configure the fleet controls, and verify a maintenance operation before admitting normal traffic again. Increasing a limit does not add CPU, provision a working instance, or prove high availability.
Availability: open Admin Settings → Server Groups (/admin-settings) with platform read access. Changes to server groups and API rate-limit configuration require Superuser authority; a visible editor is not proof that your account can save. Autoscaling additionally requires a running Scaler, Redis/database access, a configured provider and usable provisioning scripts. These controls are not enabled by a customer subscription flag.
For a failed instance, start with service health. For host layout and shared dependencies, use service topology. This page owns fleet sizing, admission controls and maintenance decisions; it does not certify a cloud machine size or a no-interruption upgrade.
Plan ready capacity
- Record peak concurrent calls, calls started per second, call duration, dialogue turns, provider quotas/latency, codecs, recording and audio processing. For API traffic, include simultaneous connections, request rates and event-loop lag.
- Measure a sustainable per-instance workload with the intended release and actual CPU, memory and network allocation. Choose an operational allowance below the failure point. Completed conversations and acceptable latency matter; successful SIP setup alone is insufficient.
- Divide target peak concurrency by that allowance and round up. Add ready spare capacity for the intended maintenance/failure case. Include provisioning time: a larger Max Instances does not cover traffic that arrives before a replacement becomes ready.
- Validate the other constraints: TelPro signaling/RTP capacity, provider quotas, database/Redis, object storage and network. A spare Voice instance does not make a single shared dependency highly available.
For example, if your own validated allowance is 75 calls per Voice instance, a target of 500 calls needs seven serving instances. Eight provide one spare: seven remaining instances offer an allowance of 525. This is arithmetic, not a newly validated load-test result. Do not apply headroom twice if 75 already includes it. Keep the workload, release and measurement evidence with the deployment runbook.
Configure a server group
- Open Server Groups and inspect Service, Servers, Instances, Utilization, Refresh, Last Action and Last Error before changing an existing group. Inventory counts and runtime health are different views.
- Choose Create, or the row's edit icon. Set the fields below. For an existing deployment, preserve the established group identity until you have checked base-number assignments and provider instance discovery.
- Save with Create or Update. Reopen the group to verify persisted values, then check the runtime outcome. A database save can succeed while the best-effort Redis capacity sync logs an error.
- For Voice routing, assign the intended group through base-number administration. Verify a permitted test call reaches the intended group; do not assume creating a group moves existing calls or assignments.
| Field | Meaning, default and constraints |
|---|---|
| Group Name | Required, unique name, 1–100 characters. Treat a rename as an operational change: it changes the Redis namespace; existing instance registrations and provisioning discovery need verification. |
| Enabled | Defaults on. Only enabled groups with autoscaling enabled enter normal autoscaling evaluation. For Voice, the value is also exported to routing state; disabling is not a standalone, verified maintenance drain. |
| Service | Voice or API, selected when creating. The API rejects role changes afterwards and permits only one API group. |
| Description | Optional operational notes. |
| Load Balancing Method | Voice only: Round Robin, Least Calls, Random (round_robin, least_calls, random); default Round Robin. |
| Max Concurrent Calls | Voice only, per-server admission setting; default 100, integer 1–10,000. This is a configured cap, not measured capacity. |
| Enable Auto-Scaling | Defaults off. Reveals the following controls; turning it on authorizes automatic infrastructure changes when the Scaler can execute them. |
| Min Instances / Max Instances | Defaults 1 / 10; integers 1–100, minimum no greater than maximum. These are decision bounds, not a promise that this many healthy instances already exist. |
| Scale Up Threshold / Scale Down Threshold | Defaults 0.7 / 0.3, values 0–1; down must be less than up. Scale up uses at least the up threshold; scale down uses below the down threshold. |
| Cooldown (seconds) | Default 300, integer 0–3,600. Below-minimum recovery can bypass cooldown. |
| Scaling Provider | Required when autoscaling is enabled. Options include Hetzner Cloud, AWS EC2, Local Docker and Custom. A selectable provider is not proof that its scripts are installed. |
| Scaling Config (JSON) | Provider-specific configuration. Invalid JSON blocks submission; whitespace parses as {}. JSON syntax validation does not validate credentials, machine availability or every provider-specific field. Use the reviewed configuration for your deployment. |
There is no editable Priority control in this form. Do not follow older instructions that ask you to set one here, or substitute legacy load-balancing names such as weighted_round_robin.
The bundled scaling implementation includes executable Hetzner and local-provider scripts. The AWS directory in this release contains a README rather than the required executable scaling scripts; Custom likewise requires a separately supplied implementation. The executor requires provider-specific scaleUp.sh and scaleDown.sh in its configured infrastructure directory. A missing scalingStatus.sh returns an unknown status rather than the same missing-script failure; verify the provider’s status/discovery behavior separately. Establish the installed provider implementation and credentials before enabling autoscaling; the Local Docker option does not demonstrate cloud HA.
The existing-group editor also shows a read-only instance summary and table: instance count/enabled count, active calls (Voice) or connections (API), Voice RTP ports, configured scaling range, operational status, server ID/address, status, provider/management origin and last heartbeat. These are observations, not per-instance stop/restart controls. Compare the instance identity and fresh heartbeat with the deployment inventory before acting.
Interpret scaling and health
For the evidence behind a decision, follow Scaler authority, decision and provider logs. Separate a leader transition, an evaluation result and provider-script completion before concluding that healthy capacity changed.
The default evaluation interval is 30 seconds (SCALING_EVALUATION_INTERVAL_MS). The leader evaluates enabled, autoscaling-enabled groups. It counts healthy and provisioning capacity, recovers a deficit below minimum, then considers utilization and cooldown. Normal utilization growth adds one instance at a time; below-minimum recovery can request several Voice instances. Provisioning in progress can defer another decision. Lowering Max Instances is not an immediate force-shrink command.
| Service | Signal used for autoscaling |
|---|---|
| Voice | Active calls divided by configured capacity of healthy members. A cap that exceeds actual sustainable capacity makes this ratio misleading. |
| API | Connection utilization and event-loop lag contribute to scaling. JSON defaults include maxConnectionsPerServer: 1000 and eventLoopLagTargetMs: 100. These are targets/capacity assumptions, not CPU or RAM limits. |
The group list is an observation aid, not the exact scaling decision trace. Its API utilization is connection-based; the Scaler also considers event-loop pressure. Its Voice health window and the Scaler's admission-to-metrics window differ. A displayed healthy count therefore does not prove spare capacity or explain every scaling decision. Check Scaler logs, fresh instance heartbeats and the affected service's health checks together.
Voice capacity sync writes the per-member cap and computes a group capacity from Redis membership, using at least one member for the calculation. Membership is not a certified healthy capacity count. API group changes also mirror scaling fields, including Enabled, into an existing API configuration record; review API rate-limit behavior after such changes.
Adjust API rate limits
The API group's API rate limits tuning icon opens the existing API configuration record. The icon requires that record to exist; its absence is not a separate API-server provisioning control. Use the platform's deployment/bootstrap owner to investigate a missing configuration rather than creating duplicate records.
| Control | Behavior |
|---|---|
| Enabled | Whether this record can be selected as the active configuration. Disabling the record does not disable TelAPI; no active record falls back to enabled default limits. |
| Enable Rate Limiting | Defaults on. Turn off on an enabled record to disable this limiter. Other authentication, scope and subscription checks remain independent. |
| API Rate Limit Max / Rate Limit Window (ms) | Defaults 600 requests / 60,000 ms per client IP. Max 1–100,000; window 1,000–3,600,000 ms. |
The two numeric controls appear when rate limiting is enabled. Enter numbers; clearing the form is not a supported “inherit” action. The request limit applies to the selected window even where helper text says “per minute”. /health and /metrics bypass this limiter; the auth-path tier has a fixed 20 requests per 60 seconds preset rather than a field in this dialog. Verify client-IP forwarding for your deployment before interpreting per-IP counts.
Save, reopen the record, then verify the intended request class in an approved test. TelAPI uses Redis revision checks and a 60-second configuration cache; database/read failures can retain cached values or defaults. A save response or disabled record alone does not prove that protection is disabled on every instance. If behavior differs, inspect configuration revision/database errors and the selected enabled record before repeating a change.
Maintain capacity during a refresh
Use the background-job refresh workflow for supported service configuration refreshes. Before starting, record instance identities, images/configuration, healthy spare capacity, active workload, the strategy, timeout and recovery owner. Ensure the provider can provision replacement instances and that the target service is eligible. API and Voice refresh depend on provider scaling; TelPro drain-restart requires at least two ready edges.
This is the API/Voice drain-restart rolling replacement sequence. restart-only bypasses draining; it is not a call-preserving alternative. In the Service Refresh workflow, reload-only is unsupported for API and Voice: Tasker records those targets as skipped and does not queue their Scaler operations. Check each target result rather than interpreting a completed job as an applied reload. A refresh may temporarily need capacity above the steady-state fleet while a replacement becomes ready. With force: false, a timed-out drain does not authorize decommissioning in the rolling lifecycle; with force enabled, the drain helper can bypass the wait immediately despite remaining workload. Do not use force to make an unexplained drain appear successful.
Refresh is not the same operation as normal autoscale-down:
| Operation | Consequence to verify |
|---|---|
| Voice autoscale-down | Removes the selected instance from new-call routing and observes active application/SIP calls. The configured drainTimeoutMs defaults to one hour in this path; expiry can proceed to deletion even with active calls. Choose a timeout compatible with real call duration and monitor the drain. |
| API autoscale-down | Selects eligible Scaler-managed instances; Terraform-managed instances are protected from this automatic selection. Its default drain timeout is 60 seconds. If connections do not drain, it undrains the instance and defers deletion. |
| Refresh Stop/cancel | Requests cancellation at workflow boundaries. The current target may finish; completed changes are not rolled back. Inspect each host/instance result rather than assuming Stop restored the previous fleet. |
After a refresh, check both replacement readiness and the removal/state of the old instance. Verify expected healthy capacity, image/configuration identity, new-call/new-connection acceptance, completed conversations and error/latency behavior. A completed job is not evidence of interruption-free service or a representative HA test. Standalone “block new calls and drain without restart” is not established by these controls.
If a replacement never becomes healthy, preserve the previous serving capacity and investigate provisioning/registration errors before retrying. If cancellation, timeout or failure leaves a mixed fleet, collect per-instance results and reconcile actual infrastructure, runtime membership and database inventory. Use deployment recovery and release compatibility for a deliberate rollback; do not force-delete state to clear a dashboard warning.
Remove a group safely
Check routing/base-number references and real instance ownership before removing a group. The row's delete action sends a normal delete request; the API rejects it while database server instances remain. A group with zero recorded instances is not proof that no orphaned infrastructure exists.
The separate API forceDelete option initiates destructive fleet removal. For populated groups it changes runtime routing/status and marks pending deletion for the Scaler; the response is not evidence that all infrastructure has been deleted. It is not exposed by this row's normal Delete action and is not a maintenance shortcut. Require a reviewed decommissioning plan, verify no required routing remains, and reconcile provider resources afterwards.
For the actual target checkboxes, strategy controls, job progress and cancellation limits, use Service Refresh. Capacity planning does not replace checking each target’s applied result.