7 posts tagged with "operations"

Delphi v0.9.19 stores each call's SIP ladder and span tree after hangup, shows DTMF digits in the SIP ladder, runs Test Call on a temporary copy of the app, switches Azure speech recognition to the caller's detected language and repairs base numbers with a duplicated country code. It pins a media certificate authority for internal media HTTPS and binds TelPro media to its routed floating address. The upgrade requires operator actions, so read the upgrade notes before you start.

Delphi v0.9.18 splits Browser SDK and browser voice into independent capabilities that each default to off. It adds a team system-audio library, bot no-input handling, per-trunk SIP maintenance mode and structured transaction logging. The upgrade requires operator actions, so read the upgrade notes before you start.

Delphi v0.9.16-patch1 improves media advertisement, speech processing and RTP observability. TelPro media advertisement is clearer for LB/NAT hosts, Azure STT turn races are quieter under load, and RTPEngine final-stat spans expose per-endpoint counters.

Delphi v0.9.14-patch3 adds contract-managed billing tools, improves call debugging with a rebuilt Debug tab and SIP export, hardens text/voice handoff, and standardizes operator log-level controls across services.

Delphi v0.9.13-patch3 moves TelAPI server protection into Admin Settings → API Server Config with IP-based tiers, adds a Send test email control under Platform Settings, hardens modular voice pipelines (STT-first link format and deferred Azure STT recovery), refreshes the TelAPI OpenAPI snapshot (clearer error models), wires password-expiry policy env vars to Tasker, and fixes Kamailio rewriting for legacy egress Refer-To URIs.

Highlights​

• Replaced TelAPI RATE_LIMIT_* environment variables and the Per API Key toggle with tiered IP-based limits stored in ApiServerConfig and reloaded through Redis.
• Added API rate-limit fields in Admin Settings → API Server Config, with clearer 429 responses (RATE_LIMIT_EXCEEDED plus optional details).
• Added Send test email on Admin Settings → Platform Settings — queues an EMAIL_SEND Tasker job to verify SMTP/SES delivery before go-live.
• Hardened modular voice pipelines: Asterisk ↔ TelPhi link format now prefers the STT listen-side sample rate; TTS output aligns to STT when the provider supports it; Azure STT resets are deferred with jitter instead of reconnecting after every playback.
• Refreshed stored TelAPI OpenAPI contracts and WebRTC-aware component registration.
• Plumbed PASSWORD_MAX_AGE_DAYS, PASSWORD_WARNING_DAYS, and NEXTAUTH_URL to Tasker so password-expiry reminder emails use the same policy as TelWeb.
• Fixed Kamailio legacy egress Refer-To rewriting when TelPhi stamps x-delphi-egress-route on loopback URIs whose host fell back to the EC2 instance name instead of DOMAIN_TELPRO / REFER_TARGET_HOST.

TelAPI rate limiting​

Server protection is now configured entirely from Admin Settings → API Server Config (see API server config):

TelAPI loads limits from Postgres, caches them in Redis, and bumps revision key voiceai:api-server-config:rev whenever a SUPERUSER saves the config. No TelAPI restart is required for limit changes to take effect.

Team daily API quotas (per-team request caps in Users and teams → API Access Configuration) are unchanged — they are a separate layer from this server-protection throttle.

When a tier limit is exceeded, TelAPI returns HTTP 429 with:

{
  "error": {
    "code": "RATE_LIMIT_EXCEEDED",
    "message": "Too many requests",
    "details": {"max": 600, "after": "1 minute", "ttlMs": 42000},
    "requestId": "…"
  }
}

TelAPI OpenAPI​

The public interactive reference at /api/reference includes the WebRTC/API snapshot:

• WebRTC enabled — /openapi/telapi-webrtc.html

Platform settings — test email​

SUPERUSERs can queue a one-off EMAIL_SEND job from Admin Settings → Platform Settings → Send test email. Use it after configuring Tasker mail transport (EMAIL_TRANSPORT, SMTP or SES env vars) and before enabling password-expiry or notification templates.

1. Enter recipient, optional subject/body → Send test email.
2. Open Job History, filter EMAIL_SEND, confirm COMPLETED.
3. For SES sandbox accounts, verify both sender and recipient identities in the AWS console.

See Platform settings and Tasker (Ops).

Modular voice audio​

For flows using separate STT and TTS providers (not unified realtime), TelPhi now:

• Picks the Asterisk websocket link format from the STT input sample rate when audio.linkFormat is auto (e.g. slin16 for 16 kHz Azure STT).
• Downgrades TTS output to the STT sample rate when the TTS vendor natively supports it, avoiding per-chunk resampling on playback.
• Schedules deferred Azure STT recognizer resets after TTS playback (with grace window and jitter) instead of forcing a full reconnect on every turn — reducing load while still recovering stuck VAD.

Unified realtime providers (OpenAI Realtime, etc.) are unchanged. Operators can still pin audio.linkFormat in flow source view.

Password expiry and Tasker​

The daily PASSWORD_EXPIRY_CHECK cron job (Tasker) now reads the same policy env vars as TelWeb:

• PASSWORD_MAX_AGE_DAYS (default 90) — must match on TelWeb and Tasker.
• PASSWORD_WARNING_DAYS (default 10) — reminder window before expiry.
• NEXTAUTH_URL on Tasker — base URL embedded in password-expiry email links (must point at the public TelWeb hostname).

Telephony​

REFER_TARGET_HOST (TelPro) sets Kamailio REFER_SBC_HOST for loopback Refer-To detection. It defaults to DOMAIN_TELPRO in the TelPro entrypoint. Set it explicitly when carrier REFER targets must align with your public SIP hostname rather than the EC2 internal hostname.

Kamailio now rewrites legacy PSTN-offload Refer-To URIs that carry x-delphi-egress-route even when the Refer-To host is the instance HOSTNAME (ip-10-…), preventing transfers from stalling when the URI is still a telpro loopback target.

Operators​

Database migration​

Run voiceai-db-migrate before restarting application services:

• Drops rateLimitPerApiKey from ApiServerConfig.
• Raises the default rateLimitMax baseline to 600.

Baseline seeding creates the default API Server Config row when missing. Existing deployments keep their saved limits except for the removed per-key flag.

Upgrade checklist​

1. Run voiceai-db-migrate on the Ops host before rolling to v0.9.13-patch3.
2. Remove RATE_LIMIT_MAX and RATE_LIMIT_WINDOW_MS from the API service env bundle.
3. Review Admin Settings → API Server Config — confirm the API tier suits your traffic profile (default: 600/min API per client IP).
4. Set PASSWORD_MAX_AGE_DAYS, PASSWORD_WARNING_DAYS, and NEXTAUTH_URL on Tasker to match TelWeb.
5. Set REFER_TARGET_HOST on TelPro when DOMAIN_TELPRO alone does not match the hostname carriers use in REFER targets.
6. Send a test email from Platform Settings and confirm the EMAIL_SEND job completes in Job History.
7. Redeploy TelAPI, TelWeb, TelPro, Tasker, and TelPhi (Voice).
8. Spot-check the stored generic API reference for the updated snapshot.

SDK compatibility​

See the SDK section for installation and the per-version compatibility matrix. Patch releases do not change the TelPhi SDK contract.

Delphi v0.9.13-patch1 extends the voice provider catalogue, improves first-use egress setup and transfer routing fallbacks, and hardens Kamailio when Redis TLS is enabled.

Highlights​

• Added Gemini Live, Inworld STT/TTS, and Grok Realtime to the modular provider catalogue and Flow Builder editor.
• Added platform-managed team defaults (OpenAI/Azure secrets, WebRTC keys) and a first-use egress setup step so new deployments get outbound routing without manual wiring.
• Improved transfer routing when flows omit an explicit egress pool — resolves team/base-number routes and a configurable platform fallback route.
• Rebuilt TelPro/Kamailio with SSL-capable ndb_redis so Redis TLS deployments connect reliably and honor the Redis CA bundle.

AI providers​

The modular audio pipeline and Flow Builder provider editor now support three additional vendor integrations:

Provider configs that reference team variables now trigger flow resync when those variables change, so published apps pick up updated credentials without a manual republish.

OpenAI Realtime internals were refactored into a shared base used by Grok Realtime, reducing duplication across realtime providers.

Team defaults and egress​

New teams (and teams provisioned through signup) now receive platform-managed defaults automatically:

• Superuser-managed team secret variables for openaiDefaultKey and azureDefaultKey, copied from platform secrets when present.
• A system-managed WebRTC API key when WebRTC is enabled — these keys no longer count toward customer API-key limits.

Team admins no longer see superuser-managed variables in the team variable list; SUPERUSERs still see the full set.

Transfer routing resolves egress more consistently when a flow does not set an explicit routingPool:

1. Use the explicit pool or route ID from the flow when provided.
2. Fall back to an enabled egress route linked to the app base number or called number.
3. Fall back to the platform default global egress route ID configured under Admin Settings → Platform settings → Egress defaults.

The first-use delphi-setup wizard adds an egress step that creates a default outbound trunk and route for the initial team/base number and syncs them to Redis/Kamailio.

Existing deployments can run the idempotent backfill script backfill-team-defaults-and-egress (from the API package) to stamp team defaults and ensure a global fallback route exists. Set DEFAULT_EGRESS_ROUTE_ID, DEFAULT_EGRESS_TRUNK_ID, and related env vars as needed before running it against the target database.

Telephony fix​

Modular and realtime flows that collect keypad input through sandbox collectDtmf() or Flow Builder dtmf_collect nodes are unchanged.

Operators​

Database migration​

Run the standard voiceai-db-migrate one-shot before restarting application services:

• Adds TeamApiKey.systemManaged and marks existing System WebRTC Call Token Key rows as system-managed so they are excluded from customer API-key counts.

Then run ./delphi-setup.sh --apply-missing if the environment has not yet completed the new first-use egress step.

Kamailio / Redis TLS​

The TelPro image now builds ndb_redis against libhiredis_ssl, which Kamailio requires for the tls=1 Redis server parameter.

When REDIS_TLS_ENABLED=true:

• Kamailio keeps the Redis hostname (instead of resolving to IP) so TLS certificate SAN validation succeeds.
• The Redis CA PEM at REDIS_TLS_CA_FILE is injected into kamailio.cfg for ndb_redis.
• An upstream Kamailio/hiredis quirk is patched so the CA file path is passed correctly to OpenSSL.

Rebuild and redeploy TelPro after upgrade. Deployments using plaintext Redis are unaffected.

Release workflow​

The release retag workflow supports manual dispatch for patch retags.

Upgrade checklist​

1. Run voiceai-db-migrate on the Ops host before rolling application containers to v0.9.13-patch1.
2. Run ./delphi-setup.sh --apply-missing to pick up the first-use egress setup step on fresh or incomplete installs.
3. Review Admin Settings → Egress defaults and set a default global egress route ID if transfers should fall back when flows omit a pool.
4. For existing environments, consider the backfill-team-defaults-and-egress script to stamp platform-managed team secrets and the global fallback route.
5. Add team variables geminiDefaultKey, inworldDefaultKey, and xaiDefaultKey before enabling the new providers in flows.
6. Redeploy TelPro — required for Kamailio Redis TLS fixes.
7. Redeploy TelPhi, TelWeb, and TelAPI for provider, egress, DTMF, and flow-sync changes.
8. After egress trunk or route changes, verify Redis egress state as documented in Egress routing.

SDK compatibility​

See the SDK section for installation and the per-version compatibility matrix. Patch releases do not change the TelPhi SDK contract; stay on the v0.9.13 SDK compatibility range.

Delphi v0.9.13 fixes and hardens SIP transfer routing, strengthens account security, improves Flow Builder recovery workflows, and gives operators a cleaner upgrade path with a dedicated migration container and clearer SIP observability.

Highlights​

• Improved SIP REFER / INVITE / BYE transfer routing, egress pool selection, and outbound SIP header/query-param handling.
• Added Flow Builder version rollback, AI checkpoints, and more resilient Flow Assistant retries.
• Added login lockout, password expiry, concurrent-session limits, a dedicated SuperAdmin Access Logs tab, and admin-triggered password reset.
• Improved billing and subscription handling with plan-based limits, pause/cancel/resubscribe paths, and safer Stripe status handling.
• Added dashboard error and caller-satisfaction views plus token usage by app.
• Added a one-shot voiceai-db-migrate container and new migration configuration for safer upgrades.
• Added an optional TelWeb legal notice banner driven by deployment env.
• Improved SIP diagnostics with richer ladder / DTMF observability, SIP_100REL_MODE, OTLP log-to-span parsing, and configurable PII scrubbing.

Telephony and Flow Builder​

Transfer handling supports stricter destination rules and improved egress routing for SIP integrations such as Genesys BYOC.

Transfer routing received another hardening pass after the first release-candidate notes:

• SIP REFER, REFER-Replaces, INVITE bridge, peer-to-peer re-INVITE, and BYE-based disconnect-reconnect transfers share the same validation path.
• Kamailio egress routing now handles routable Refer-To URIs and carrier/PBX egress pools more consistently.
• Egress trunks and base-number routing changes auto-sync their Redis state, reducing stale transfer routing after admin edits.
• Platform and team admins manage outbound routing from a unified Egress surface with Trunks and Pools sub-tabs — platform-wide at /admin-settings/egress, team-scoped on Settings → Team. Pools group trunks for round-robin or priority-failover transfers; flows reference them via routingPool / Route ID. See Egress routing.

Operators should review Voice and TelPro logs after upgrade because the call lifecycle now records more transfer, hangup, and SIP ladder detail.

Flow Builder also gains stronger recovery tools:

• Restore a previously published flow from Version History.
• Save and restore AI checkpoints while working with the AI Flow Builder.
• Recover failed Flow Assistant turns with automatic retry behavior.

Security and Sessions​

v0.9.13 adds several account-protection controls:

• Accounts lock after repeated failed login attempts.
• Passwords can expire on a configurable schedule, with warning notifications before expiry.
• Production deployments can cap concurrent sessions per user.
• SuperAdmin passwords now follow a stricter minimum length.
• SuperAdmins can review authentication and role-management audit events on Admin Settings → Access Logs (/admin-settings/access-logs) — filter by action, user, IP, date range, or SuperAdmin-only activity.
• Team admins can optionally override platform transfer restrictions per team from Settings → Team.
• Admins can trigger password-reset email for users.

The session model now uses SESSION_TIMEOUT as the main timeout control for both server session lifetime and client idle logout behavior. The previous warning-timer behavior is no longer the user-facing control.

Billing and Insights​

Plan limits now live on the plan model, including API-key limits, custom LLM-key allowance, and minimum commitment metadata. Subscription flows better handle pause, cancellation, and resubscribe paths, and Stripe trialing status is normalized before the database enum migration.

Dashboard reporting adds error widgets, caller-satisfaction views, and token usage by app so admins can see which apps drive usage and where calls fail.

Azure TTS integrations can pass through custom SSML when a flow or provider configuration supplies it — test audio before publishing.

API​

For WebRTC-enabled deployments, TelAPI continues to expose SDK session, runtime, endpoint discovery, and /ws/session surfaces. When the webrtc feature is disabled for SIP-only deployments, those WebRTC/browser SDK surfaces are not registered; health endpoints remain available.

Operators​

v0.9.13 introduces a dedicated one-shot migration image:

• voiceai-db-migrate runs Prisma migrations and release seeding separately from long-running services. Ops init.sh invokes it during deploy.
• delphi-setup (first-use wizard, --apply-missing, dev seeds) also runs from the voiceai-db-migrate image via ./delphi-setup.sh on the Ops host — it is no longer shipped in voiceai-telweb.
• MIGRATION_DATABASE_URL is required for the migration container and should point at a privileged direct Postgres connection.
• DATABASE_SECRET_* variables can optionally refresh database credentials from AWS Secrets Manager at runtime.

The release also expands the telephony and observability surface:

• LEGAL_BANNER_TEXT / LEGAL_BANNER_HTML can show a public legal notice on unauthenticated TelWeb pages.
• SIP_100REL_MODE controls reliable provisional response behavior for carrier interoperability.
• SIP_CAPTURE_ENABLE_PJSIP_LOGGER now defaults to enabled in the Voice stack, increasing SIP message visibility and log volume.
• The log-to-span pipeline can convert SIP and media logs into OTLP spans for SigNoz analysis.
• SIP ladder debugging now includes DTMF evidence from SIP INFO, RTP/RFC4733 negotiation, media, and ARI-side events where available.
• ENABLE_PII_LOGGING=false (default) scrubs OTel collector logs/traces before SigNoz export and redacts TelPhi/TelAPI transcript + caller-number fields written to Postgres; MSISDN_UNMASKED_DIGITS controls trailing digits kept on masked numbers. SIP ladder logs stay unmasked; recording S3 sidecars can retain full transcript when recording is enabled.
• Transfer and egress routing changes affect TelPhi, TelSys, and TelPro; validate Redis egress state after trunk changes.

Upgrade checklist​

1. Audit subscriptions before migration and resolve any rows still using the old TRIALING status.
2. Add MIGRATION_DATABASE_URL for the voiceai-db-migrate container.
3. Run database migrations before restarting the application services (Ops init.sh or the db-migrate one-shot).
4. Run ./delphi-setup.sh --apply-missing on the Ops host when the release adds setup registry steps.
5. Review SESSION_TIMEOUT, MAX_CONCURRENT_SESSIONS, SESSION_HEARTBEAT_TTL, PASSWORD_MAX_AGE_DAYS, and PASSWORD_WARNING_DAYS.
6. Review SIP logging capacity because SIP_CAPTURE_ENABLE_PJSIP_LOGGER emits more detailed TelSys SIP logs.
7. Set SIP_100REL_MODE if your carrier requires a specific PRACK / reliable provisional response mode.
8. Review ENABLE_PII_LOGGING and MSISDN_UNMASKED_DIGITS on every collector and on Voice/API app containers. Production should keep PII logging disabled except short support sessions; staging/dev may enable it for debugging with test traffic.
9. Re-sync or verify Redis egress routing state after changing trunks, transfer restrictions, or base-number routing.
10. Update operators and support teams on the new Flow Builder rollback.
11. Review egress Trunks and Pools in Admin Settings and team Settings → Team before configuring flow transfers that reference a routingPool.
12. Point SUPERUSERs at Access logs for login-lockout and role-change investigations.

SDK compatibility​

See the SDK section for installation and the per-version compatibility matrix.