Media, audio and speech
How call audio is encoded, cleaned and detected, which prepared audio plays to callers, how synthesized speech is cached and which speech providers, voices and languages a call uses. Browse the feature map or configuration finder.
Availability: Next. Entries that depend on an optional feature, a role or a deployment choice name it in their own Availability line.
Features in this family
- Voice codecs on SIP calls
- Audio format between the voice service and the AI runtime
- AI input and output audio format
- Audio preprocessing and noise reduction
- Barge-in while the bot is speaking
- System audio library
- Play system audio on call events
- System audio storage backend
- Cache synthesized speech for playback
- Media upload and playback server
- Speech recognition providers
- Speech context phrases
- Speech synthesis providers and voices
- Detect and switch the caller's language
- Flow primary language and language switching
Voice codecs on SIP calls
The voice service offers Opus, G.711 A-law and µ-law and G.722 in SDP, and each trunk can state a codec preference order for the carrier side.
For: Integrators, Team administrators
| Level | Where to change it | Owner section |
|---|---|---|
| Platform settings | SIP trunk and egress trunk › Codec Preferences(codecPreferences, PCMA, PCMU, G722, OPUS) | Routing and technical fields |
Read more:
- Reference: Codecs and DTMF
- Reference: SDP and media security
- Reference: Trunk fields
Related features: Audio format between the voice service and the AI runtime, DTMF transport over SIP (RFC 4733), SRTP media encryption.
Audio format between the voice service and the AI runtime
Each phone or browser entry point carries audio to the AI runtime as 16, 24 or 48 kHz linear PCM or G.711, 16 kHz by default, and the runtime resamples as needed.
For: Application builders, Integrators
| Level | Where to change it | Owner section |
|---|---|---|
| Flow and node settings | Entry point inspector › Audio(linkFormat, auto, slin16, slin24, slin48, alaw, ulaw) | AI audio format and preprocessing |
| Application settings | Application endpoint › Audio(Endpoint.audio.linkFormat) | AI audio format and preprocessing |
Read more:
- Reference: AI audio format and preprocessing
Related features: AI input and output audio format, Voice codecs on SIP calls.
AI input and output audio format
The audio format sent to and received from the AI provider is chosen per direction from what that provider supports, with 24 kHz PCM as the usual default.
For: Application builders
| Level | Where to change it | Owner section |
|---|---|---|
| Flow and node settings | Provider editor › Audio › AI Input / AI Output(ai.input, ai.output, pcm16, ulaw, alaw) | AI audio format and preprocessing |
Read more:
- Reference: AI audio format and preprocessing
- Set it up: 3. Check credentials and audio requirements
Related features: Audio format between the voice service and the AI runtime.
Audio preprocessing and noise reduction
Caller audio can pass through a preprocessing service that removes background noise with RNNoise or DeepFilterNet before it reaches speech recognition or the AI.
Availability: Requires the Audio preprocessing feature. Needs the audio preprocessing service in the deployment and PCM16 AI input.
For: Application builders, Deployment operators
| Level | Where to change it | Owner section |
|---|---|---|
| Flow and node settings | Provider editor › Audio › Audio Preprocessing(audioPreprocessing, enabled, denoise, denoiseBackend, denoiseAttenLimDb, denoisePostFilterBeta) | AI audio format and preprocessing |
| Services and containers | Voice environment(AUDIOPROC_ENABLED, FEATURE_AUDIO_PREPROCESSING) | AudioProc |
Read more:
- Set it up: AI audio format and preprocessing
- Reference: AudioProc
- Reference: Web UI feature conditions
Related features: AI input and output audio format.
Barge-in while the bot is speaking
Callers can interrupt realtime responses by speaking when interruption is enabled in the provider settings.
For: Application builders, Integrators
| Level | Where to change it | Owner section |
|---|---|---|
| Flow and node settings | Provider editor › Realtime Advanced Settings › Allow callers to interrupt realtime responses(interruptResponse) | Realtime Advanced Settings |
Read more:
- Reference: Realtime Advanced Settings
System audio library
Team administrators upload WAV clips or generate clips from text, and the platform prepares them for telephone playback in versions that flows can reference.
For: Team administrators
| Level | Where to change it | Owner section |
|---|---|---|
| Team settings | System audio(SystemAudioAsset, SystemAudioVersion, WAV, TTS) | Upload team audio |
Read more:
- Set it up: Upload team audio
- Set it up: Create audio from text
- Troubleshoot: Resolve upload and preparation errors
- Reference: Set storage allowances per plan
Related features: Play system audio on call events, System audio storage backend.
Play system audio on call events
A flow binds library clips to bot delays, long-running actions, transfers and errors, with an optional delay for long actions.
For: Application builders
| Level | Where to change it | Owner section |
|---|---|---|
| Flow and node settings | Flow Settings › System audio triggers(botDelay, longAction, transfer, error, delayMs) | Choose audio for App triggers |
Read more:
- Set it up: Choose audio for App triggers
Related features: System audio library, Audio while a transfer is in progress.
System audio storage backend
The operator stores prepared system audio on local disk or in S3-compatible buckets and runs the provisioning and cleanup jobs that keep it consistent.
For: Deployment operators
| Level | Where to change it | Owner section |
|---|---|---|
| Services and containers | Media upload environment(SYSTEM_AUDIO_STORAGE_BACKEND, SYSTEM_AUDIO_S3_*, SYSTEM_AUDIO_CACHE_*, SYSTEM_AUDIO_CLEANUP_TOKEN) | Configure the services |
Read more:
- Set it up: Choose a storage backend
- Reference: Provisioning jobs
- Set it up: Verify the deployment
Related features: System audio library, Media upload and playback server.
Cache synthesized speech for playback
Repeated bot phrases can be served from a shared media cache or a local cache instead of being synthesized again, set per flow with a retention limit.
Availability: Requires the TTS media cache feature.
For: Application builders, Deployment operators
| Level | Where to change it | Owner section |
|---|---|---|
| Flow and node settings | Flow Settings › TTS cache backend(ttsCaching, inherit, off, media, local, ttsCacheMaxRetention, disableTtsCache) | Choose TTS caching |
| Services and containers | Feature environment(FEATURE_TTS_MEDIA_CACHE, TTS_MEDIA_CACHE_*) | Deployment choices and effective service configuration |
Read more:
- Set it up: Choose TTS caching
- Reference: Deployment choices and effective service configuration
Related features: Media upload and playback server, Speech synthesis providers and voices.
Media upload and playback server
A media server stores cached speech and system audio and serves it to the voice service over HTTPS on the media domain.
For: Deployment operators
| Level | Where to change it | Owner section |
|---|---|---|
| Services and containers | Media environment(DOMAIN_MEDIA, MEDIA_ROOT, MEDIA_UPLOAD_LISTEN_ADDR, MEDIA_UPLOAD_TOKEN, TELPHI_MEDIA_*) | Media variables |
Read more:
- Reference: Media variables
Related features: System audio storage backend.
Speech recognition providers
Modular voice flows recognize caller speech with Azure Speech or Inworld, each with its own language, endpoint and recognition controls.
For: Application builders
| Level | Where to change it | Owner section |
|---|---|---|
| Flow and node settings | Provider editor › STT(azure_stt, inworld_stt) | Configure modular speech recognition |
Read more:
- Set it up: Configure modular speech recognition
- Reference: Azure STT controls
- Reference: Inworld STT controls
Related features: Speech context phrases, Detect and switch the caller's language.
Speech context phrases
A list of expected words and phrases helps the recognizer with names, product terms and other vocabulary that callers use.
For: Application builders
| Level | Where to change it | Owner section |
|---|---|---|
| Flow and node settings | Provider editor › STT › Speech contexts(sttSpeechContexts, phrases, sttContextId) | Speech context phrases |
Read more:
- Set it up: Speech context phrases
Related features: Speech recognition providers.
Speech synthesis providers and voices
Bot replies are spoken by OpenAI, Azure or Inworld text-to-speech, with a voice, speaking rate and output format set per provider.
For: Application builders
| Level | Where to change it | Owner section |
|---|---|---|
| Flow and node settings | Provider editor › TTS(openai_tts, azure_tts, inworld_tts, voiceName, speed, responseFormat) | Configure speech synthesis |
Read more:
- Set it up: Configure speech synthesis
- Reference: OpenAI TTS
- Reference: Azure TTS
- Reference: Inworld TTS
Detect and switch the caller's language
Recognition can listen for up to four candidate languages at call start or up to ten continuously, and can switch to a detected language above a confidence threshold.
For: Application builders
| Level | Where to change it | Owner section |
|---|---|---|
| Flow and node settings | Provider editor › STT › Language detection(languageDetectionActivate, languageDetectionMode, alternativeLanguages, languageDetectionMinConfidence, languageDetectionAutoSwitch) | Where to set the properties |
Read more:
- Set it up: Where to set the properties
- How it works: Detected-language switching
Related features: Flow primary language and language switching, Speech recognition providers.
Flow primary language and language switching
A flow sets its primary language and whether the conversation may switch to another language during the call.
For: Application builders
| Level | Where to change it | Owner section |
|---|---|---|
| Flow and node settings | Flow Settings › Language(language.primary, language.mode, strict, flexible, language.allowedLanguages) | Set language defaults |
Read more:
- Set it up: Set language defaults
Related features: Detect and switch the caller's language.