Skip to main content
Version: Next — unreleased

Deploy and verify system audio

Use this procedure to provide storage for the system audio library. The library covers uploads, preparation, authenticated previews and deletion. From 0.9.19 it also supports App-selected audio triggers for SIP calls, with internal media delivery to TelPhi.

Availability: from 0.9.18. You need deployment operator access to the Web UI, API, Tasker and media service configuration, and to the AWS account when you use S3. No virus scanner is enabled by default.

Choose a storage backend​

The media service stores system audio in one of two backends, selected with SYSTEM_AUDIO_STORAGE_BACKEND:

BackendUse it whenHow buckets or directories are chosen
filesystemDevelopment or single-host installations (default)One private directory per owner under the system audio root
s3Production installations on AWSOne private bucket per team plus one platform bucket, created automatically or mapped manually

Each team's files stay in that team's directory or bucket. The API and Tasker derive the storage scope from database ownership, never from a browser request, and check team membership on every request. There is no fallback to an unscoped directory or a shared bucket.

Environment variables defines every variable named on this page, with its default and service.

Configure the services​

  1. Apply the 0.9.18 database migrations before you start the dependent services. They keep existing records and select the newest ready platform file as the current shared file.
  2. On the Web UI, API and Tasker, set SYSTEM_AUDIO_STORAGE_BASE_URL (or keep the existing TTS_MEDIA_CACHE_BASE_URL fallback) and TTS_MEDIA_CACHE_UPLOAD_TOKEN. For Next call playback, configure the same connection on TelPhi. The token must match MEDIA_UPLOAD_TOKEN on the media service. The Web UI handles browser uploads and previews, so API-only configuration is not enough. Keep the token internal; authenticated browser preview URLs are not bot-facing playback URLs.
  3. Set TTS_MEDIA_CACHE_CA_BUNDLE when the internal media endpoint uses a private certificate authority. Keep TLS verification on and follow Media HTTPS trust when installing or rotating the certificate.
  4. Set the same SYSTEM_AUDIO_CLEANUP_TOKEN on Tasker and the media service. Restrict the internal media routes to trusted services.
  5. Optionally set SYSTEM_AUDIO_UPLOAD_MAX_MIB on the Web UI to lower the per-file limit. It accepts an integer from 1 to 100; the default is 100.
  6. Leave SYSTEM_AUDIO_CACHE_MAX_BYTES=0 so the media service reads verified files directly. If you enable the cache, it is isolated per owner and bucket under SYSTEM_AUDIO_CACHE_DIR.

When the storage base URL or upload token is missing, uploads and text-to-speech requests fail with System audio storage is not configured. An invalid URL gives Invalid system audio storage configuration. Both checks run before storage is reserved. They do not test the token, connectivity or bucket permissions, so complete the verification below.

Provision S3 buckets automatically​

  1. On the media service, set SYSTEM_AUDIO_STORAGE_BACKEND=s3, SYSTEM_AUDIO_S3_REGION, an environment-specific SYSTEM_AUDIO_S3_PREFIX, SYSTEM_AUDIO_S3_BUCKET_PREFIX and SYSTEM_AUDIO_S3_ACCOUNT_ID. Leave SYSTEM_AUDIO_S3_BUCKETS empty.
  2. Choose a globally unique bucket prefix of 3 to 26 lowercase letters, digits or hyphens that starts and ends with a letter or digit. The account ID is the 12-digit AWS account. Reserve the prefix for this environment only.
  3. Grant only the media service identity bucket creation and configuration for that prefix, plus object read, write and delete. The API, Tasker and browsers need no bucket credentials. The media service accepts AWS credentials from the host or its workload role.
  4. Allow the media service to reach S3 and STS. Before it creates a bucket, it confirms the account with STS GetCallerIdentity.
  5. Run the baseline seed and start Tasker.

The media service names buckets <bucket-prefix>-<team-id> for teams and <bucket-prefix>-platform for shared audio. For each bucket it checks the account, region and ownership tags, blocks public access, disables ACLs, enforces encryption and requires HTTPS before any upload. Existing KMS encryption stays in place and needs matching key permissions.

Provisioning stops on conflicting ownership tags, a non-empty untagged bucket, enabled or suspended versioning, or a failed security setting. An empty untagged bucket left by an interrupted creation is recovered.

Provisioning jobs​

A new team queues a provisioning job. The scheduled job system-audio-provision runs every hour at minute 17, covers existing active teams in pages of 100 and repairs missed jobs. Each job allows 10 attempts with exponential backoff. Uploads also check that the bucket is ready, so a delayed job does not block the first upload.

To reconcile at once, a Super Admin can select Run Now on the system-audio-provision schedule, as described in Change a recurring schedule deliberately. Alternatively, queue the same job with the script that ships in the scripts/system-audio/ directory of the Delphi platform source. Run it from a copy of that source for your release, with dependencies installed and DATABASE_URL set to the intended database:

pnpm exec tsx scripts/system-audio/reconcile-buckets.ts --enqueue

Deleted teams are skipped, and deleting a team does not remove its bucket. Monitor your AWS bucket quota and failed Tasker jobs.

Terraform module​

The AWS Terraform configurations for staging and production use the system-audio-buckets module. Set system_audio_enabled (default false) and system_audio_bucket_prefix, and assign the media role or user. When enabled, the module creates the platform bucket and attaches the media policy; team buckets are created at runtime. Its system_audio_media_environment output lists the media settings.

Terraform does not switch a running media service to S3, because service configuration keeps existing values. Apply the output values, clear any manual mapping and restart the media service only after the migration below succeeds.

Map existing buckets manually​

If you provision buckets yourself, set SYSTEM_AUDIO_S3_BUCKETS to a JSON map from platform and each team ID to a distinct existing private bucket. Leave SYSTEM_AUDIO_S3_BUCKET_PREFIX empty in this mode. A team without a mapping fails closed. With manual mapping or the filesystem backend, the provisioning job does nothing and creates no AWS resources.

Keep cleanup working​

From 0.9.19, deletion first checks saved App configuration and active-call references. A referenced file cannot be deleted. Saved version history retains its audio, including previous versions while runtime configuration synchronization is incomplete. Clearing the latest selection does not release all historical references; history is retained while its flow exists. Deleting a flow or an App releases references owned by its flows and all their saved versions. These retained bytes continue to count toward the team's allowance.

Tasker runs a recovery job every hour (at minute 17, UTC) that releases call references older than 24 hours, so a reference left by a TelPhi crash or deployment is released within about 25 hours. Calls are limited to two hours, so this recovery window preserves live calls. Apply the system-audio reference recovery database migration and keep the Tasker scheduler and workers running; the recovery job does not need storage cleanup credentials. Obsolete policy references are removed by the same migration.

An accepted deletion hides a file at once and stops its upload and preparation jobs. Tasker deletes the stored objects after a ten-minute safety delay and releases storage only when every deletion succeeds. A failed cleanup stays retryable and keeps counting toward the team's allowance. Shared replacement retains the previous file if it is still referenced; a new call using an older runtime snapshot must still resolve that file.

  • Use buckets on which versioning has never been enabled. Enabled or suspended versioning, object locks or denied deletes prevent storage from being released.
  • Do not add lifecycle expiration rules that remove live files.
  • Back up the database and stored objects together, with a retention policy that suits your installation.

When failed uploads hold storage, restore the storage and cleanup configuration first. Then ask a Team Admin to select Reclaim failed uploads and check the cleanup job in Background jobs. Do not release quota by editing the database.

Preparation​

Tasker verifies each source file's size and SHA-256, runs any configured source checks, converts the audio into a metadata-free PCM preview and publishes it only after verifying the stored preview. FFmpeg is required for conversion, including A-law input. Files stay unavailable for preview until they are Ready.

The prepared file is mono 16-bit PCM WAV at 8 kHz. From 0.9.19, App selections use this rendition automatically. TelPhi resolves ownership and the pinned version before retrieving internal media, including on cache hits. Managed library caching is separate from the optional media-service S3 cache and from an external playback request's cache option.

A source check that rejects a file blocks it from becoming ready. A check that times out or is unavailable also fails the file, which can then be retried. Files that are already ready are not checked again when you add a check.

Migrate existing storage​

  1. Back up the metadata and files together. Pause writes, preparation and cleanup.
  2. Export the asset-to-owner manifest from the database. Run pnpm exec tsx scripts/system-audio/export-storage-owners.ts from the same platform source copy and save its output to a file. Resolve any object without a known owner; never assign it to a team by guessing.
  3. For existing S3 storage, first copy a consistent snapshot of the old prefix to a private filesystem location.
  4. Set SYSTEM_AUDIO_MIGRATION_OWNERS to the manifest path and run media-upload -system-audio-migrate-source PATH as a dry run. The command accepts both the old asset/version layout and the owner/asset/version layout.
  5. Add -system-audio-migrate-apply to copy the files. The command checks each object's byte count and SHA-256 and keeps the source files.
  6. Switch the serving configuration only after a complete, verified copy, then resume writes.

After new uploads reach the new storage, changing the backend variable back is not a safe rollback. Pause writes and run a verified reverse migration, or return to a compatible application version.

Verify the deployment​

  1. With two test teams and one platform administrator, upload, prepare, preview, search, page through and delete files.
  2. Copy an asset ID from one team into a request for the other team and confirm that listing, preview and deletion are denied with an authorization error.
  3. Upload a new shared file and confirm that the current shared file stays available until the new one is Ready.
  4. For S3, confirm private access, a separate bucket per team, provisioning for a new team and rejection of a wrong account ID.
  5. Delete a file and confirm that the team's storage use drops after the safety delay.
  6. For App-triggered playback, select a Ready file in a test App, save and publish, and confirm the new runtime configuration reached TelPhi. Make a synthetic SIP call, trigger the selected event and verify the audio and its stop condition. Test a failed media retrieval and confirm the call follows its existing recovery behavior.
  7. Attempt deletion while a saved App configuration references the file and confirm rejection. Replace the shared file and confirm an existing saved selection still previews and plays the old version. A database publication followed by failed runtime synchronization must also retain the audio needed by the previous runtime configuration.