Deploy, verify and update Delphi

Use this journey to bring a deployment into service or change an existing one. Success means that the intended release and configuration are running, an authorized user can complete a test interaction, and you have a usable recovery point.

Availability: This procedure applies to deployment bundles using Linux service hosts, /opt/deployment/.env and /opt/services/<service>/. You need authorized host, registry, configuration-store and backup access. A Delphi application administrator role alone does not grant these infrastructure permissions. Use the service roles and optional modules listed in your deployment's release manifest.

For an existing installation, start at Prepare the change and recovery point. For a new installation, follow the steps in order.

1. Identify the environment and release

Record these before running an installation or update:

Item	What you need to establish
Deployment and namespace	The intended customer/environment and its NAMESPACE; do not share configuration namespaces accidentally.
Release identity	The agreed container set and configuration bundle reference, including the previous set for an update. Image tags and CONFIG_REF are separate inputs.
Enabled services	The roles, modules and network paths actually used by this deployment, including its telephony or browser channel.
Access and connectivity	Registry/config-bucket access, SSM/Secrets Manager permissions, DNS/TLS, database/Redis connectivity and the configured provider path.
Acceptance interaction	A dedicated test app/channel, an authorized test user and the expected result.

Use Release compatibility to select the container set and Prerequisites for a Delphi deployment to prepare hosts, network, cloud access and DNS. Compare their service lists with your selected deployment manifest.

This page covers deployment bundles on service hosts only. To prepare a new host, start with Provision instances and bootstrap a role.

2. Place configuration in the correct layer

In words: the deployment identity selects the bundle; its manifests select configuration; initialization applies it to the service; health and a real test interaction establish whether the result works.

Layer	Purpose	What to preserve before changing it
Bootstrap /opt/deployment/.env	Deployment identity and image/configuration references used by host scripts	The previous protected bootstrap file and exact references
Service bundle and vars.yaml	Compose files, scripts and declared variables/defaults	The exact previous bundle, including object version or immutable release reference
SSM and Secrets Manager	Runtime settings and credentials resolved at initialization	Parameter/secret identities and versions; recoverable previous values in approved secure storage
Application database	Teams, flows, routing and other saved application configuration	A verified database recovery point; any relevant configuration export separately
External object storage	Recordings, media and backup objects	The retention and recovery arrangements for those objects

For ordinary declared variables, the resolver collects local bootstrap values, then namespace-level SSM, service-level SSM, discovered Secrets Manager values and explicitly referenced external secrets. Later matching values override earlier ones. A manifest default fills an unresolved or empty value; composed variables are built afterwards. Only variables declared in the manifest enter the resolved output.

Avoid defining the same setting in several stores. If a change seems ignored, check its service, namespace and higher-priority source before repeating the restart. Find the variable and its consumers before deciding which hosts to touch.

On a provisioned host, inspect the expected API variables without requesting their resolved secret values:

/opt/services/common/show-env.sh --service api

A saved UI setting, a saved flow draft and a changed environment variable have different activation paths. Publish flow changes where required; use the setting's administration procedure for database-backed changes; initialize the affected service for environment changes. Saving one layer does not prove another has been activated.

3. Initialize dependencies, then application services

Provision the hosts and service bundles as described in Provision instances and bootstrap a role. Before application startup, establish the dependencies it needs: network/egress, database and Redis, required TLS material and any selected media/telephony services. Verify those dependencies before proceeding to their consumers.

Run each selected role's supplied init.sh from its service directory. For example, on the Ops host:

cd /opt/services/ops
./init.sh

Ops initialization runs the migration/baseline-seed container unless explicitly skipped. If it fails, the script refuses to start the Ops services. Investigate that failure before starting application versions that expect the migrated schema. Do not use skip options to make an incomplete first installation appear successful.

After successful initialization, perform the separate first-use registry setup on Ops:

cd /opt/services/ops
./delphi-setup.sh

The wrapper prepares the environment for the setup container. Follow Run first-use setup for the administrator and registry steps. For an upgrade, apply only the setup steps required by that release; first-use setup is not a replacement for migrations.

4. Prepare the change and recovery point

Before updating a running deployment:

1. Record the current image/configuration references, affected services and intended observable change.
2. Preserve bootstrap, bundle and parameter/secret versions. A TelWeb configuration export covers selected application configuration; it does not replace a full database backup or cloud secret recovery.
3. Verify the latest required database backup completed successfully. A storage object existing is insufficient if the backup job failed. Rehearse restoration into an isolated database before treating the recovery path as tested.
4. Preserve external recording/media objects required by your recovery scenario. A database dump does not contain them.
5. Plan traffic handling and maintenance access. A direct host restart can interrupt calls or browser sessions; use the deployment's applicable drain/rolling-refresh procedure where available.

Use configuration recovery for the detailed recovery checklist and activation/restart guidance to identify affected services. Do not interpret drain as a guarantee that every interrupted operation can resume.

Choose the affected consumers

Start with the changed value and its owning configuration layer. A store update, a new bundle and a saved database setting have different activation paths. Use the selected release's service manifest to identify consumers; a similar variable name does not prove every service reads it.

Change	Activation path	Verification boundary
Image tag	Apply update.sh --ecr-tag <image-tag> on each affected service host through the prepared deployment workflow.	Verify the actual running image identity. A reused tag or a skipped/failed pull does not prove new bytes are running.
Generated bundle	Apply update.sh --config-ref <bundle-ref> on affected hosts.	Verify the intended Compose, proxy/collector configuration and variable manifest were retrieved. Keep the image/bundle pair compatible.
Bootstrap values	Use supported update.sh options where available; operator-reviewed edits to other host bootstrap fields require that host's initialization workflow.	The script backs up bootstrap only when its supported variables change; a manual edit or plain restart does not establish a backup.
Startup SSM/Secrets Manager values or TLS material	Reread configuration through the affected service's initialization path; update.sh --restart-only keeps cached images.	Verify the new effective value or certificate/trust path at every consumer. This is not a general hot reload.
Database-backed settings	Follow the setting's own activation path.	Do not restart all hosts solely because a TelWeb form was saved. Some settings reload or invalidate caches; others need explicit synchronization.
TelAPI server rate-limit settings	The rate-limit loader refreshes on a Redis revision change or expiry of its 60-second cache.	No host restart is inherently required; a database/read failure can retain cached settings or defaults. Verify the effective limit rather than treating save success as propagation proof.
Database schema/setup	Follow the selected migration/setup release.	Normal Ops initialization can run migrations and seed before starting Ops; restart-only skips migration. An image restart cannot reverse committed schema changes.

For database authentication, distinguish startup resolution from the separate authentication-triggered credential refresh. That optional runtime mechanism does not reload arbitrary environment variables or all database consumers.

Service role	Identify the changed consumers	Plan and verify
Web	Public domains/proxy configuration, auth/session and password-policy environment, database/Redis connections and startup availability settings.	Verify sign-in and the affected UI path. Caddy's certificate renewal does not apply an arbitrary changed domain/Caddyfile. Coordinate password/reminder policy with Ops.
API	Public ingress, signing/authentication material, database/Redis connections and configured integration/media consumers.	Use the API group rollout path where configured; confirm healthy remaining capacity and affected sessions. A direct host restart can interrupt connections.
Voice	TelSys SIP and TelPhi runtime/provider configuration, database/Redis connections, recording/media and logging settings.	Arrange routing removal/drain and inspect active work before replacement. Verify SIP and media end to end; no general uninterrupted-call guarantee follows.
TelPro	SIP/TLS listener and advertised addresses, routing/Redis, proxy/media log settings and configured browser transport.	Use the applicable reconciler or planned host maintenance. Verify alternate ready edges before a rolling strategy; singleton edge maintenance can interrupt ingress.
Database	PostgreSQL, Redis and PgBouncer credentials, listener/TLS material and storage configuration.	Coordinate all consumers and their trust/credential sources. A bundled database restart affects them; managed databases use their own operator lifecycle.
Media	Server TLS, upload authentication, advertised names and storage path.	Coordinate presenting service and API/Voice trust consumers. Verify upload and retrieval through the application, not only server startup.
Ops	Scaler/Tasker environment, mail delivery, backup storage, reminder policy and migration connection.	Distinguish runtime environment from saved cron/job configuration. Verify the affected job and scaling behavior; do not rerun migrations merely to refresh a runtime value.
SigNoz	Backend, ClickHouse/ZooKeeper, retention and storage settings.	Plan the telemetry interruption and verify fresh logs/traces afterwards. Buffering does not promise no dropped telemetry.
Squid	Proxy ACLs, upstream route and listener.	Check dependencies on outbound registry, AWS and provider access before changing the proxy. Verify consumer connectivity afterwards.

Choose order from the changed dependency graph and release requirements. Make required proxy/trust dependencies usable before their consumers; coordinate a database or credential change with all clients. Establish the selected schema before starting application images that require it. Change shared infrastructure only when it is affected, and verify each stage before proceeding. For scaled API/Voice groups, confirm working replacement capacity and healthy remaining instances; “one at a time” alone is not proof of availability. The Service Refresh procedure below explains its supported targets, cancellation and force limits.

For TelPro on this release, normal initialization installs native redis-tools if redis-cli is missing; the host therefore needs access to its package repositories. Copying only the reconciler script is insufficient. The idle reconciler reads the existing container's Redis settings; refresh work can fetch current configuration. Its timer runs every 30 seconds and health/heartbeat entries expire after 120 seconds. Verify continuing heartbeats, actual dialog/media counts and the intended multi-edge behavior. A failed count probe is unknown, not evidence of zero active calls.

5. Apply the intended change

On a selected, prepared API host, preview an image/configuration update with the actual release references:

cd /opt/services/api
./update.sh --ecr-tag <image-tag> --config-ref <bundle-ref> --dry-run

The angle-bracket values are placeholders. A dry run displays the plan; it does not prove that images, bundles or dependencies are available. After checking the plan and arranging traffic handling, run the same command without --dry-run to apply it.

For a configuration-store change that should retain cached images:

cd /opt/services/api
./update.sh --restart-only

Restart-only is not an image upgrade. API and Ops initialization skip image pulls in this mode, but can still fetch configuration bundles. Ops also skips migrations. Do not use it when the intended change requires new images or a database migration.

The update script backs up bootstrap configuration when it changes bootstrap values. A restart-only run does not guarantee a new bootstrap backup. Retain the recovery evidence from the preceding step yourself.

Refresh running services deliberately

Use Admin Settings → Service Groups → Refresh several services to queue a coordinated refresh of TelAPI, TelPhi or TelPro after a reviewed configuration change. The screen is not an image/configuration editor or a general restart button for Web, database, Media or Ops. Keep the intended release/configuration identity and recovery point from the preceding steps.

Availability: from 0.9.19. Starting or stopping a refresh requires Super Admin. TelAPI requires an enabled, scaling-enabled API server group; TelPhi requires an enabled, scaling-enabled voice server group. TelPro uses its host reconciler instead. Enabled checkboxes do not prove that replacement capacity, a working reconciler or a healthy failover path exists.

Select every required service before starting: one job sequences its service and instance targets, and one Stop request applies to the remaining batch. This is broader than maintenance on one selected group. Closing and reopening the batch recovers its latest global job; inspect that job and its complete target results before starting another operation.

The former Service Refresh tab has been removed. Bookmarks to /admin-settings/service-refresh redirect to /admin-settings/server-groups/maintenance, with Refresh several services open. Existing refresh job APIs remain available. The released documentation retains the earlier screen for those releases.

Target one server group

Availability: the Server Groups maintenance controls described here are available from 0.9.19. Reading the view requires a platform administration reader role; starting maintenance and requesting Stop require Super Admin. Ops must configure the fixed TelPro inventory, routing integration and reporting reconciler capabilities. A visible group does not establish an accepted failover path.

Open Admin Settings → Service Groups and inspect the group's details before starting. TelPro shows separate SIP and media identity owners, each edge's admission state and report age. Active calls counts a logical call once; SIP dialogs and RTP sessions are separate edge counters. A stale report, missing probe or incomplete call snapshot produces Unknown rather than evidence that the edge is empty. Use Show calls for the selected edge; expand a call to inspect its SIP legs and selected Voice server. Find conversation performs an on-demand lookup and may have no result before the Voice server creates its record. Voice group details similarly distinguish canonical routed calls from the server's own counter.

Before TelPro handover, Ops must verify the two-edge prerequisites: stable per-edge dialog listeners/Record-Route/contact addresses and per-call RTP anchoring remain reachable when new-call admission moves. Floating ownership does not transfer TCP/TLS connections or service state. Hetzner behavior depends on the configured topology; configured-http is an integration contract. AWS routing and actual carrier continuity remain untested, so keep HA disabled until the relevant live checks pass.

1. Use the single operation form’s Target selector to choose All edges or a specific TelPro hostname. Group-wide and single-edge work share this form; selecting another target loads that target’s latest operation. Verify fresh target telemetry and an accepting peer. TelPro machines are fixed Ops resources; these controls do not provision, remove or autoscale them. API and Voice refresh retain their existing scaling prerequisites.
2. Choose Drain without restart on one TelPro edge to block new calls and wait for its real workload to finish. Enter an operator reason and a timeout from 1 to 1440 minutes; the default is 60 minutes. Drain-only does not fetch configuration or restart containers. A timeout or failed handover leaves admission blocked, including across timer runs and host restart.
3. Read the returned job and per-target progress. A lease prevents a second scoped maintenance job from overlapping the group. If the request fails or its result is uncertain, inspect the existing job before retrying. Reopening an API or Voice maintenance view recovers the latest job for that group; an edge panel recovers its last reconciler job. The latest status, error and target outcomes stay visible. Expand Operation history for a scrollable, height-limited timeline, newest events first; Show older events reveals another 20 entries. Events are rendered on demand from the fetched job result, not paginated from the server. Expand Technical details only when you need the job identifier or complete diagnostic JSON. The final result retains completed targets, skipped targets and cancellation details. If progress cannot be loaded, use Retry progress before starting another operation; a missing progress response does not prove that the previous job stopped.
4. Use Resume accepting calls when maintenance is finished. Resume verifies services before reopening admission. A selected edge must have fresh telemetry; a stale or failed peer does not block targeted Resume, while unknown target telemetry still does. It is a separate action from Stop and does not automatically move SIP or media ownership back. Resolve an unhealthy target before retrying Resume.
5. Stop after current safe boundary lets the current target finish and skips remaining targets. Completed changes remain applied. Check the final per-target result, admission and routing owners; Stop is not rollback.

Refresh configuration drains, applies configuration, verifies and resumes; Restart service can interrupt calls. Proceed after timeout even with active calls permits interruption in the refresh path. It is never available for drain-only. Activate TLS certificate selects the server-TLS operation: the candidate key pair, identity, trust chain and remaining validity are validated before replacement. Normal activation reloads and verifies the certificate through a new TLS connection. If reload fails, fallback requires an accepting peer, blocked admission and a successful drain before recreating Kamailio. Failure retains or restores the previous material where possible. After rollback, restore the authoritative certificate source to the verified material before Resume; a matching live fingerprint alone does not repair the secret source. The host pins the expected certificate fingerprint, so recreation rejects a different source instead of silently reinstalling it. Restart fallback keeps admission blocked until a new connection verifies the certificate and routing restoration succeeds. Retain the served fingerprint, expiry and certificate/key secret version IDs in the operation record; missing source identifiers remain Unknown. Failed recovery leaves the edge blocked for explicit operator recovery. Establish the actual material and changed consumer using platform TLS verification.

Before accepting continuity, measure approved answered calls, bidirectional media and new calls through the actual carrier/topology in both handover directions. Component tests and a completed job do not prove uninterrupted media or carrier acceptance. Follow capacity during maintenance and SIP diagnosis for workload and routing verification.

Choose targets and a strategy

1. Confirm which services consume the changed settings and which running instances will be affected. Verify remaining capacity, current calls/sessions and the maintenance/recovery plan before proceeding.
2. Select the service checkboxes. The form initially selects TelPro only; this is not an automatic recommendation for every change. TelAPI/TelPhi can be disabled when their scaling prerequisites are absent.
3. Choose the strategy and complete the applicable controls below. Do not combine unrelated service changes merely to make one job.
4. Choose Start refresh once and record the returned job ID. The API queues SERVICE_CONFIG_REFRESH with one allowed attempt; queue acceptance is not refresh completion. If the response is uncertain, look for the existing job before submitting another.

Control	Effect and important limit
Strategy → Drain and restart (recommended)	Attempts the service's drain/rolling path before replacement or restart. Form default. TelPro requires at least two registered ready edges for this strategy; it rejects a smaller set. This is a prerequisite check, not a tested failover guarantee.
Restart only (no drain)	Omits drain waiting. Active calls/connections can be interrupted. API/Voice still use their scaler refresh machinery; this is not a promise of an in-place process restart.
Reload only (TelPro trunk TLS)	Do not apply this to API or TelPhi: their handler records those targets as skipped. The label is narrower than the default command: without an explicit component, the handler selects TelPro server-tls. See the TLS caveat below before using it.
Reason	Optional trimmed note, up to 500 characters at the API. Use a change reference, never secrets. An omitted reason receives an actor-based manual reason.
Drain timeout (minutes)	Shown for drain-restart; form default 60, accepted form range 1–1440 minutes. It is sent as milliseconds. This is not a guaranteed total job duration: provisioning, verification and orchestration add separate waits.
Skip drain wait and restart immediately	Default off. Opts into bypassing the wait even with active work; do not use it to “fix” a stuck job without accepting interruption. It is a separate flag from the selected strategy.

The form has no TelPro component selector. The API also supports an explicit component, but that does not make it a visible UI choice. For default reload-only, the edge attempts server-TLS rotation/verification and can fall back to restarting Kamailio if the reload fails. Therefore “Reload only” is not a reliable no-restart or no-call-impact promise. Use TLS activation to establish the exact material and consumer before choosing this path.

TelPro's host reconciler applies restart changes through update.sh --restart-only; it does not fetch a new image simply because a refresh was requested. API and TelPhi use scaler operations with replacement-instance behavior. Verify the resulting image and configuration identity separately rather than treating all three services as the same deployment mechanism.

Read the job and stop future work safely

The screen shows Latest refresh job, its ID/status, error, Refresh timeline and result details. Keep the job ID for job history. Inspect every selected service and instance: skipped, partial success, cancellation and a failed later target are different outcomes. An earlier target's applied change is not rolled back because another target failed.

Use Stop refresh only for a pending/running job when you want to prevent later work. This requests cancellation between targets: the currently refreshing host finishes; remaining hosts/services are skipped where their orchestration checks the cancellation flag. It is not immediate process termination, rollback or an “undrain everything now” action. A stop request can arrive during the final target, and the result can be marked canceled after that target finished. Recheck actual instance state and result before starting another job.

If a refresh is slow, distinguish waiting for active work, replacement readiness, edge acknowledgement and failure. Preserve the first error and per-target progress. Do not repeatedly force, stop and resubmit: those actions can overlap external work and make the applied configuration harder to establish.

Verify before restoring normal traffic

Confirm each intended target's result, effective configuration and image, healthy routing and actual service behavior. For voice/edge work, use an approved synthetic call through the changed path; for API work, verify an authorized request and the affected session path. A healthy process or completed job alone does not establish audio, external-provider access, secret adoption or uninterrupted existing calls.

If a target is skipped or fails, keep the already-applied target list with the recovery record and resolve the failed layer using the following recovery procedure. For services outside this screen, follow the appropriate host deployment procedure; do not select a superficially similar target as a substitute.

6. Verify each layer before admitting normal traffic

Check	Expected evidence	If it fails
Initialization	Successful script exit; migration result where applicable; intended running image/configuration identity	Stop the rollout and inspect the failed stage. Do not assume an old running container is the new release.
Service and routing health	Required containers/dependencies healthy and the intended upstream reachable through its normal ingress	Check dependency connectivity, TLS, registry/config access and drain state.
User access	Authorized user can sign in and reach the intended team/app	Check identity, roles and feature availability rather than restarting unrelated services.
Test interaction	The dedicated phone/browser interaction reaches the intended published flow and produces its expected result	Follow call diagnostics for SIP, or the applicable integration/channel procedure.
Operational evidence	A fresh conversation and the expected logs, traces or selected artifacts for that test	Use the logging evidence map; an intentionally disabled recording is not a failed call.
Recovery readiness	Identified usable recovery point and an owner for rollback	Keep the change incomplete until the missing recovery prerequisite is addressed.

TelAPI exposes /health: its handler returns an error when Redis is unavailable, database state is degraded or the instance is draining. A successful response is useful routing evidence, but it does not prove that provider calls, SIP/media, authentication or every database operation work. Test through the normal ingress as well as checking service initialization.

Record the release, configuration references, test time and correlation identifiers with the outcome. Do not include credentials, real call content or full environment dumps in the change record.

7. Recover at the failed layer

• Wrong runtime setting: restore the intended parameter/secret version, activate its consumers and repeat the affected checks.
• Wrong image or bundle: select the known compatible previous references and use the applicable redeployment procedure. Check schema compatibility before downgrading application images.
• Schema/data failure: use the verified database recovery procedure and coordinate application/configuration versions. Replacing an image does not reverse a migration.
• Missing external artifacts: recover them from the storage recovery path; restoring Postgres alone cannot recreate their contents.

After recovery, repeat the same health and test-interaction checks. Keep normal traffic restricted until the relevant acceptance checks pass. Restore deployment configuration and restore Postgres provide the detailed recovery procedures.

Use Service topology to locate dependencies, Service health for role-specific verification, and Release compatibility to record the exact component set before and after an update.

Verify the legal-document catalogue

When deploying or investigating a wrong/missing legal document, check the catalogue metadata and served files together. Team administrators review and accept documents; operators verify the deployed catalogue. Delphi has no legal-document publication dashboard or routine publication command.

1. Record the intended document type/name, version, effective date, file path and active state from the approved deployment data. Preserve the existing version/consent records and the corresponding file assets through your established backup and recovery process.
2. Compare the entries returned by legalConsent.currentDocuments with that intended catalogue. This is a public read used during registration. It selects active versions by descending effective date, then creation date, retaining one per document type; it does not compare version labels numerically.
3. Open each returned file path in the deployed environment and verify that the file is reachable and matches the reviewed document/version. A valid database row or successful HTTP response does not establish that the correct text was deployed. Keep independently recorded file identities in your release evidence; the product consent schema does not store a content hash.
4. Ask an authorized team administrator to inspect the team's current/accepted version comparison without submitting acceptance for a deployment check. Verify the registration document list separately when self-service registration is enabled.

An active future-dated version can appear immediately. Selection does not filter effective dates against today's date, so setting a future date is not a scheduling mechanism. Coordinate activation and file delivery through the deployment's reviewed change procedure; stop if the intended procedure or rollback is not defined.

Do not rerun baseline.legal-documents as a generic publish or repair command. That setup step first deactivates every active version and then upserts/reactivates its two repository baseline documents. It preserves historical rows/consents but can replace the active catalogue and update a baseline version's metadata. A database or application-configuration restore alone is not proof that the matching legal files were restored. Preserve and verify both boundaries; do not overwrite a previously accepted file and describe the old consent record as proof of acceptance of new bytes.

Verify platform TLS

Use this procedure when installing or rotating certificate material, or when the stored configuration says TLS is enabled but the expected listener or verified connection is unavailable. First identify the connection: public edge TLS, carrier-facing SIP trust/client authentication, and internal database/Redis encryption are separate configurations. A renewed public certificate does not rotate every internal trust bundle.

Material or connection	Owner and activation boundary
Public API/TelPro server certificate and private key	Edge settings select domain, management mode and platform secret keys. Renewal writes secret values and service environment material; a separate refresh activates it on hosts. Load-balancer TLS termination can make API backend TLS deliberately unnecessary.
Inbound SIP client trust	Assigned CA material is exported to Redis and aggregated by the TelPro watcher. Per-trunk SIP checks enforce the configured client-certificate/name requirements; do not assume isolated trust stores per trunk.
Outbound SIP client identity	CA, client certificate and client key references feed exported egress TLS material. The watcher requires client cert/key and a target before producing a client profile; references alone do not establish loaded authentication.
PostgreSQL/PgBouncer and Redis	Database-host server material plus application-host trust bundles and each consumer's TLS settings. A shared namespace or certificate label is not evidence that both connection legs use verified TLS.

Establish the expected configuration

1. Record the environment, exact consuming service, endpoint hostname/port, TLS termination point and intended certificate identity/expiry. Read service topology before assuming this installation uses the bundled database host rather than managed infrastructure.
2. Review the relevant platform configuration references and credential-storage scope. Inspect identifiers/status rather than copying PEM private keys or secret JSON into logs or tickets.
3. Check the effective deployment variables below, including the actual container configuration after recreation. Updating a secret or parameter is not proof an existing process received it. Use the established deployment/refresh procedure, with the background-job and service-health evidence for that target.
4. Compare successful materialization/activation with the listener or connection actually used. Check certificate identity, expiry, chain/name verification and application connectivity separately. These source paths do not certify a deployed endpoint or uninterrupted calls during rotation.

Public edge certificate lifecycle

EDGE_TLS_RENEW handles api and telpro. Its effective domain comes from the respective edge setting, otherwise Tasker's DOMAIN_API / DOMAIN_TELPRO. Automated issuance uses Certbot DNS-01 with the dns-hetzner-cloud plugin and requires HETZNER_API_TOKEN, DNS permissions and outbound connectivity. It is not a generic certificate issuer for every DNS provider.

Setting or result	Operational meaning
EDGE_TLS_RENEWAL_THRESHOLD_DAYS	Tasker default 30. A parseable certificate with more remaining whole days is skipped unless forced.
EDGE_TLS_CERTBOT_STAGING	Default false; true asks Certbot for its staging issuer. A staging-issued result is not a production trust result.
CERTBOT_EMAIL	Issuance contact; configure the intended operator contact for this deployment rather than relying on the repository default.
managementMode=manual	Ordinary renewal skips this target. A forced job bypasses manual mode and the age threshold; it can replace manually managed material. Force does not bypass the API backend-TLS-disabled check.
API_TLS_ENABLED in Tasker	Explicit override for whether API backend TLS is enabled; otherwise the handler reads that service's TLS_ENABLED parameter from SSM. Missing/read-failed SSM state is treated as disabled and API renewal/refresh is skipped. This does not inspect the load balancer's live certificate.
EDGE_TLS_FULLCHAIN_B64, EDGE_TLS_PRIVKEY_B64	Base64 certificate chain/private key provided to the service environment; base64 is transport encoding, not encryption. EDGE_TLS_CA_BUNDLE_B64 is optional additional CA material.
EDGE_TLS_VERSION	SSM signal bumped after secret synchronization. A bump failure is logged but does not roll back already written certificate material.

Renewal writes platform certificate/key values and merges the two edge values into the service's consolidated Secrets Manager object, then updates recorded timestamps and enqueues EDGE_TLS_REFRESH only when at least one target renewed. Refresh invokes the existing drain-restart service refresh path with force:false; it is a separate job, not an in-process certificate reload. Check its target outcomes. A completed renewal job may have skipped all targets, and partial external writes are not rolled back atomically. A stored expiresAt derived from whole remaining days is not a substitute for examining the certificate's actual expiry.

Verify the startup policy for the selected service. From 0.9.19, TelPro requires its authoritative secret source. The deployment now streams certificate, key and optional CA material through stdin into container tmpfs instead of passing it as container environment; the Docker host retains public fingerprints and recovery controls, not PEM or base64 key material. Missing, mismatched or unavailable source material blocks startup. TelPro does not fall back to host PEM files or silently disable required TLS. Recreation after activation or rollback must resolve the source matching the activated fingerprint. Follow the edge certificate activation and recovery procedure.

Other service callers of the shared startup resolver may accept existing base64 material or matching host PEM files and disable TLS when material is missing. Verify their effective runtime state rather than inferring it from the requested flag. The shared decoder checks certificate/private-key parsing and matching public keys; that check alone does not establish expiry, hostname coverage or trust-chain validity. Local make certs material mounted at /tls-in is not production issuance or automatic renewal. These TelPro guarantees do not imply that every service's credential workflow is memory-only.

Activate SIP trust and client material

Saving platform-secret metadata or a value is not equivalent to exporting every consumer. Inbound SIP configuration uses the platform-wide Redis sync; egress trunk updates have granular synchronization. Follow SIP configuration activation and the egress save boundary. The resolver can turn an inaccessible/missing TLS secret into empty material, so successful export alone is not proof usable PEM reached the edge.

The exporter writes voiceai:tls:materials:sip:* and voiceai:tls:materials:egress:*, then updates voiceai:tls:materials:version. With TelPro TLS_ENABLED=true, its watcher polls that version every WATCH_INTERVAL_SECONDS (default 5), regenerates tls.cfg, and attempts kamcmd tls.reload.

• Inbound CA entries are combined into one bundle; client-certificate inspection is configured globally and the SIP route applies the trunk's requirements.
• Outbound profile generation skips a trunk without resolved client cert/key or target address. The generated default client profile disables certificate verification; do not assume setting a per-trunk verification switch proves a suitable verified client profile was generated and selected. Confirm the effective profile and connection.
• If tls.reload fails or kamcmd is absent, the script logs the failure but still records the observed material version. An unchanged version does not automatically trigger a subsequent retry. Check the watcher and runtime state; a successful file replacement is not a successful reload.
• Removing material from the exported set rewrites configuration, but the script does not establish secure erasure of old per-trunk files or rollback of in-flight TLS sessions.

Never paste exported TLS-material payloads into diagnostics: outbound exports can contain private-key PEM. Record object IDs, version signal, timestamps and sanitized activation errors. When recovery is needed, use the reviewed deployment path and configuration recovery; a metadata restore alone cannot recover the external secret value or prove the previously loaded TLS state returned.

Establish and rotate Media HTTPS trust

Availability: from 0.9.19. Earlier versions decode the material to host files and can generate a self-signed certificate when none is present.

The internal Media host serves HTTPS with a certificate issued from a per-environment media certificate authority (CA). Consumers pin that CA, not the individual server certificate, so reissuing the server certificate for a new address or expiry needs only a Media host update. Rotating the CA also requires updating every consumer.

Issue the material from an operator workstation, never on a Docker host, with .infrastructure/scripts/issue-media-tls.sh --namespace <namespace> --ip <media-private-ip> [--dns <name>] [--days 825]. Without --apply the script issues and validates locally only. With --apply it stores, without printing values:

Stored value	Location	Purpose
Media CA certificate and key (MEDIA_CA_CERT_B64, MEDIA_CA_KEY_B64)	Secret <namespace>/pki/media-ca, created once and reused	The media CA (default validity 3650 days). It sits outside every service prefix, so no host loads it.
Media server certificate chain (MEDIA_TLS_FULLCHAIN_B64)	Media parameter, base64 of server certificate plus CA	Public certificate chain for the Media host
Media server private key (MEDIA_TLS_PRIVKEY_B64)	Media secret	Server private key
Consumer trust bundle (TTS_MEDIA_CACHE_CA_BUNDLE)	API and Voice parameters, base64 of the CA, written only when changed	Consumer trust. Web and Ops read the same variable; set it there yourself.

Media initialization then selects material in this order:

1. The stored certificate chain and private key, when both are valid.
2. Otherwise, PEM files under an explicitly configured MEDIA_TLS_CERT_HOST_PATH, with a warning to migrate to the issued material. This path is for existing installations only; the default is empty.
3. Otherwise, initialization fails with No valid media TLS material. There is no self-signed fallback.

Initialization rejects material when the key does not match the certificate, the certificate expires within one day, the certificate does not cover the host's private IP, or the server certificate does not verify against the issuer certificates in the chain. A chain value that contains a private key block is ignored with a warning; reissue it. The material is validated in memory and delivered to the Media proxy container over standard input into memory-only storage; it is not written to the Docker host, and old PEM files under the service's tls/ directory are removed. After delivery, initialization checks that the listener serves that certificate and that it verifies.

A container restart or host reboot starts the proxy without material, and it waits. The media-tls-reconcile.timer unit checks every 30 seconds and redelivers the material when the running container has none. If Media HTTPS stops answering after a restart, check that timer before reissuing anything.

Store the trust bundle in TTS_MEDIA_CACHE_CA_BUNDLE for each service using the endpoint. A single-line base64 encoding of the PEM avoids multiline configuration transport problems; API and Voice also accept PEM and normalize literal escaped newlines. When you rotate the CA, apply the new server material and consumer configuration in a coordinated maintenance window, then verify the restarted services.

On an API or Voice host, the shipped /opt/services/common/check-tts-media-cache-tls.sh --docker <container-name> helper can read the running container's cache URL and bundle, compare certificate fingerprints, and display OpenSSL/curl verification results. With a CA bundle, a fingerprint difference is expected; chain verification is authoritative. Inspect the output: the helper's exit status alone is not proof that every TLS check succeeded. Verify the chain and hostname for the actual URL, then make a permitted synthetic cache request or system-audio preview through each affected application path. A TLS handshake does not prove upload authorization or storage readiness.

If verification fails, check the selected material, URL/SAN, validity period and effective consumer bundle before changing anything else. Restore a still-valid coordinated server/trust pair or correct the intended pair; do not disable certificate verification. Use Deploy and verify system audio for the separate library-token, cleanup and storage checks.

Verify internal database and Redis encryption

For a bundled database host, prepare-tls-server.sh materializes INTERNAL_CA_CRT_B64, INTERNAL_TLS_CERT_B64 and INTERNAL_TLS_KEY_B64 under its tls/ directory; key mode is 0600 and ownership is prepared for the database image. Empty inputs produce placeholders, not certificates. The script does not provide issuance or cryptographic validation of the trio.

Application-host prepare-tls.sh writes DATABASE_SSL_CA_BUNDLE_B64 and REDIS_TLS_CA_BUNDLE_B64 into separate CA files; deprecated INTERNAL_CA_CRT_B64 is a fallback for either missing bundle. Without either input it writes an empty placeholder. Match the CA to the actual peer: PgBouncer, direct PostgreSQL/managed database, or Redis. Do not assume the public edge chain is the correct trust bundle.

Control	Behavior
POSTGRES_TLS_ENABLED=on	Bundled server enables SSL; default off. Certificate/key must be nonempty. POSTGRES_FORCE_SSL=on requires TLS on and switches nonlocal authentication rules to TLS-only.
PGBOUNCER_CLIENT_TLS_SSLMODE, PGBOUNCER_SERVER_TLS_SSLMODE	Separate client→PgBouncer and PgBouncer→PostgreSQL policies; bundle defaults are empty. Verify both legs instead of inferring backend encryption from a client's connection.
DATABASE_SSL_MODE	Shared Node helper: unset preserves URL behavior; disable explicitly disables TLS; require encrypts without certificate verification; verify-ca verifies chain but suppresses hostname checking; verify-full retains hostname checking. Explicit mode strips competing ssl* URL parameters.
DATABASE_SSL_CA_FILE	Shared helper default /etc/ssl/database/ca.crt; verify modes require the file to be readable. Readability alone does not establish a useful valid bundle. DATABASE_SSL_REJECT_UNAUTHORIZED=false or PG_SSL_INSECURE=true weakens verification and must not be described as verified TLS.
Database-host REDIS_TLS_ENABLED, REDIS_TLS_PORT, REDIS_FORCE_TLS	Defaults false, 6380, false. TLS-enabled bundled Redis can keep its plaintext listener until FORCE is true; FORCE requires TLS enabled. Server TLS uses --tls-auth-clients no, not required mutual client authentication.
Shared Node Redis helper	REDIS_TLS_ENABLED defaults false; REDIS_TLS_REJECT_UNAUTHORIZED defaults true. Optional REDIS_TLS_SERVERNAME supplies SNI; absent/unreadable/empty custom CA can fall back to system trust. REDIS_PORT still selects the destination port (default 6379); enabling TLS does not automatically switch this helper to 6380.
TelPro Redis tools	Entrypoint/watcher use REDIS_TLS_PORT (default 6380) when TLS is enabled, and invoke redis-cli with --insecure, with an explicit CA only when available. Do not extend the Node helper's verification guarantees to these tools or to every SIP-engine connection.

Prepare valid server material and client trust before tightening listener policy. Verify each consumer on the intended TLS port and each PgBouncer leg before removing plaintext access. This is a sequencing requirement, not a zero-downtime guarantee or a universal command. Managed AWS database/cache deployments have their own server and certificate lifecycle; the local database Compose switches do not configure those managed endpoints. Runtime database credentials and migration credentials also have separate ownership; a migration connection succeeding does not establish that normal service connections use the intended identity or TLS policy.

After rotation, verify the affected services' effective settings, successful connection, certificate chain/name and expected expiry. If a consumer fails, preserve the exact error and old/new certificate identities without secret contents; stop before disabling verification or globally forcing refresh as a workaround. A rollback needs still-valid old credentials/material and compatible trust/listener policy. Revoked or expired credentials cannot be made valid by restoring a database snapshot.

For database identities, secret formats, startup resolution and runtime refresh, use Configure and rotate database credentials. A successful migration, application query and backup each exercise a different consumer.

For fleet growth or replacement, use Capacity and maintenance to check server-group controls, API limits and the different drain-timeout outcomes before changing infrastructure.

Related

• Provision instances and bootstrap a role: to prepare a new host before the first deployment.
• Environment variable reference: to find the role and default of a variable you change.
• Release compatibility: to select a matching set of images and configuration.
• Choose and verify a configuration recovery: to roll back a change that failed verification.