Skip to main content
Version: 0.9.13

Internal encryption

How to flip Postgres and Redis from plaintext to TLS across every service without an outage. Each step is reversible and the system stays usable in every intermediate state.

The shape of the rollout​

  • Postgres has two legs: app → PgBouncer (Leg 1) and PgBouncer → Postgres (Leg 2). PgBouncer brokers between them so clients can upgrade independently.
  • Redis ships in plaintext mode and supports a parallel TLS listener; both stay up during transition. Each consumer flips at its own pace.

Cert material​

Two layers: material on the Database host (what Postgres, PgBouncer, and the Redis server present to clients) and trust bundles on every app host (what Node, TelPhi, and tooling use to verify TLS to Postgres and Redis). prepare-tls-server.sh / prepare-tls.sh in the deployment bundle decode Secrets Manager values and write files under each service’s ./tls/ directory before Compose starts.

Database host — server TLS (Postgres, PgBouncer, Redis)​

NameSourceScopeDefaultDescription
INTERNAL_CA_CRT_B64Secrets Managerdatabase—Base64 PEM of the CA used for Postgres/PgBouncer (and related server TLS) on the Database service.
INTERNAL_TLS_CERT_B64Secrets Managerdatabase—Base64 PEM of the server cert presented by Postgres and PgBouncer.
INTERNAL_TLS_KEY_B64Secrets Managerdatabase—Base64 PEM of the server private key. Decoded with mode 0600, owner 70:70.

SANs on the server cert must cover the Database instance’s private DNS (or whatever hostname apps use in DATABASE_URL when they terminate TLS at PgBouncer).

App hosts — verify Postgres and Redis (api, web, voice, ops, telpro)​

Apps no longer reuse INTERNAL_CA_CRT_B64 as the default trust anchor for outbound Postgres and Redis. Each consuming service should carry two optional base64 PEM bundles in its per-service secrets JSON; prepare-tls.sh decodes them to ./tls/database-ca.crt and ./tls/redis-ca.crt and the Compose file bind-mounts them read-only. If a bundle is unset, the script still writes an empty placeholder so the bind mount succeeds (correct for DATABASE_SSL_MODE=disable / plaintext Redis).

NameSourceScopeDefaultDescription
DATABASE_SSL_CA_BUNDLE_B64Secrets Managerapi · web · voice · ops—Base64 PEM CA bundle for PostgreSQL TLS (`verify-ca` / `verify-full`). Must sign the cert the client actually sees (PgBouncer when you route through it; RDS/Aurora CA when you connect straight to AWS).
DATABASE_SSL_CA_FILEenvapi · web · voice · ops/etc/ssl/database/ca.crtIn-container path to the decoded DB CA file (from `DATABASE_SSL_CA_BUNDLE_B64`).
REDIS_TLS_CA_BUNDLE_B64Secrets Managerapi · web · voice · ops · telpro—Base64 PEM CA bundle for Redis TLS verification. For ElastiCache with Amazon-issued server certs you can leave this empty and rely on the system trust store.
REDIS_TLS_CA_FILEenvapi · web · voice · ops · telpro/etc/ssl/redis/ca.crtIn-container path to the decoded Redis CA file (from `REDIS_TLS_CA_BUNDLE_B64`). TelPro/Kamailio defaults to a bind-mounted path under `/etc/kamailio/tls-internal/`.

INTERNAL_CA_CRT_B64 on app hosts remains supported as a deprecated fallback for older deployments that have not yet split secrets; new environments should populate DATABASE_SSL_CA_BUNDLE_B64 and REDIS_TLS_CA_BUNDLE_B64 explicitly.

Postgres rollout​

Order, one step per environment / deploy cycle:

  1. Provision server TLS on the Database host — push INTERNAL_CA_CRT_B64, INTERNAL_TLS_CERT_B64, and INTERNAL_TLS_KEY_B64 into Secrets Manager (see Database operations).
  2. POSTGRES_TLS_ENABLED=on — appends -c ssl=on to the Postgres command. Plaintext listener still works.
  3. PGBOUNCER_SERVER_TLS_SSLMODE=verify-ca — PgBouncer → Postgres now uses TLS.
  4. PGBOUNCER_CLIENT_TLS_SSLMODE=allow — clients can negotiate TLS to PgBouncer but aren't required.
  5. Flip each app's DATABASE_SSL_MODE=verify-full — one service at a time: api → web → voice → ops → others. Ensure each service’s secrets include DATABASE_SSL_CA_BUNDLE_B64 whose PEM signs PgBouncer’s certificate (the TLS peer for Leg 1). When DATABASE_URL points directly at RDS/Aurora, use the appropriate AWS RDS global CA bundle instead.
  6. PGBOUNCER_CLIENT_TLS_SSLMODE=require — refuse plaintext on Leg 1.
  7. (Optional) POSTGRES_FORCE_SSL=on — rewrites pg_hba.conf to hostssl-only. Belt-and-braces.

Each app’s DATABASE_SSL_CA_FILE defaults to /etc/ssl/database/ca.crt, populated from DATABASE_SSL_CA_BUNDLE_B64 by prepare-tls.sh.

Redis rollout​

Redis runs both listeners during transition: plaintext on :6379 and TLS on REDIS_TLS_PORT (default :6380).

  1. REDIS_TLS_ENABLED=true — restarts Redis with both listeners; the OTel collector's REDIS_ENDPOINT is auto-switched to :6380 by init.sh.
  2. Flip each app — set REDIS_TLS_ENABLED=true and point REDIS_TLS_CA_FILE at /etc/ssl/redis/ca.crt (default). Populate REDIS_TLS_CA_BUNDLE_B64 when you need a private CA; otherwise an empty placeholder + system trust store is normal for Amazon-signed Redis endpoints. Cycle order: api, web, voice, ops, telpro.
  3. REDIS_FORCE_TLS=true — once every client moved, set --port 0 to drop the plaintext listener.
TelPro Kamailio

The voiceai-telpro Kamailio image needs hiredis_ssl baked in for ndb_redis to talk TLS. Plain redis-cli health probes and bash tooling don't need the SSL-enabled image. Ship the SSL build of voiceai-telpro before flipping REDIS_TLS_ENABLED=true on TelPro. Decode REDIS_TLS_CA_BUNDLE_B64 into the Kamailio-visible CA path (same bind-mount pattern as other services); INTERNAL_CA_CRT_B64 remains a deprecated fallback on TelPro for older bundles.

Verification​

# Postgres TLS server-side
docker compose exec voiceai-postgres psql -U voiceai \
  -c "SELECT ssl, version FROM pg_stat_ssl WHERE pid=pg_backend_pid();"

# Postgres TLS client-side from an app instance
psql "postgresql://voiceai@<db-host>:5432/voiceai?sslmode=verify-full&sslrootcert=/etc/ssl/database/ca.crt" -c "SELECT 1;"

# Redis TLS listener
docker exec voiceai-redis redis-cli -p 6380 --tls \
  --cacert /tmp/redis-tls/ca.crt --insecure ping

# Redis TLS client-side from an app instance
redis-cli -h <db-host> -p 6380 --tls \
  --cacert /etc/ssl/redis/ca.crt -a "${REDIS_PASSWORD}" ping

Rollback​

Every step is reversible by setting the variable back and rerunning ./update.sh --restart-only on the relevant host. Because plaintext listeners stay up during the rollout, a partial rollback leaves the system functional.

See also​