Database service operations

The Database service provides centralized PostgreSQL and Redis for every other service. Postgres is the primary data store; Redis serves as cache, session store, pub/sub bus, and leader-election backend. Data lives on an attached block storage volume at /mnt/data.

Overview

Instance

Instance vCPU and RAM are in Provision instances.

Field	Value
Network	Private only — no public IP
Block storage	Attached volume at /mnt/data — typically 30 GB (staging) / 50 GB (production)
Scalable	No — single instance

Containers

Container	Image	Port	Purpose
voiceai-postgres	postgres:17-alpine	5432	PostgreSQL primary
voiceai-pgbouncer	edoburu/pgbouncer:v1.25.1-p0	6432 (in-cluster)	Connection pooler (TLS-aware)
voiceai-redis	redis:7-alpine	6379 plain · 6380 TLS (when enabled)	Cache / pub-sub / leader-election
voiceai-otel-collector	otel/opentelemetry-collector-contrib:0.150.1	—	Telemetry collector (also scrapes Postgres + Redis metrics)

PgBouncer fronts Postgres and brokers TLS for the SSL rollout (Leg 1 = app → pgbouncer, Leg 2 = pgbouncer → Postgres).

Data layout

/mnt/data/                  # attached volume
├── postgres/               # PostgreSQL data
├── redis/                  # Redis RDB
└── logs/postgresql/        # log files

Postgres consumers

Postgres is the durable source of truth for configuration and business data that must survive restarts, instance replacement, and Redis loss.

Consumer	Access	Main use
TelWeb	Read / write	Platform, team, user, app, flow, provider, notification-template, and admin configuration from the dashboard.
TelAPI	Read / write	API-facing platform state, call tokens, channel/session metadata, and client-visible configuration.
TelPhi	Read / write	Conversation records, summarized call state, provider and flow configuration, recording metadata, and runtime health metadata.
Tasker	Read / write	Cron schedules, queued jobs, notification rendering, recording post-processing results, conversation diagnostics persistence, password-expiry checks, cleanup jobs, and DB backups.
Scaler	Read / write	Server groups, desired capacity, scaling configuration, instance lifecycle state, and observed server metrics.

Configuration stored here includes ingress and egress SIP trunks, phone numbers, header rules, server groups, autoscaling policy, default provider settings, team variables and secret references, S3 bucket metadata for recordings, platform audio metadata, notification templates, cron schedules, user and team management, and app / flow / bot settings such as STT, TTS, LLM, action, and header-forwarding configuration.

Redis consumers

Redis is runtime coordination state, not the backup source of truth. Losing Redis may interrupt active calls or routing until services rebuild their state, but durable configuration should be restored from Postgres and the boot-time config sources.

Consumer	Access	Main use
TelPro / Kamailio	Read / write	SIP dialog state, dispatcher sets, Voice instance allowlists, registrations, egress trunk caches, and outbound header allowlist caches.
TelAPI	Read / write	Browser-to-ARI channel streams and short-lived API runtime state.
TelWeb	Read / write	Dashboard and platform runtime caches; durable settings still live in Postgres.
TelPhi	Read / write	Voice instance heartbeats, call health metrics, cached flow/provider data, and Redis streams used during live sessions.
Scaler	Read / write	Leader election, health metric reads, scale decision state, and dispatcher / server-list updates for Voice routing.
Tasker	Read / write	Leader election and the Redis/Postgres consistency check.

Runbook

Volume attachment (fresh deployment)

mkfs.ext4 -F /dev/disk/by-id/scsi-0HC_Volume_XXXXX
mount -o discard,defaults /dev/disk/by-id/scsi-0HC_Volume_XXXXX /mnt/data
echo '/dev/disk/by-id/scsi-0HC_Volume_XXXXX /mnt/data ext4 discard,nofail,defaults 0 0' >> /etc/fstab

cd /opt/services/database
./init.sh

Keep the database volume independent from the instance so a replacement host can reattach it safely.

Changing passwords

Updating Secrets Manager and Postgres is required (Postgres does not re-read env on reconnect):

# 1. Update POSTGRES_PASSWORD in Secrets Manager, then restart Compose:
./update.sh --restart-only

# 2. Update inside Postgres:
docker compose exec voiceai-postgres psql -U voiceai -c "ALTER USER voiceai PASSWORD 'new-secret';"

Redis only needs a Compose restart (password is read on start).

Internal TLS rollout

Postgres and Redis both ship in plaintext mode and are flipped to TLS via a staged rollout. Full sequences live in security/internal-encryption. Highlights:

Postgres order:

1. Provision server cert material in Secrets Manager on the Database host (INTERNAL_TLS_CERT_B64, INTERNAL_TLS_KEY_B64, INTERNAL_CA_CRT_B64 — the CA chain Postgres/PgBouncer use for their listeners).
2. POSTGRES_TLS_ENABLED=on.
3. PGBOUNCER_SERVER_TLS_SSLMODE=verify-ca.
4. PGBOUNCER_CLIENT_TLS_SSLMODE=allow.
5. Flip each app's DATABASE_SSL_MODE=verify-full, with DATABASE_SSL_CA_BUNDLE_B64 (and default DATABASE_SSL_CA_FILE=/etc/ssl/database/ca.crt) matching the CA that signs PgBouncer's cert (or the RDS/Aurora CA when connecting straight to AWS).
6. PGBOUNCER_CLIENT_TLS_SSLMODE=require.
7. (Optional) POSTGRES_FORCE_SSL=on.

Redis order (independent of Postgres):

1. REDIS_TLS_ENABLED=true (both plaintext and TLS listeners stay up; OTel auto-follows).
2. Flip each app's REDIS_TLS_ENABLED=true and set REDIS_TLS_CA_FILE=/etc/ssl/redis/ca.crt (default). Supply REDIS_TLS_CA_BUNDLE_B64 when verifying against a private CA; otherwise an empty placeholder and the system trust store is typical for Amazon-signed Redis.
3. REDIS_FORCE_TLS=true to drop the plaintext listener.

The TelPro Kamailio ndb_redis module additionally requires a voiceai-telpro image built with hiredis_ssl.

Backups

# Postgres dump
docker compose exec voiceai-postgres pg_dump -U voiceai voiceai > backup.sql

# Restore
cat backup.sql | docker compose exec -T voiceai-postgres psql -U voiceai voiceai

# Redis RDB
docker compose exec voiceai-redis redis-cli -a $REDIS_PASSWORD BGSAVE

The Tasker on Ops runs scheduled Postgres backups to S3_BUCKET / S3_PREFIX.

Configuration

PostgreSQL

Name	Source	Scope	Default	Description
POSTGRES_USER	SSM	all	voiceai	Database superuser.
POSTGRES_PASSWORD	Secrets Manager	all	—	Database superuser password.
POSTGRES_DB	SSM	all	voiceai	Default database name.
POSTGRES_TLS_ENABLED	SSM	all	off	Turn on TLS in the postgres command.
POSTGRES_FORCE_SSL	SSM	all	off	pg_hba.conf hostssl-only (requires TLS).
PGBOUNCER_CLIENT_TLS_SSLMODE	SSM	all	—	app → pgbouncer leg.
PGBOUNCER_SERVER_TLS_SSLMODE	SSM	all	—	pgbouncer → postgres leg.

Redis

Name	Source	Scope	Default	Description
REDIS_PASSWORD	Secrets Manager	all	—	Redis auth password.
REDIS_TLS_ENABLED	SSM	all	false	Enable TLS listener (`:6380`).
REDIS_TLS_PORT	SSM	all	6380	TLS listener port.
REDIS_FORCE_TLS	SSM	all	false	Disable plaintext listener once every client moved.

TLS material (decoded by init.sh)

Name	Source	Scope	Default	Description
INTERNAL_CA_CRT_B64	Secrets Manager	database	—	Base64 PEM of the CA used for Postgres/PgBouncer server TLS on this host.
INTERNAL_TLS_CERT_B64	Secrets Manager	database	—	Base64 PEM of the server cert (Postgres / PgBouncer / Redis server share it on this host).
INTERNAL_TLS_KEY_B64	Secrets Manager	database	—	Base64 PEM of the server private key.

OTel monitoring user

Name	Source	Scope	Default	Description
POSTGRES_MONITORING_USER	SSM	all	monitoring	Created by postgres-init.sh on first boot.
POSTGRES_MONITORING_PASSWORD	Secrets Manager	all	—	Monitoring user password.

Troubleshooting

Symptom	Likely cause	Check
Postgres not accepting connections	Container crashed or disk full	docker compose ps; df -h /mnt/data.
Redis not accepting connections	Password drift across services	Verify REDIS_PASSWORD matches everywhere.
Slow queries	Missing indexes or high load	SELECT * FROM pg_stat_activity; enable slow log via ALTER SYSTEM.
Disk full	Volume at capacity	df -h /mnt/data; grow the attached database volume.
Redis memory high	Stale streams / keys	INFO memory; clean up via SCAN + DEL or fix the producer.
Monitoring user errors	postgres-init.sh only runs on first boot	Manually CREATE USER monitoring and grant the SigNoz role.

Useful Redis keys

docker exec -it voiceai-redis redis-cli -a "${REDIS_PASSWORD}"

KEYS *stream*                          # TelAPI channel streams
KEYS *leader*                          # Scaler / Tasker leader election
KEYS voiceai:scaler:*                  # Scaler keys
KEYS *dialog*                          # Kamailio dialog state
KEYS voiceai:egress*                   # Egress trunk configs
SMEMBERS voiceai:telsys:server_ips     # Voice instance allowlist

See also

• API operations — primary Postgres consumer.
• Voice operations — secondary Postgres consumer (TelPhi).
• Ops operations — Tasker runs scheduled DB backups.
• Security: internal encryption — Postgres + Redis TLS rollout playbook.