Database service operations

platform v0.9.11verified 2026-05-14

The Database service provides centralized PostgreSQL and Redis for every other service. Postgres is the primary data store; Redis serves as cache, session store, pub/sub bus, and leader-election backend. Data lives on an attached block storage volume at /mnt/data.

Overview

Containers

Container	Image	Port	Purpose
voiceai-postgres	postgres:17-alpine	5432	PostgreSQL primary
voiceai-pgbouncer	edoburu/pgbouncer:v1.25.1-p0	6432 (in-cluster)	Connection pooler (TLS-aware)
voiceai-redis	redis:7-alpine	6379 plain · 6380 TLS (when enabled)	Cache / pub-sub / leader-election
voiceai-otel-collector	otel/opentelemetry-collector-contrib:0.150.1	—	Telemetry collector (also scrapes Postgres + Redis metrics)

PgBouncer fronts Postgres and brokers TLS for the SSL rollout (Leg 1 = app → pgbouncer, Leg 2 = pgbouncer → Postgres).

Data layout

/mnt/data/                  # attached volume
├── postgres/               # PostgreSQL data
├── redis/                  # Redis RDB
└── logs/postgresql/        # log files

Runbook

Volume attachment (fresh deployment)

mkfs.ext4 -F /dev/disk/by-id/scsi-0HC_Volume_XXXXX
mount -o discard,defaults /dev/disk/by-id/scsi-0HC_Volume_XXXXX /mnt/data
echo '/dev/disk/by-id/scsi-0HC_Volume_XXXXX /mnt/data ext4 discard,nofail,defaults 0 0' >> /etc/fstab

cd /opt/services/database
./init.sh

Keep the database volume independent from the instance so a replacement host can reattach it safely.

Changing passwords

Updating Secrets Manager and Postgres is required (Postgres does not re-read env on reconnect):

# 1. Update POSTGRES_PASSWORD in Secrets Manager, then restart Compose:
./update.sh --restart-only

# 2. Update inside Postgres:
docker compose exec voiceai-postgres psql -U voiceai -c "ALTER USER voiceai PASSWORD 'new-secret';"

Redis only needs a Compose restart (password is read on start).

Internal TLS rollout

Postgres and Redis both ship in plaintext mode and are flipped to TLS via a staged rollout. Full sequences live in security/internal-encryption. Highlights:

Postgres order:

1. Provision certs in Secrets Manager (INTERNAL_TLS_CERT_B64, INTERNAL_TLS_KEY_B64, INTERNAL_CA_CRT_B64).
2. POSTGRES_TLS_ENABLED=on.
3. PGBOUNCER_SERVER_TLS_SSLMODE=verify-ca.
4. PGBOUNCER_CLIENT_TLS_SSLMODE=allow.
5. Flip each app's DATABASE_SSL_MODE=verify-full.
6. PGBOUNCER_CLIENT_TLS_SSLMODE=require.
7. (Optional) POSTGRES_FORCE_SSL=on.

Redis order (independent of Postgres):

1. REDIS_TLS_ENABLED=true (both plaintext and TLS listeners stay up; OTel auto-follows).
2. Flip each app's REDIS_TLS_ENABLED=true + REDIS_TLS_CA_FILE=/etc/ssl/internal/ca.crt one at a time.
3. REDIS_FORCE_TLS=true to drop the plaintext listener.

The TelPro Kamailio ndb_redis module additionally requires a voiceai-telpro image built with hiredis_ssl.

Backups

# Postgres dump
docker compose exec voiceai-postgres pg_dump -U voiceai voiceai > backup.sql

# Restore
cat backup.sql | docker compose exec -T voiceai-postgres psql -U voiceai voiceai

# Redis RDB
docker compose exec voiceai-redis redis-cli -a $REDIS_PASSWORD BGSAVE

The Tasker on Ops runs scheduled Postgres backups to S3_BUCKET / S3_PREFIX.

Configuration

PostgreSQL

Name	Source	Scope	Default	Description
POSTGRES_USER	SSM	all	voiceai	Database superuser.
POSTGRES_PASSWORD	Secrets Manager	all	—	Database superuser password.
POSTGRES_DB	SSM	all	voiceai	Default database name.
POSTGRES_TLS_ENABLED	SSM	all	off	Turn on TLS in the postgres command.
POSTGRES_FORCE_SSL	SSM	all	off	pg_hba.conf hostssl-only (requires TLS).
PGBOUNCER_CLIENT_TLS_SSLMODE	SSM	all	—	app → pgbouncer leg.
PGBOUNCER_SERVER_TLS_SSLMODE	SSM	all	—	pgbouncer → postgres leg.

Redis

Name	Source	Scope	Default	Description
REDIS_PASSWORD	Secrets Manager	all	—	Redis auth password.
REDIS_TLS_ENABLED	SSM	all	false	Enable TLS listener (`:6380`).
REDIS_TLS_PORT	SSM	all	6380	TLS listener port.
REDIS_FORCE_TLS	SSM	all	false	Disable plaintext listener once every client moved.

TLS material (decoded by init.sh)

Name	Source	Scope	Default	Description
INTERNAL_CA_CRT_B64	Secrets Manager	all	—	Base64 PEM of the internal CA.
INTERNAL_TLS_CERT_B64	Secrets Manager	all	—	Base64 PEM of the server cert (Postgres / pgbouncer / Redis share it).
INTERNAL_TLS_KEY_B64	Secrets Manager	all	—	Base64 PEM of the server private key.

OTel monitoring user

Name	Source	Scope	Default	Description
POSTGRES_MONITORING_USER	SSM	all	monitoring	Created by postgres-init.sh on first boot.
POSTGRES_MONITORING_PASSWORD	Secrets Manager	all	—	Monitoring user password.

Troubleshooting

Symptom	Likely cause	Check
Postgres not accepting connections	Container crashed or disk full	docker compose ps; df -h /mnt/data.
Redis not accepting connections	Password drift across services	Verify REDIS_PASSWORD matches everywhere.
Slow queries	Missing indexes or high load	SELECT * FROM pg_stat_activity; enable slow log via ALTER SYSTEM.
Disk full	Volume at capacity	df -h /mnt/data; grow the attached database volume.
Redis memory high	Stale streams / keys	INFO memory; clean up via SCAN + DEL or fix the producer.
Monitoring user errors	postgres-init.sh only runs on first boot	Manually CREATE USER monitoring and grant the SigNoz role.

Useful Redis keys

docker exec -it voiceai-redis redis-cli -a "${REDIS_PASSWORD}"

KEYS *stream*                          # TelAPI channel streams
KEYS *leader*                          # Scaler / Tasker leader election
KEYS voiceai:scaler:*                  # Scaler keys
KEYS *dialog*                          # Kamailio dialog state
KEYS voiceai:egress*                   # Egress trunk configs
SMEMBERS voiceai:telsys:server_ips     # Voice instance allowlist

See also

• API operations — primary Postgres consumer.
• Voice operations — secondary Postgres consumer (TelPhi).
• Ops operations — Tasker runs scheduled DB backups.
• Security: internal encryption — Postgres + Redis TLS rollout playbook.