Ops service operations

The Ops service runs infrastructure management and background processing.

• Scaler — autoscales API and Voice instances based on utilization metrics. Provider-agnostic via per-provider shell scripts (scaleUp.sh, scaleDown.sh, scalingStatus.sh).
• Tasker — runs scheduled and queued background tasks (DB backups, maintenance, recording processing, email notifications) on a Redis-backed job queue.

Both components use Redis-based leader election with separate keys so only one active leader processes at a time.

Overview

Instance

Field	Value
Network	Private only — no public IP
Scalable	No — single instance

Containers

Container	Base	Purpose
voiceai-scaler	Node 24-alpine	Distributed scaling orchestrator
voiceai-tasker	Node 24-alpine	Background job + cron runner
voiceai-db-migrate	Node 24-alpine	One-shot Prisma migration and release seeding container
voiceai-otel-collector	otel/opentelemetry-collector-contrib:0.150.1	Telemetry collector

Neither the Scaler nor the Tasker exposes an HTTP health endpoint. Their health is determined by container status and log output.

Scaler decision loop

1. Every SCALING_EVALUATION_INTERVAL_MS (default 30s), the leader reads utilization metrics from Redis against scaleUpThreshold / scaleDownThreshold + min/max from ServerGroup.scalingConfig in Postgres.
2. On scale up, fetch optional bootstrap secrets from Secrets Manager (secretsName), generate a cloud-init via generate-cloud-init.sh, run scaleUp.sh.
3. Poll scalingStatus.sh until ready, then wait for a service heartbeat in Redis (~5 min for cloud-init + container startup).
4. Voice: append to the Kamailio dispatcher set in Redis. API: managed LB picks up via labels.
5. Cooldown — gates the next decision.

Scale-down reverses the flow: remove from routing, drain active calls (Voice waits up to 60 min), delete the server, clean Redis + Postgres state.

Tasker work model

The Tasker is two cooperating loops over Postgres tables — TaskerCron for schedules and TaskerJob for the queue:

• Scheduler (leader only). Every SCHEDULER_POLL_INTERVAL_MS (default 10s) the leader reads enabled TaskerCron rows; for any whose nextRunAt is due it inserts a TaskerJob and advances nextRunAt from the cron expression. Because only the leader schedules, each due run is enqueued exactly once across the fleet.
• Worker (every instance). Every WORKER_POLL_INTERVAL_MS (default 5s) each instance claims up to WORKER_CONCURRENCY (default 5) pending jobs with SELECT … FOR UPDATE SKIP LOCKED, so a job is never picked up twice. Failures retry with exponential backoff + jitter up to maxAttempts, then move to DEAD. Jobs stuck RUNNING longer than 5 minutes (e.g. a crashed instance) are recovered to PENDING.

Jobs reach the queue two ways: the scheduler creates them from cron rows, or another service enqueues them directly in response to an event (call hangup, registration, Stripe webhook, scaling event, …).

Tasker jobs and handlers

Each TaskerJob.type maps to exactly one handler. The handlers that ship in the Tasker:

Scheduled by default

These TaskerCron rows are seeded when the deployment is provisioned. Billing and registration schedules are filtered out at provisioning time when their flag is off (see below).

Job type	What it does	Default schedule	Feature flag
BACKUP_DATABASE	pg_dump of the platform DB uploaded to S3 (S3_BUCKET / S3_PREFIX), pruned to retentionDays.	Every 6h (0 */6 * * *)	—
BILLING_RETRY_FAILED_OVERAGE_PAYMENTS	Retry Stripe charges for subscriptions with failed overage invoices.	Every 6h (0 */6 * * *)	subscriptionManagement
BILLING_EXPIRE_GRACE_PERIODS	Cancel paused subscriptions past the 30-day grace period and release their numbers.	Daily 03:00 (0 3 * * *)	subscriptionManagement
BILLING_SETTLE_MONTHLY_OVERAGES	Bill accumulated overages for semiannual / yearly subscriptions.	Monthly, 1st 00:00 (0 0 1 * *)	subscriptionManagement
BILLING_CLEANUP_CANCELLED_TEAMS	Soft-delete teams whose subscription was cancelled past the retention period.	Daily 04:00 (0 4 * * *)	subscriptionManagement
CLEANUP_ABANDONED_REGISTRATIONS	Delete PendingRegistration rows never completed within retentionDays.	Daily 04:00 (0 4 * * *)	registration
TASKER_CLEANUP_JOBS	Delete completed / dead TaskerJob rows older than retention (default 180 days).	Daily 03:00 (0 3 * * *)	—
SCORING_CONVERSATIONS	Batch-score conversations not yet scored. Seeded disabled — enable only if you batch-score.	Every 10 min (*/10 * * * *)	qaScoring

Event-driven

Enqueued directly by other services as work happens — no cron row.

Job type	What it does	Enqueued by	Feature flag
EMAIL_SEND	Send a transactional email via SMTP or SES.	Flows, Platform settings test sender	—
NOTIFICATION_SEND	Render a DB notification template and deliver it.	Auth, registration, Stripe webhooks, Scaler, usage alerts, password-expiry	—
SMS_SEND	Send an SMS via Vonage.	Voice flow steps	sms
AUDIO_RECORDING_PROCESS	Stitch, transcode and upload call-recording segments to the team's S3 bucket.	TelPhi recording uploader	—
PERSIST_CONVERSATION_SPAN_TREE	Persist the SignOz span tree for a finished conversation.	TelPhi at call end	—
PERSIST_CONVERSATION_SIP_LADDER	Persist the SIP ladder for a finished conversation.	TelPhi at call end	—
SCORING_CONVERSATIONS	Score a single conversation right after hangup (when a transcript exists).	TelPhi flow engine	qaScoring
BILLING_SUBSCRIPTION_ROLLOVER	Roll a subscription into its next billing period.	Billing events	subscriptionManagement
BILLING_CALCULATE_OVERAGES	Compute usage overages for a billing period.	Billing events	subscriptionManagement

Available but not seeded

Handlers that exist but have no default schedule. Add a schedule from the Tasker admin in TelWeb (see Cron schedules) or trigger them ad-hoc.

Job type	What it does	Feature flag
TWILIO_RECONCILE_NUMBERS	Compare DB phone numbers against the Twilio account; optional autoFix.	—
REDIS_DB_CONSISTENCY_CHECK	Reconcile Redis routing / state against Postgres; optional autoFix.	—
PASSWORD_EXPIRY_CHECK	Warn users whose password nears the PASSWORD_MAX_AGE_DAYS limit (default 90); enqueues NOTIFICATION_SEND with a link built from NEXTAUTH_URL.	—
TTS_MEDIA_CACHE_CLEANUP	Delete expired TtsMediaCache rows and their files from the Media service.	ttsMediaCache

How feature flags gate Tasker work

Flags reach the Tasker in two layers:

1. Provisioning-time filter. The cron seeder drops the billing schedules (BILLING_*) when subscriptionManagement is off, and the registration-cleanup schedule when registration is off — those TaskerCron rows are never created.
2. Handler-time guard. Some handlers re-check a flag before doing work, so even a manually enqueued job safely no-ops when the feature is disabled:
   • SCORING_CONVERSATIONS requires qaScoring, and additionally honours per-plan / per-team / per-app QA toggles — a conversation that fails any check is marked SKIPPED.
   • SMS_SEND is only enqueued upstream when sms is on.
   • TTS_MEDIA_CACHE_CLEANUP targets the Media service cache, which only exists when ttsMediaCache is on.

Flag keys and their FEATURE_* env mapping are in Feature flags.

Runbook

Database migrations (v0.9.13+)

v0.9.13 introduces voiceai-db-migrate as the release migration path. Run it from the Ops host before restarting long-running application services on a new platform tag.

cd /opt/services/ops
./init.sh
docker compose --profile migration run --rm voiceai-db-migrate

The container requires MIGRATION_DATABASE_URL, which should point at a privileged direct Postgres connection. Keep normal app services on their standard DATABASE_URL; do not give long-running services the migration role.

After the migration completes, roll Web/API/Voice services to the new tag and run any required setup registry steps from this host (see below).

Platform setup (delphi-setup)

Schema migrations and baseline seed run automatically during init.sh (unless --skip-migrations). Interactive first-use work, upgrade registry steps, and optional dev seeds use the delphi-setup CLI in the same voiceai-db-migrate image.

From the Ops host:

cd /opt/services/ops

# Interactive first use (platform superuser, etc.)
./delphi-setup.sh

# After a platform tag bump — apply only new registry steps
./delphi-setup.sh --apply-missing

# Skip ECR pull when the db-migrate image is already local
./delphi-setup.sh --no-pull

delphi-setup.sh loads proxy + bootstrap vars, fetches SSM/Secrets Manager env into process memory, materializes TLS CA bundles, logs into ECR, and runs docker compose --profile migration run --rm db-migrate delphi-setup. Use ssh -t for interactive prompts. Full flag reference: delphi-setup CLI.

Do not run from TelWeb

delphi-setup is no longer installed in the voiceai-telweb image. Always use ./delphi-setup.sh on the Ops instance (or the raw Compose form documented in the CLI reference after replicating the in-memory env).

Email (SMTP or SES)

EMAIL_TRANSPORT selects between classic SMTP (default) and AWS SES on EC2.

SMTP: needs SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASS. Outbound traffic goes through Squid.

SES on EC2:

EMAIL_TRANSPORT=ses
AWS_USE_INSTANCE_PROFILE=true
AWS_REGION=eu-central-1
EMAIL_SENDER_ADDRESS=noreply@yourdomain.tld
EMAIL_SENDER_NAME="Delphi"

SES API calls honor HTTPS_PROXY / HTTP_PROXY / NO_PROXY (v0.9.13-patch3+). Include email.<region>.amazonaws.com in NO_PROXY when the proxy should not handle SES traffic.

SES IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ses:SendEmail", "ses:SendRawEmail"],
      "Resource": "arn:aws:ses:<region>:<account-id>:identity/<verified-domain>"
    }
  ]
}

EC2 / IMDS prerequisites:

1. IMDS hop limit = 2 — Docker bridge adds a hop, the default of 1 stops the container from reaching 169.254.169.254:

   aws ec2 modify-instance-metadata-options \
     --instance-id i-xxxxxxxx \
     --http-put-response-hop-limit 2 \
     --http-tokens required

2. Attach the instance role with the SES policy above.
3. If running with HTTPS_PROXY / HTTP_PROXY, add 169.254.169.254 to NO_PROXY.
4. SES account: verify sender domain (preferred) or address, enable DKIM, request production access if the account is sandbox-only.

Static AWS keys (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) must be unset on instance-profile deployments — the AWS SDK default chain prefers them and the IMDS intent gets defeated.

Verification:

docker compose logs tasker | grep 'SES: Initialized'
docker compose logs tasker | grep NotificationService

Bootstrap variables for scaled instances

When the Scaler creates a new Voice / API instance it passes its own env to generate-cloud-init.sh. The values come from a mix of sources:

Variable	Source
ENVIRONMENT	bootstrap (source: local)
ECR_REGISTRY, ECR_TAG	bootstrap
NAMESPACE, CONFIG_BUCKET, CONFIG_REF	SSM
BASTION_PUBLIC_KEY	SSM
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION	bootstrap
HTTP_PROXY / HTTPS_PROXY	SSM

Important: changing NAMESPACE / CONFIG_BUCKET / BASTION_PUBLIC_KEY in /opt/deployment/.env but not in SSM is a footgun — the SSM value wins after fetch-env.sh. Always update SSM.

Configuration

Scaler

Name	Source	Scope	Default	Description
SCALING_EVALUATION_INTERVAL_MS	SSM	all	30000	How often the leader evaluates scaling rules.
LEADER_ELECTION_KEY	SSM	all	voiceai:scaler:leader	Redis key for leader election.
LEADER_HEARTBEAT_INTERVAL_MS	SSM	all	5000	Leader heartbeat interval.
LEADER_LOCK_TTL_MS	SSM	all	10000	Leader lock TTL.
HETZNER_API_TOKEN	Secrets Manager	all	—	Provider API token (when scaling on Hetzner).
HTTP_PROXY	SSM	all	—	Squid proxy for outbound API calls.
NO_PROXY	SSM	all	localhost,127.0.0.1,10.0.0.0/8	Proxy bypass list; add 169.254.169.254 for IMDS access.

Tasker

Name	Source	Scope	Default	Description
WORKER_CONCURRENCY	SSM	all	5	Max concurrent job workers.
WORKER_POLL_INTERVAL_MS	SSM	all	5000	Worker poll interval.
SCHEDULER_POLL_INTERVAL_MS	SSM	all	10000	Scheduler poll interval.
TASKER_LEADER_ELECTION_KEY	SSM	all	voiceai:tasker:leader	Redis key for leader election.
S3_BUCKET	SSM	all	—	S3 bucket for database backups.
S3_PREFIX	SSM	all	database-dumps	Key prefix for backups.
EMAIL_TRANSPORT	SSM	all	smtp	smtp | ses.
SMTP_HOST	SSM	all	—	SMTP host (when EMAIL_TRANSPORT=smtp).
SMTP_PORT	SSM	all	—	SMTP port.
SMTP_USER	Secrets Manager	all	—	SMTP username.
SMTP_PASS	Secrets Manager	all	—	SMTP password.
AWS_USE_INSTANCE_PROFILE	SSM	all	—	true on EC2 SES deployments.
AWS_REGION	SSM	all	eu-central-1	AWS region for SES and other AWS clients.
EMAIL_SENDER_ADDRESS	SSM	all	—	From address (must be a verified SES identity).
NEXTAUTH_URL	SSM	all	—	Public TelWeb base URL embedded in password-expiry reminder links. Must match the TelWeb service value.
PASSWORD_MAX_AGE_DAYS	env	all	90	Password age limit — must match TelWeb for consistent expiry enforcement and emails.
PASSWORD_WARNING_DAYS	env	all	10	Days before expiry to send the reminder email — must match TelWeb.

Troubleshooting

Scaler

Symptom	Likely cause	Check
No scaling happening	Not running or not leader	GET voiceai:scaler:leader in Redis.
Scale-up fails	Provider API credentials missing or rate-limited	Scaler logs for API errors; verify HETZNER_API_TOKEN / AWS credentials.
New instance not healthy	Cloud-init failed	SSH via bastion; cat /var/log/cloud-init-output.log.
Scale-down too aggressive	Evaluation interval too short	Bump SCALING_EVALUATION_INTERVAL_MS.
SMTP / SES emails missing	Tasker side	Scaler enqueues only; check Tasker logs.

Tasker

Symptom	Likely cause	Check
Jobs not running	Not leader (for scheduled jobs)	GET voiceai:tasker:leader.
Jobs stuck	Worker concurrency or Redis	WORKER_CONCURRENCY; verify Redis connectivity.
SES 403 / MessageRejected	Sender unverified or IAM missing	Verify identity in SES console; check IAM policy.
SES CredentialsProviderError	IMDS hop limit / role	See SES section in Runbook.
SMTP failures	Squid blocking or auth wrong	Check Squid ACLs; rotate SMTP creds.

See also

• Voice operations — scale target.
• API operations — scale target.
• Database operations — Tasker runs DB backups here.
• Squid operations — required for SMTP and provider API calls.