Provision instances and bootstrap a role

Use this procedure to turn prepared cloud resources into running Delphi hosts. Each host learns its role from a bootstrap file, downloads that role's configuration bundle and starts the role's services. Complete Prerequisites for a Delphi deployment first.

To update or refresh a running deployment, use Deploy, verify and update Delphi instead. This page covers the first start of a host.

Availability: From 0.9.18, configuration fetch retries are adjustable on every role with FETCH_CONFIG_ATTEMPTS (default 5) and FETCH_CONFIG_RETRY_DELAY (default 5 seconds). Keep the defaults unless bundle publication often overlaps with host start. If you change them, set them in the bootstrap file before you provision, as described in Write the bootstrap file.

Plan the provisioning order

The reference Terraform configuration creates hosts in five phases, so that each role finds its dependencies running.

The proxy, telemetry, media and data hosts start first, then the edge, web and operations hosts, then the API, then Voice, and the bastion last.

• The API load balancer holds the managed certificate for DOMAIN_API, forwards to TELAPI_PORT and checks /health. It uses sticky sessions and selects its targets by role label.
• Voice starts only after Database, SigNoz, Web and TelPro exist.
• Ops can later create more API and Voice hosts through scaling. Those hosts follow the same bootstrap path.

Write the bootstrap file

Provisioning writes /opt/deployment/.env on every host with mode 0600. This file is the only local configuration on the host; everything else comes from the bundle, the parameter store and the secrets store.

Group	Variables
Identity	ENVIRONMENT, HOSTNAME, SERVER_ROLE
Images	ECR_REGISTRY, ECR_TAG
Cloud access	AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION
Configuration	CONFIG_BUCKET, CONFIG_REF, CONFIG_BASE_PATH, NAMESPACE
Network	PUBLIC_IP, PRIVATE_IP, FLOATING_IP, HTTP_PROXY, HTTPS_PROXY, NO_PROXY
Role-specific	Any extra values that the role's provisioning definition adds

When PRIVATE_IP is missing, host initialization uses the first private 10.x address of the host's eth0 interface. When PUBLIC_IP is missing, the reference Hetzner Cloud host initialization reads it from the provider's metadata service. TelPro stops with an error if it cannot find PUBLIC_IP.

Hosts that Ops creates through scaling also receive their role secret from the secrets store, appended to the bootstrap file before initialization. The Environment variable reference defines each variable.

Provisioning also writes /opt/deployment/metadata.json with the host name, environment, role, registry, region, bucket, bundle reference and creation time.

Understand host initialization

On first boot, cloud-init prepares the host and then starts the role. The sequence below shows the order.

cloud-init installs the tools, prepares storage and networking, and then hands over to init-service, which fetches the bundle, runs the role's init.sh and marks the host ready.

cloud-init performs these steps:

1. Installs base packages, creates the deploy user in the docker group and adds the bastion SSH key.
2. Writes the cloud credentials to /root/.aws/credentials with mode 0600.
3. Waits for and mounts the data volume on roles that have one. It formats the volume only when it is new.
4. On private hosts, writes the proxy settings to /etc/environment.d/proxy.conf and waits up to three minutes for the proxy.
5. Installs Docker Engine with the Compose plugin, the AWS command-line tools and yq. Docker keeps three log files of 20 MB per container.
6. Assigns the floating address, if the host has one.
7. Writes /opt/deployment/cloud-init-complete and starts /usr/local/bin/init-service.

init-service first runs a connectivity test. The test only reports DNS, HTTPS and registry reachability; it does not stop initialization.

Fetch the configuration bundle

fetch-config downloads the bundle for SERVER_ROLE from CONFIG_BUCKET. It reads the proxy file and the bootstrap file, so values you set in /opt/deployment/.env apply.

• Inputs: SERVER_ROLE and CONFIG_BUCKET are required. The bundle reference comes from the first argument or CONFIG_REF. The path prefix is CONFIG_BASE_PATH, default configs.
• Consistency: The bundle manifest points to one immutable generation. fetch-config refuses an invalid pointer instead of mixing files from different publications.
• Integrity: It downloads the role directory and the common directory, checks every file against its SHA-256 checksum and only then replaces the installed files. It keeps the host's .env.
• Record: It stores a copy of the manifest in /opt/deployment/config-state/<role>.manifest.json.
• Retries: fetch-config tries up to FETCH_CONFIG_ATTEMPTS times in total. The wait before each repeat is FETCH_CONFIG_RETRY_DELAY multiplied by the number of the failed attempt, so the defaults wait 5, 10, 15 and 20 seconds.

After the last attempt, it prints Configuration fetch failed after N attempts and init-service stops with hints about credentials, bucket and bundle reference.

To raise the retry budget for a deployment whose bundles publish slowly, add the variables to the bootstrap file:

FETCH_CONFIG_ATTEMPTS=8
FETCH_CONFIG_RETRY_DELAY=10

Every role's init.sh also installs the current fetch-config as /usr/local/bin/fetch-config with fetch-config --install. The provisioning copy and the installed copy honor the same variables.

Run role initialization

init-service runs /opt/services/<role>/init.sh --skip-fetch, because the bundle is already current. Each init.sh loads parameters and secrets into process memory, starts the role's containers and removes any leftover plaintext .env in the service directory.

Option	Roles	Effect
--skip-fetch	All	Uses the installed bundle instead of fetching it again.
--skip-migrations	Ops	Starts Ops without the one-shot migration and baseline seed.
--restart-only	Ops	Skips migrations and restarts the Ops services.

Role-specific behavior to expect:

• Ops runs the one-shot db-migrate container before it starts Scaler and Tasker. If migration or baseline seed fails, it prints refusing to start ops services and stops. Do not skip migrations to get past this on a first installation; see Bootstrap and migration credentials.
• API waits until TelAPI answers /health on TELAPI_PORT.
• TelPro also enables its configuration refresh timer. See Operate the telephony edge.

Retry a failed initialization

Provisioning installs two systemd units per role:

• <role>-init-retry.timer runs initialization 5 minutes after boot and then every 2 minutes. It stops once /opt/deployment/service-ready exists and disables itself after a success.
• <role>-update.service runs at every boot. It fetches the bundle, logs in to the registry and runs the role's update. A reboot therefore picks up the current bundle and images.

To retry by hand after you fix the cause, run:

sudo /usr/local/bin/init-service

Read these files when initialization fails:

File	Contents
/opt/deployment/status	Short summary and pointers to the logs
/opt/deployment/init-progress.log	Progress of the initialization steps
/opt/deployment/init-service.log	Output of init-service and init.sh
/var/log/cloud-init-output.log	Output of the cloud-init phase

Run first-use setup

After the Ops host has initialized and migrations have succeeded, create the first administrator and the platform defaults. Run the setup wrapper on the Ops host:

cd /opt/services/ops
./delphi-setup.sh

The wrapper prepares the same environment as initialization: bootstrap file, proxy, parameters and secrets in memory, TLS files and registry login. It then runs the setup in the one-shot migration container. Secrets stay in memory and are never written to disk.

The interactive setup walks through these steps in order:

1. Create the Super Admin account.
2. Create the default team and add the Super Admin as Team Admin.
3. Assign the base number.
4. Set the default outbound egress route.
5. Create the subscription plans. If subscription management is turned off (FEATURE_SUBSCRIPTION_MANAGEMENT), setup creates one internal unlimited plan instead.
6. Synchronize subscription prices with Stripe. This step runs only with subscription management turned on. Without STRIPE_SECRET_KEY, setup skips it and you run setup again once the key is available.

Wrapper option	Effect
none	Runs the interactive setup.
--apply-missing	Applies pending steps without prompts, using values from the environment.
--no-pull	Skips registry login and the image pull.
--dev-seeds	Adds test and example data. Do not use it in production.
-- <options>	Passes the remaining options to the setup program unchanged.

The setup program also accepts --verify-super-admin-readiness, which checks that at least two usable Super Admin accounts exist. Use it before you hand the deployment over.

For administrator recovery, the setup variables and the --force-step and --reset-password options, see Recover Super Admin access. For an upgrade, apply only the setup steps that the release requires.

Verify the deployment

1. On each host, confirm that /opt/deployment/service-ready exists and that /opt/deployment/status reports completion.
2. Run docker compose ps in /opt/services/<role> and confirm that every container is running and healthy.
3. Run /opt/services/common/show-env.sh --service <role> and confirm that the role's required variables are listed.
4. Open the TelWeb address and sign in as the Super Admin.
5. Run ./delphi-setup.sh -- --verify-super-admin-readiness on the Ops host.
6. Follow the probe checks in Service health for API, Voice and TelPro.
7. Place a test call to a published application and confirm that its conversation appears in TelWeb.

Related

• Deploy, verify and update Delphi: to update images, bundles and configuration after the first start.
• Environment variable reference: to look up a variable that initialization reports as missing.
• Service health: to interpret a probe that fails after initialization.
• Configuration recovery: to capture a recovery point once the deployment works.