Environment resolution

platform v0.9.11verified 2026-05-14

fetch-env.sh is the script every service runs to assemble its runtime environment. It reads a vars.yaml manifest, fetches values from AWS, and prints export VAR=value lines that init.sh evaluates into the shell that runs docker compose up.

ENV_EXPORTS="$(common/fetch-env.sh \
    --manifest /opt/services/api/vars.yaml \
    --bootstrap /opt/deployment/.env \
    --format export)"
eval "$ENV_EXPORTS"

The two formats:

• --format export — print export VAR=value to stdout (no file written, secrets stay in process memory). This is what init.sh uses.
• --format file --output <path> — write VAR=value lines to a file. Discouraged in production; useful for diffing what the host would resolve.

Resolution order

For each variable declared in the manifest, later sources override earlier ones in this order:

1. Bootstrap / source: local — read from /opt/deployment/.env first.
2. AWS SSM Parameter Store — every parameter under /${NAMESPACE}/${SERVICE}/ is pulled (aws ssm get-parameters-by-path --recursive --with-decryption). Parameter name basename becomes the variable name.
3. AWS Secrets Manager auto-discovery — every secret whose name starts with ${NAMESPACE}/${SERVICE}/ is fetched, parsed as JSON, and its keys merged.
4. External SM secret (secret.arn_from) — for variables flagged with secret, fetch the secret named by the bootstrap variable in arn_from, parse JSON if type: json, pick the key.
5. Manifest default (default:) — used only if every source above produced nothing.
6. Composition (compose.template) — runs in a second pass over the env that's been resolved so far. Variables flagged compose: are skipped in step 1–5 and built last.

The script is "last write wins" within steps 1–4 — if the same variable name appears in both SSM and SM, SM wins. In practice you should never split a single variable across both stores; pick one based on sensitive.

Namespace conventions

The NAMESPACE (e.g. voiceai/staging) plus the manifest's service field decide the AWS paths:

Source	Path pattern	Example
SSM Parameter	/${NAMESPACE}/${SERVICE}/<VAR_NAME>	/voiceai/staging/api/REDIS_HOST
Secrets Manager	${NAMESPACE}/${SERVICE}/* (auto-discovered)	voiceai/staging/api/secrets

A single SM secret named ${NAMESPACE}/<service>/secrets containing one JSON object with every sensitive key is the default and easiest to audit. Splitting into multiple per-purpose secrets is supported — fetch-env.sh discovers and merges all of them.

For paginated SSM responses, fetch-env.sh walks NextToken until exhausted; for SM it lists all secrets under the prefix and fetches each.

Secrets Manager discovery by tag (optional)

By default fetch-env.sh discovers SM secrets by name prefix only. Set SM_DISCOVERY=name,tag (and optionally SM_TAG_KEY=delphi:service) on the host to also discover secrets tagged with that key. The default tag-key value is delphi:service and the expected value is ${NAMESPACE}/${SERVICE}.

This is useful when secrets must live outside the deployment's namespace prefix (different account, different naming scheme) but still belong to a service for retrieval.

Reading the script's stderr

fetch-env.sh writes one line per source to stderr:

  Manifest: /opt/services/api/vars.yaml
  Service:  api
  Namespace: voiceai/staging
  Collecting bootstrap variables...
  Fetching SSM: /voiceai/staging/api/
  Discovering SM secrets by name: voiceai/staging/api/*
  Found 1 secret(s) total
  Resolving secret.arn_from variables...
  Building .env from manifest...
  Composed DATABASE_URL from template
  Validating required variables...
  Exported 47 variables to process environment

docker compose up runs after this. If validation fails, the script exits non-zero with a list of missing variables — init.sh aborts before any container starts.

The precedence gotcha

Variables without source: local are fetched from SSM / SM even if they exist in the bootstrap .env. The eval "$ENV_EXPORTS" in init.sh overwrites any previously sourced bootstrap values.

To ensure a variable always comes from the bootstrap, mark it source: local in vars.yaml. The most common bug here is NAMESPACE itself: if you set it in /opt/deployment/.env for cloud-init purposes but also put a copy in SSM for some reason, the SSM value will silently take effect after fetch-env.sh runs and may not match the path the script just used to fetch the SSM tree. Always treat NAMESPACE, ENVIRONMENT, HOSTNAME, ECR_*, CONFIG_*, and AWS credentials as bootstrap-only.

What "live in memory" means

The eval in init.sh injects every resolved variable into the current shell. docker compose up inherits that shell's environment and substitutes ${VAR} references in docker-compose.yaml. The variables are then passed into each container per the environment: block.

The values never land in a file under /opt/services/<service>/. init.sh actively shreds any leftover .env file as a safety net so manual operator debugging can't accidentally leak secrets to disk.

If you need to inspect what would resolve, re-run fetch-env.sh --format export interactively — but be aware that piping to a file or tee will defeat the in-memory guarantee.

Adding a new variable to a service

1. Declare it in the service's vars.yaml (see vars.yaml schema).

2. Reference it in docker-compose.yaml (e.g. - MY_VAR=${MY_VAR} under the right service's environment: block).

3. Create the matching SSM parameter or Secrets Manager key with your seed script, AWS CLI, console, or provisioning workflow.

4. Set the value:

   # Non-sensitive (SSM)
   aws ssm put-parameter \
     --name "/voiceai/staging/api/MY_VAR" \
     --value "..." --type String --overwrite
   
   # Sensitive (Secrets Manager) — replace the JSON in the secret with one that contains the new key:
   aws secretsmanager put-secret-value \
     --secret-id "voiceai/staging/api/secrets" \
     --secret-string '{"DATABASE_URL":"…", "REDIS_PASSWORD":"…", "MY_VAR":"…"}'

5. Re-resolve env on the affected service:

   cd /opt/services/api && ./update.sh --restart-only

Inspecting what would resolve

# What does this host think it has?
sudo /opt/services/common/fetch-env.sh \
  --manifest /opt/services/api/vars.yaml \
  --bootstrap /opt/deployment/.env \
  --format export | wc -l

# Which variables does each service expect?
/opt/services/common/show-env.sh --service api
/opt/services/common/show-env.sh --all

show-env.sh is the structured-output sibling of fetch-env.sh: it reads every vars.yaml in the bundle and prints what each service expects. Use it to diff what a service wants against what the AWS account has.

See also

• vars.yaml schema
• init.sh and update.sh
• Managed database secrets