SigNoz service operations

The SigNoz service is the centralized observability backend. Every service runs a local OTel collector that forwards telemetry to SigNoz OTLP endpoints on 10.0.1.10. The UI is exposed via the Web service Caddy reverse proxy on DOMAIN_SIGNOZ.

Current releases use SigNoz Foundry (foundryctl forge / cast). The old /opt/signoz-repo + deploy/docker layout is legacy. You can do every step on this page from the SigNoz host after jumping through the bastion — you do not need the Delphi git repo.

Overview

Instance

Field	Value
Network	Private only — static private IP 10.0.1.10
Scalable	No — single instance

Sizing and retention by environment

Baseline instance type (4 vCPU / 8 GB RAM) is in Provision instances. Delphi staging runs at that size with a 100 GB attached volume and SigNoz's default ClickHouse TTLs (7 days for logs and traces, 30 days for metrics). That is enough for day-to-day platform work but is not a production target — production should keep telemetry longer for post-incident review, plan more disk, and may need a larger instance under heavier query load.

	Staging (reference)	Production (typical)
Attached volume	100 GB at /mnt/signoz-data	150 GB or more
Log / trace retention	7 days (defaults)	14–30 days
Metric retention	30 days (defaults)	30 days (often sufficient)

Disk grows with retention

ClickHouse stores logs and traces on /mnt/signoz-data. Longer TTLs and higher call volume both increase disk use — extend retention and volume size together. Before raising TTLs in production, confirm headroom with df -h /mnt/signoz-data and watch the storage alerts in Monitoring in SigNoz.

Containers

SigNoz is generated by Foundry into /opt/services/signoz/pours/deployment/compose.yaml. init.sh rewrites data volumes to bind mounts under ${SIGNOZ_DATA_DIR} (default /mnt/signoz-data).

Container	Port	Purpose
ingester	4317 / 4318 / 13133	OTLP ingest (signoz/signoz-otel-collector)
signoz-signoz-0	8080	UI + API (SQLite metastore)
signoz-telemetrystore-clickhouse-0-0	9000 / 8123	ClickHouse
signoz-telemetrykeeper-zookeeper-0	2181	ZooKeeper
voiceai-otel-collector-export	4327 / 13134	Optional external export fan-out

The per-host otel/opentelemetry-collector-contrib sidecar on other services is a different image. Do not bump it as part of a SigNoz UI/API upgrade.

Access

Method	URL	Notes
Public	https://${DOMAIN_SIGNOZ}	Through Web instance Caddy
Private direct	http://10.0.1.10:8080	From any instance
OTLP gRPC	10.0.1.10:4317	Used by every service's OTel collector
OTLP HTTP	10.0.1.10:4318	Used by every service's OTel collector

hostname -I on the SigNoz host may list a Tailscale IP first (100.64.x). Platform OTLP stays on 10.0.1.10.

Which stack is this host on?

# Foundry (current)
test -f /opt/services/signoz/pours/deployment/compose.yaml && echo foundry

# Legacy (needs migration)
test -d /opt/signoz-repo/deploy/docker && echo legacy

Use the Upgrade tab for a version bump, a legacy → Foundry migrate, or alert-rule sync after an upgrade.

Runbook

Deploy and update

All commands run on the SigNoz host (10.0.1.10) after ssh through the environment bastion.

ssh <bastion>
ssh <signoz-host>          # or: ssh -i /root/.ssh/bastion_key root@10.0.1.10

cd /opt/services/signoz
./init.sh                  # fetch config, render Foundry casting, pull images, start
./update.sh --restart-only # re-read SSM / Secrets Manager; does not pull a new SigNoz version

init.sh:

• Loads /opt/deployment/.env, then SSM / Secrets Manager via fetch-env.sh.
• Installs or refreshes foundryctl, renders casting.yaml, runs foundryctl cast.
• Converts named volumes to bind mounts on /mnt/signoz-data.
• chown 101:101 /mnt/signoz-data/clickhouse (ClickHouse UID).
• Injects proxy env + extra_hosts so containers reach the internet through Squid.
• Configures SMTP on the query-service container when SMTP_HOST is set.
• Applies git-tracked alert JSON via apply-alert-rules.sh when SIGNOZ_API_KEY is set.

Flags: --skip-fetch, --skip-images, --restart-only, --migrate-from-legacy.

Shared update.sh flags: --signoz-version, --config-ref, --restart-only, --dry-run.

Do not use --restart-only for a version bump

--restart-only skips image pulls. Use ./update.sh --signoz-version <x.y.z> (Upgrade tab).

Volume attachment (fresh deployment)

mkfs.ext4 -F /dev/disk/by-id/scsi-0HC_Volume_XXXXX
mount -o discard,defaults /dev/disk/by-id/scsi-0HC_Volume_XXXXX /mnt/signoz-data
# add the same device to /etc/fstab

cd /opt/services/signoz && ./init.sh
./verify-signoz-usable.sh

Transparent proxy (redsocks + iptables)

SigNoz containers need outbound HTTP / HTTPS / SMTP (webhooks, email). Many paths ignore HTTP_PROXY. When HTTP_PROXY is set, init.sh installs redsocks and redirects SIGNOZ_TRANSPARENT_PROXY_PORTS (default 80,443,25,465,587):

systemctl status redsocks
/usr/local/bin/redsocks-iptables.sh status
/usr/local/bin/redsocks-iptables.sh restart

Retention

ClickHouse table TTLs control how long telemetry is kept. SigNoz ships with 7 days for logs and traces and 30 days for metrics.

For production, 14–30 days of logs and traces is a common starting point. Grow /mnt/signoz-data before raising TTLs.

docker compose -f /opt/services/signoz/pours/deployment/compose.yaml \
  exec signoz-telemetrystore-clickhouse-0-0 clickhouse-client --query "
  ALTER TABLE signoz_traces.signoz_index_v2
  MODIFY TTL toDateTime(timestamp) + INTERVAL 14 DAY;"

Apply similar changes to the other trace and log tables for your SigNoz version. Metrics tables usually keep the 30-day default.

Dashboards and alerts

Import Delphi dashboards from /opt/services/signoz/dashboards/ in the SigNoz UI (Dashboards → Import). Using SigNoz during a call investigation is covered in Monitoring in SigNoz. Applying the shipped alert JSON after a deploy or upgrade is on the Upgrade tab.

Verify the stack

UI-up is not enough. ClickHouse must accept writes:

cd /opt/services/signoz
./verify-signoz-usable.sh
curl -s http://127.0.0.1:8080/api/v1/version
docker compose -f /opt/services/signoz/pours/deployment/compose.yaml ps

verify-signoz-usable.sh exits 0 only if an OTLP probe lands in ClickHouse. Use --repair if replicas are readonly after a migrate.

Upgrade

What this tab covers

Use this on a live host when you need to:

1. Bump signoz/signoz (for example 0.133.0 → 0.139.0).
2. Move a legacy /opt/signoz-repo install onto Foundry.
3. Re-apply Delphi alert JSON after an upgrade (this is what used to fail with HTTP 401).

Pinned versions live in Third-party version sources. The live pin on a host is SIGNOZ_VERSION in /opt/deployment/.env (wins) or SSM /${NAMESPACE}/signoz/SIGNOZ_VERSION.

1. Upgrade SigNoz in place (Foundry already installed)

Do not pass --restart-only. That skips the image pull.

ssh <bastion>
ssh <signoz-host>

# Optional: refresh foundryctl so forge/cast defaults match the target release
curl -fsSL https://signoz.io/foundry.sh | bash
export PATH="${HOME}/.local/bin:/usr/local/bin:${PATH}"

# Backup the SQLite metastore (users, dashboards, alert rules) before schema hops
cp -a /mnt/signoz-data/sqlite "/mnt/signoz-data/sqlite.bak.$(date +%Y%m%d-%H%M)"

cd /opt/services/signoz
./update.sh --signoz-version 0.139.0
./verify-signoz-usable.sh
curl -s http://127.0.0.1:8080/api/v1/version

update.sh --signoz-version writes /opt/deployment/.env, tries to update SSM, then runs init.sh (re-render casting + foundryctl cast + pull signoz/signoz:v0.139.0).

The deploy IAM user on the host often cannot ssm:PutParameter. That is OK: init.sh prefers /opt/deployment/.env over SSM for image pins. Ask someone with AWS console/CLI rights to overwrite SSM later so the next rebuild does not surprise you.

init.sh also re-fetches CONFIG_REF from S3. The export-collector compose pin reverts to whatever that prefix currently has. Re-check docker ps --filter name=otel-collector-export after a bump.

Upstream stops between 0.133 and 0.139

SigNoz documents 0.135 (dashboards v2) and 0.137 (saved-view repair; unrepairable views are deleted). Direct jumps 0.133.0 → 0.138.0 → 0.139.0 succeeded on Delphi staging (SQLite reported no new migrations to run; existing rules and a user dashboard stayed). Still back up /mnt/signoz-data/sqlite first. v0.139.0 deprecates the v1 alert-history endpoints (Delphi does not call them).

Leave these pins alone

Pin	Typical default	Why
SIGNOZ_OTELCOL_TAG	0.142.0 (signoz/signoz-otel-collector)	Independent of signoz/signoz. Newer tags may need ClickHouse 25.12.
CLICKHOUSE_IMAGE	clickhouse/clickhouse-server:25.5.6	Do not bump as a side effect of a UI/API upgrade.

After the upgrade, confirm Settings / GET /api/v1/version matches the target, then spot-check dashboards, saved views, and that OTLP still lands (./verify-signoz-usable.sh). Then run Alert-rule sync below.

2. Migrate from legacy (/opt/signoz-repo) to Foundry

Official upstream notes: SigNoz deploy/MIGRATION.md.

# Optional same-window version bump
echo 'SIGNOZ_VERSION=0.139.0' >> /opt/deployment/.env

cp -a /mnt/signoz-data/sqlite "/mnt/signoz-data/sqlite.bak.$(date +%Y%m%d-%H%M)"
cd /opt/services/signoz
./init.sh --migrate-from-legacy
./verify-signoz-usable.sh

init.sh refuses to write /opt/deployment/signoz-foundry-migrated until the ClickHouse write probe passes.

If replicas are readonly after migrate, the usual cause is a ClickHouse replica macro mismatch (legacy default vs Foundry default). Set SIGNOZ_CLICKHOUSE_REPLICA / SIGNOZ_CLICKHOUSE_SHARD in /opt/deployment/.env, re-run ./init.sh --restart-only, then ./verify-signoz-usable.sh --repair.

Rollback to legacy (only if the migrate flag file is not written, or you still have the backup)

cd /opt/services/signoz/pours/deployment && docker compose down   # no -v
BACKUP=$(ls -td /opt/deployment/signoz-legacy-backup-* | head -1)
cd "$BACKUP/deploy-docker" && docker compose up -d
rm -f /opt/deployment/signoz-foundry-migrated

3. Alert-rule sync (required after 0.139)

init.sh calls /opt/services/signoz/apply-alert-rules.sh. That helper POSTs / PUTs JSON from /opt/services/signoz/alerts/ to http://127.0.0.1:8080/api/v1/rules.

Three things must be true or the sync is a no-op / 400:

Requirement	What goes wrong without it	What to do on the host
Editor or Admin API key	Missing key → script skips. Viewer key → GET 200, POST/PUT 403. Unauthenticated curl used to look like 401000.	SigNoz UI → Settings → Service Accounts → create an Editor (or Admin) key.
v5 rule JSON	0.139 rejects version: "v4" and the old builderQueries map (alert rule definition is not valid).	Use the alerts/*.json that ship with current config (version: "v5", condition.compositeQuery.queries).
A notification channel	0.139 returns at least one channel is required when preferredChannels is empty.	Create Error / Teams (or any channel) in the UI. The helper fills those names when they exist.

Store the key (do not print it)

1. Create the Editor service-account key in the SigNoz UI.
2. Add SIGNOZ_API_KEY to Secrets Manager secret ${NAMESPACE}/signoz/secrets (JSON object). Terraform ignores drift on that secret, so this is a console / CLI edit, not a Terraform apply.
3. Re-load env and apply without restarting SigNoz:

cd /opt/services/signoz
# fetch-env.sh puts SIGNOZ_API_KEY in the process environment; do not echo it
set -a
. /opt/services/common/fetch-env.sh
set +a
./apply-alert-rules.sh

Optional override: SIGNOZ_ALERT_CHANNELS=Error,Teams if the auto-detected names are wrong.

Re-sync is idempotent (match by alert name, POST new / PUT existing). User-created rules that are not in alerts/*.json are left alone.

Shipped starter rules:

File	Alert	What it watches
call-error-rate-spike.json	CallErrorRateSpike	Voice pipeline error rate (PromQL)
conversation-runtime-crash.json	ConversationRuntimeCrash	TelPhi fatal runtime error logs
sip-trunk-down.json	SipTrunkDown	Failed kamailio.trunk_probe spans
stt-tts-provider-failure.json	SttTtsProviderFailure	Failed STT/TTS spans
telphi-no-traces.json	TelphiNoTraces	No TelPhi traces (process / collector down)

4. Per-host contrib collector (not this host's SigNoz bump)

Bumping otel/opentelemetry-collector-contrib is a compose / CONFIG_REF rollout on each service host, not ./update.sh --signoz-version. On each host:

cd /opt/services/<role>
./update.sh --config-ref <current-ref>    # not --restart-only; must pull
docker ps --filter name=otel-collector --format '{{.Names}} {{.Image}}'

Always let update.sh run fetch-env.sh. A bare docker compose up otel-collector can leave POSTGRES_MONITORING_PASSWORD empty; contrib 0.159.0 fails closed (receivers::postgresql: missing password).

The Foundry ingester on the SigNoz host stays on signoz/signoz-otel-collector (SIGNOZ_OTELCOL_TAG).

Configuration

Declared in the SigNoz vars.yaml and loaded by fetch-env.sh. Inspect on the host with common/show-env.sh --service signoz --check-aws (it will not print secret values).

/opt/deployment/.env wins over SSM for SIGNOZ_VERSION, SIGNOZ_OTELCOL_TAG, and CLICKHOUSE_IMAGE. Prefer ./update.sh --signoz-version so both are written together.

Name	Source	Scope	Default	Description
SIGNOZ_DATA_DIR	env	all	/mnt/signoz-data	Data directory on the attached volume.
SIGNOZ_VERSION	SSM	all	0.139.0	signoz/signoz:v<version>. Override in /opt/deployment/.env or ./update.sh --signoz-version.
SIGNOZ_OTELCOL_TAG	SSM	all	0.142.0	Foundry ingester image tag (signoz/signoz-otel-collector). Independent of SIGNOZ_VERSION.
CLICKHOUSE_IMAGE	SSM	all	clickhouse/clickhouse-server:25.5.6	ClickHouse image. Do not bump during a UI/API-only upgrade.
SIGNOZ_API_KEY	Secrets Manager	all	—	Editor or Admin service-account key for apply-alert-rules.sh (header SIGNOZ-API-KEY).
SMTP_HOST	SSM	all	—	SMTP server for alert emails.
SMTP_PORT	SSM	all	587	SMTP port.
SMTP_USER	Secrets Manager	all	—	SMTP username.
SMTP_PASS	Secrets Manager	all	—	SMTP password.
SMTP_FROM	Secrets Manager	all	—	Sender address.
HTTP_PROXY	env	all	—	http://<squid-ip>:3128 (private servers). Enables redsocks transparent proxy.
HTTPS_PROXY	env	all	—	http://<squid-ip>:3128 (private servers).
OTEL_EXPORT_BACKEND_HOST	SSM	all	—	When set (usually 10.0.1.10), Foundry ingester fans logs out to the export collector.

The init script maps SMTP vars into SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__SMARTHOST, __FROM, __AUTH__USERNAME, and __AUTH__PASSWORD on the query-service container.

Troubleshooting

Symptom	Likely cause	Check
No telemetry arriving	Collectors cannot reach 10.0.1.10:4317	Network; docker compose ps in pours/deployment/.
UI not loading	signoz-signoz-0 down	docker compose -f /opt/services/signoz/pours/deployment/compose.yaml ps
ClickHouse disk full	Retention too long / volume too small	df -h /mnt/signoz-data; lower TTL or grow the volume.
Slow queries	ClickHouse overload	ClickHouse logs; consider a bigger instance.
Proxy errors on startup	Squid unreachable	Squid up; init.sh proxy injection output.
Email alerts missing	SMTP missing or Squid blocking SMTP	SMTP_HOST; redsocks ports include 25/465/587.
Alert sync skipped	SIGNOZ_API_KEY unset	Add an Editor key to ${NAMESPACE}/signoz/secrets, then ./apply-alert-rules.sh.
Alert sync 403	Key is Viewer, not Editor/Admin	Create a new Editor service account in the SigNoz UI.
Alert sync 400 “definition is not valid”	Rule JSON still v4	Need version: "v5" and compositeQuery.queries (current alerts/*.json).
Alert sync 400 “at least one channel”	No notification channel	Create a channel in the UI; re-run the helper.
Readonly ClickHouse replicas after migrate	Replica macro mismatch	./verify-signoz-usable.sh --repair; set SIGNOZ_CLICKHOUSE_REPLICA if it persists.
Export collector pin reset after upgrade	init.sh re-fetched CONFIG_REF from S3	Check the export collector image; re-apply the intended compose pin.
Database host collector crash-loops on 0.159	POSTGRES_MONITORING_PASSWORD empty	Recreate via ./update.sh (runs fetch-env.sh); do not docker compose up bare.

See also

• Monitoring in SigNoz — dashboards, alerts, call traces, and query patterns.
• Web operations — Caddy reverse-proxies the SigNoz UI.
• Squid operations — proxy for SigNoz outbound traffic.
• Third-party version sources — current SIGNOZ_VERSION and collector pins.
• Bootstrap and init — shared init.sh / update.sh / fetch-env.sh flow.

External

• SigNoz docs
• Upgrade to 0.135
• Upgrade to 0.137
• Service accounts