SigNoz service operations

The SigNoz service is the centralized observability backend. Every service runs a local OTel collector that forwards telemetry to SigNoz's OTLP endpoints on 10.0.1.10. The UI is exposed publicly via the Web service Caddy reverse proxy on DOMAIN_SIGNOZ.

Overview

Instance

Field	Value
Network	Private only — static private IP 10.0.1.10
Scalable	No — single instance

Sizing and retention by environment

Baseline instance type (4 vCPU / 8 GB RAM) is in Provision instances. Delphi staging runs at that size with a 100 GB attached volume and SigNoz's default ClickHouse TTLs (7 days for logs and traces, 30 days for metrics). That is enough for day-to-day platform work but is not a production target — production should keep telemetry longer for post-incident review, plan more disk, and may need a larger instance under heavier query load.

	Staging (reference)	Production (typical)
Attached volume	100 GB at /mnt/signoz-data	150 GB or more
Log / trace retention	7 days (defaults)	14–30 days
Metric retention	30 days (defaults)	30 days (often sufficient)

Disk grows with retention

ClickHouse stores logs and traces on /mnt/signoz-data. Longer TTLs and higher call volume both increase disk use — extend retention and volume size together. Before raising TTLs in production, confirm headroom with df -h /mnt/signoz-data and watch the storage alerts in Monitoring in SigNoz.

When to add CPU and RAM

Production may need 4–8 vCPU / 8–16 GB RAM when many operators query traces at once, dashboards scan wide time ranges, or ClickHouse merge work falls behind during peak call traffic. If queries time out or the ClickHouse container is memory-bound while disk usage is healthy, scale the instance before lengthening retention further.

Containers

SigNoz uses the upstream official Docker Compose from the SigNoz repository (cloned to /opt/signoz-repo). init.sh modifies the official compose file to use bind mounts on the attached volume and injects proxy / SMTP variables.

Container	Purpose
signoz-otel-collector	OTLP receiver (gRPC :4317, HTTP :4318)
query-service	Query API + frontend (:8080)
clickhouse	Time-series store (:9000 native, :8123 HTTP)
zookeeper-1	ClickHouse coordination
alertmanager	Alerts
frontend	Web UI (proxied via query-service)

Access

Method	URL	Notes
Public	https://${DOMAIN_SIGNOZ}	Through Web instance Caddy
Private direct	http://10.0.1.10:8080	From any instance
OTLP gRPC	10.0.1.10:4317	Used by every service's OTel collector
OTLP HTTP	10.0.1.10:4318	Used by every service's OTel collector

Runbook

Deploy and update

cd /opt/services/signoz
./init.sh                       # clones / pulls signoz repo, prepares bind mounts, starts compose
./update.sh --restart-only

Flags include --skip-git (skip the repo update), --skip-images (skip Docker pull), --skip-fetch, --restart-only.

The init script:

• Clones the upstream https://github.com/SigNoz/signoz.git to /opt/signoz-repo.
• Converts named volumes to bind mounts on /mnt/signoz-data.
• chown 101:101 /mnt/signoz-data/clickhouse (ClickHouse UID).
• Disables SigNoz Cloud's OpAMP manager (not needed for self-hosted).
• Injects proxy env + extra_hosts so containers can reach the internet through Squid.
• Configures SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__* if SMTP_HOST is set in the bootstrap .env.

Transparent proxy (redsocks + iptables)

SigNoz containers need outbound HTTP / HTTPS (webhooks, email). Many apps don't respect HTTP_PROXY. The init script configures a transparent proxy via redsocks + iptables:

systemctl status redsocks
/usr/local/bin/redsocks-iptables.sh status
/usr/local/bin/redsocks-iptables.sh restart

Volume attachment (fresh deployment)

mkfs.ext4 -F /dev/disk/by-id/scsi-0HC_Volume_XXXXX
mount -o discard,defaults /dev/disk/by-id/scsi-0HC_Volume_XXXXX /mnt/signoz-data
echo '...auto-mount...' >> /etc/fstab

cd /opt/services/signoz && ./init.sh

Retention

ClickHouse table TTLs control how long telemetry is kept. SigNoz ships with 7 days for logs and traces and 30 days for metrics — this is what Delphi staging runs today.

For production, 14–30 days of logs and traces is a common starting point so operators can revisit calls and incidents after the fact (see Monitoring in SigNoz). Each extension needs enough free space on /mnt/signoz-data; grow signoz_volume_size at provision time or resize the attached volume before changing TTLs.

To extend log/trace retention, run ALTER TABLE ... MODIFY TTL on the relevant ClickHouse tables. Example — extend trace index retention to 14 days:

docker compose -f /opt/signoz-repo/deploy/docker/docker-compose.yaml exec clickhouse clickhouse-client --query "
  ALTER TABLE signoz_traces.signoz_index_v2
  MODIFY TTL toDateTime(timestamp) + INTERVAL 14 DAY;"

Apply similar changes to the other trace and log tables SigNoz uses for your version; check upstream docs if table names differ after an upgrade. Metrics tables usually keep the default 30-day TTL unless you have a specific compliance requirement.

Dashboards and alerts

This operations page covers the SigNoz backend itself. For importing Delphi dashboards, creating alert rules, and using SigNoz during call investigations, use Monitoring in SigNoz.

Configuration

SigNoz uses the upstream Docker Compose. Variables below are injected by init.sh and do not follow the standard vars.yaml pattern.

Name	Source	Scope	Default	Description
SIGNOZ_DATA_DIR	env	all	/mnt/signoz-data	Data directory on attached volume.
ENVIRONMENT	env	all	—	Environment name (bootstrap).
HTTP_PROXY	env	all	—	http://<squid-ip>:3128 (private servers).
HTTPS_PROXY	env	all	—	http://<squid-ip>:3128 (private servers).
SMTP_HOST	env	all	—	SMTP host (for alert emails).
SMTP_PORT	env	all	587	SMTP port.
SMTP_USER	env	all	—	SMTP username.
SMTP_PASS	env	all	—	SMTP password.
SMTP_FROM	env	all	—	Sender address.

The init script translates SMTP vars into SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__SMARTHOST, __FROM, __AUTH__USERNAME, and __AUTH__PASSWORD on the query-service container.

Troubleshooting

Symptom	Likely cause	Check
No telemetry arriving	OTel collectors can't reach 10.0.1.10:4317	Network reachability; SigNoz containers running.
UI not loading	query-service or frontend down	docker compose ps in /opt/signoz-repo/deploy/docker/.
ClickHouse disk full	Retention too long / volume too small	df -h /mnt/signoz-data; lower TTL or grow signoz_volume_size.
Slow queries	ClickHouse overload	ClickHouse container logs; consider a bigger instance.
Proxy errors on startup	Squid unreachable	Verify Squid; check init.sh proxy injection output.
Email alerts missing	SMTP misconfigured or Squid blocking SMTP	SMTP_HOST set; alertmanager logs.

See also

• Monitoring in SigNoz — dashboards, alerts, call traces, and query patterns.
• Web operations — Caddy reverse-proxies the SigNoz UI.
• Squid operations — proxy for SigNoz outbound traffic.

External

• SigNoz docs