SigNoz service operations
The SigNoz service is the centralized observability backend. Every service runs a local OTel collector that forwards telemetry to SigNoz's OTLP endpoints on 10.0.1.10. The UI is exposed publicly via the Web service Caddy reverse proxy on DOMAIN_SIGNOZ.
- Overview
- Runbook
- Configuration
- Troubleshooting
Containers
SigNoz uses the upstream official Docker Compose from the SigNoz repository (cloned to /opt/signoz-repo). init.sh modifies the official compose file to use bind mounts on the attached volume and injects proxy / SMTP variables.
| Container | Purpose |
|---|---|
signoz-otel-collector | OTLP receiver (gRPC :4317, HTTP :4318) |
query-service | Query API + frontend (:8080) |
clickhouse | Time-series store (:9000 native, :8123 HTTP) |
zookeeper-1 | ClickHouse coordination |
alertmanager | Alerts |
frontend | Web UI (proxied via query-service) |
Access
| Method | URL | Notes |
|---|---|---|
| Public | https://${DOMAIN_SIGNOZ} | Through Web instance Caddy |
| Private direct | http://10.0.1.10:8080 | From any instance |
| OTLP gRPC | 10.0.1.10:4317 | Used by every service's OTel collector |
| OTLP HTTP | 10.0.1.10:4318 | Used by every service's OTel collector |
Deploy and update
cd /opt/services/signoz
./init.sh # clones / pulls signoz repo, prepares bind mounts, starts compose
./update.sh --restart-only
Flags include --skip-git (skip the repo update), --skip-images (skip Docker pull), --skip-fetch, --restart-only.
The init script:
- Clones the upstream
https://github.com/SigNoz/signoz.gitto/opt/signoz-repo. - Converts named volumes to bind mounts on
/mnt/signoz-data. chown 101:101 /mnt/signoz-data/clickhouse(ClickHouse UID).- Disables SigNoz Cloud's OpAMP manager (not needed for self-hosted).
- Injects proxy env +
extra_hostsso containers can reach the internet through Squid. - Configures
SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__*ifSMTP_HOSTis set in the bootstrap.env.
Transparent proxy (redsocks + iptables)
SigNoz containers need outbound HTTP / HTTPS (webhooks, email). Many apps don't respect HTTP_PROXY. The init script configures a transparent proxy via redsocks + iptables:
systemctl status redsocks
/usr/local/bin/redsocks-iptables.sh status
/usr/local/bin/redsocks-iptables.sh restart
Volume attachment (fresh deployment)
mkfs.ext4 -F /dev/disk/by-id/scsi-0HC_Volume_XXXXX
mount -o discard,defaults /dev/disk/by-id/scsi-0HC_Volume_XXXXX /mnt/signoz-data
echo '...auto-mount...' >> /etc/fstab
cd /opt/services/signoz && ./init.sh
Retention
ClickHouse TTL controls retention. Defaults are 7 days for logs / traces and 30 days for metrics. To change:
docker compose exec clickhouse clickhouse-client --query "
ALTER TABLE signoz_traces.signoz_index_v2
MODIFY TTL toDateTime(timestamp) + INTERVAL 14 DAY;"
Dashboards and alerts
This operations page covers the SigNoz backend itself. For importing Delphi dashboards, creating alert rules, and using SigNoz during call investigations, use Monitoring in SigNoz.
SigNoz uses the upstream Docker Compose. Variables below are injected by init.sh and do not follow the standard vars.yaml pattern.
| Name | Source | Scope | Default | Description |
|---|---|---|---|---|
SIGNOZ_DATA_DIR | env | all | /mnt/signoz-data | Data directory on attached volume. |
ENVIRONMENT | env | all | — | Environment name (bootstrap). |
HTTP_PROXY | env | all | — | http://<squid-ip>:3128 (private servers). |
HTTPS_PROXY | env | all | — | http://<squid-ip>:3128 (private servers). |
SMTP_HOST | env | all | — | SMTP host (for alert emails). |
SMTP_PORT | env | all | 587 | SMTP port. |
SMTP_USER | env | all | — | SMTP username. |
SMTP_PASS | env | all | — | SMTP password. |
SMTP_FROM | env | all | — | Sender address. |
The init script translates SMTP vars into SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__SMARTHOST, __FROM, __AUTH__USERNAME, and __AUTH__PASSWORD on the query-service container.
| Symptom | Likely cause | Check |
|---|---|---|
| No telemetry arriving | OTel collectors can't reach 10.0.1.10:4317 | Network reachability; SigNoz containers running. |
| UI not loading | query-service or frontend down | docker compose ps in /opt/signoz-repo/deploy/docker/. |
| ClickHouse disk full | Retention too long / volume too small | df -h /mnt/signoz-data; lower TTL or grow signoz_volume_size. |
| Slow queries | ClickHouse overload | ClickHouse container logs; consider a bigger instance. |
| Proxy errors on startup | Squid unreachable | Verify Squid; check init.sh proxy injection output. |
| Email alerts missing | SMTP misconfigured or Squid blocking SMTP | SMTP_HOST set; alertmanager logs. |
See also
- Monitoring in SigNoz — dashboards, alerts, call traces, and query patterns.
- Web operations — Caddy reverse-proxies the SigNoz UI.
- Squid operations — proxy for SigNoz outbound traffic.