Verify security event export and retention

Use this procedure to establish whether a security event reached an approved external destination and how long its separate copies remain available. Access Logs, the audit database, SigNoz and the external destination are different evidence stores. A visible database row does not prove telemetry export succeeded.

Availability: Admin Settings → Access Logs requires platform-wide read access (Super Admin or Super Viewer); team membership alone is insufficient. Collector configuration and destination credentials require deployment/operator access separately. The product has no security-export form or export/download operation in the inspected Access Logs UI and audit API. External forwarding is deployment configuration, not a button on that screen.

1. Find a known event and record the time correctly

For a privileged-role event, use the platform-role workflow to identify its challenge and recorded outcome; do not change a real role just to generate a test record.

Open /admin-settings/access-logs. Choose an approved existing event or a synthetic test from your agreed test process. Do not create repeated failed logins against a real account to test forwarding.

Control	Meaning and limit
Action	Filters a supported action, including login/failure/logout, password/reset/session events, CRUD and privileged-role/recovery events. Blank means no action filter within this feed.
Search	Debounced by 300 ms and trimmed. Searches username plus selected email, actor email, target email, IP and MSISDN fields; it is not a full JSON/body search. Username uses case-insensitive matching; JSON-field filters do not request that same mode.
From / To	Browser-local datetime inputs converted to ISO instants for the API. Bounds are inclusive. Rows display date/time in UTC. Record the conversion when comparing another tool's timezone.
Super Admins only	Uses the role captured at event time, with an additional details.new.role match; it does not look up the actor's current role. Historical rows without the expected fields can be omitted.
Refresh / Load more	Refetches or fetches the next page. The UI requests 100 rows per page; the API permits 1–200, default 50. Loading one page is not an export of all matching events.

A — From and B — To accept browser-local date/time; the result rows show UTC. C — Super Admins only filters recorded event-time role fields. This crop shows only the filters, with no event or personal data. Select the image for full size.

The feed covers categories AUTH, ROLE_MANAGEMENT, SIP_TRUNK, EGRESS_TRUNK, EGRESS_ROUTE and PLATFORM_SECRET. It is not the complete audit store. In particular, the act of reading security logs emits SECURITY_LOGS_ACCESSED but is not automatically included by this category list. Application history is a separate app-scoped query, with team access checks and filtered test-flow activity.

Rows show Date, Time, Type, User, IP, MSISDN, User agent and Details. Empty values do not establish that no actor or address existed: producer fields, redaction and display fallbacks affect them. The audit table stores ID, timestamp, category, action, optional user/username, target/application IDs and JSON details. It has no telemetry-delivery receipt column. Use Reconstruct an audited change to distinguish the actor from the affected resource, interpret before/after objects and follow multi-stage role outcomes.

For read-only API investigation, auditLog.listAccessEvents accepts action, search, superuserOnly, ISO from/to, limit and cursor. Follow nextCursor; results sort by descending timestamp then ID. Keep the same filters when paging. The API does not provide a downloadable encrypted archive, a delivery acknowledgement or an immutable snapshot of a changing dataset. The validator does not enforce that From precedes To; correct an inverted range before diagnosing missing events.

Investigate an access event

For a reported login lockout, open Access Logs with platform-wide read access. Choose the failed-login action, search the affected account, and set the browser-local From/To range around the reported attempt. Read the reason and actor/target fields, then widen to all actions to find related lock, reset or session events. Missing rows are not proof that no event occurred: check the time conversion, filters, loaded pages and producer/persistence path.

Use account recovery for an authorized reset or unlock; this read-only feed does not unlock an account. Do not generate failed logins against a real user to test the workflow. For a role incident, use the audited-change sequence and role-administration procedure; a started challenge is not completed authorization.

For SUPER_ADMIN_RECOVERY_ELEVATED or SUPER_ADMIN_RECOVERY_DENIED, use the exceptional recovery verification. These setup-operator events differ from the ordinary in-app role challenge; correlate the asserted operator identity, change reference, target and outcome.

2. Check the external feed separately

On a small screen, scroll the diagram horizontally if needed to read every branch.

This is a live-ingest fan-out, not a query that backfills historical audit rows or exports all old SigNoz records. Verify the producer and telemetry path first using logs and diagnostics. Database persistence and telemetry may fail independently.

1. Identify the actual SigNoz host and deployed collector configuration. Confirm its source/release identity through release compatibility; a source template does not prove that configuration is running.
2. Check OTEL_EXPORT_BACKEND_HOST. Its default is empty, which omits the ingester's export branch. OTEL_EXPORT_BACKEND_PORT defaults to 4327. The rendering path adds logs/export only for a nonempty host.
3. Check the export collector's effective rendered configuration at /opt/deployment/otel-collector-export.rendered.yaml, using an approved read-only inspection that does not expose credentials. A healthy process can be in sink mode, accepting and discarding logs.
4. Verify the intended destination endpoint/token pair and active exporter list. The supplied options use DYNATRACE_OTLP_ENDPOINT/DYNATRACE_API_TOKEN and SIEM_HEC_ENDPOINT/SIEM_HEC_TOKEN, with optional SIEM channel/index settings. Keep values in the deployment's secret/configuration system; do not paste them into support evidence.
5. Reconcile the intended source event and time against ingester, export collector and receiving-system evidence. Confirm received fields and duplication behavior, not merely HTTP acceptance or collector health.

The initializer uses sink mode when neither complete endpoint/token pair exists. If either pair exists, it copies the full template, which still lists both destination exporters. It does not automatically remove an unconfigured second destination. Check the rendered pipeline and collector configuration validation before assuming that supplying one pair creates a valid single-destination setup.

The exporter labelled otlphttp/splunk_hec is implemented with the OTLP HTTP exporter and custom headers. Its label and environment names do not establish compatibility with a native HEC ingestion endpoint or an arbitrary SIEM. Verify the destination's actual accepted protocol with the integration owner; no universal external endpoint is specified here.

The service initialization path renders and starts the collector; simply editing an environment source does not verify activation. Follow the deployment's reviewed refresh procedure in deployment, then recheck effective configuration and receipt. No generic command here promises a safe restart for every topology.

3. Verify selection, protection and delivery limits

The export classifier accepts explicit security_event: true and recognizes selected security/audit attributes, numeric HTTP status codes 401, 403 and 419, plus body-pattern heuristics. It then drops records not marked as security events. These rules are not identical to the Access Logs category filter, and text heuristics can miss relevant events or include unrelated matches. Inspect actual normalized attributes/body when an event is missing; do not infer completeness from an action name alone.

The pipeline adds source/platform tags and applies its configured PII transform. ENABLE_PII_LOGGING defaults to false; setting it to literal true bypasses these collector PII replacements. MSISDN_UNMASKED_DIGITS defaults to 4. The inspected transform masks selected body patterns and telephone attributes, not every possible nested JSON value. Shared logger redaction is an earlier, separate layer; native or third-party producers need their own review. Redaction is not a guarantee that a record is anonymous or safe for an unrestricted destination.

The ingester-to-export branch sets tls.insecure: true; the collector's OTLP receivers have no TLS/authentication configuration in this template. External transport protection depends on the configured endpoint and deployment. This path is not an encrypted export package or a guarantee of encrypted transport on every hop. Network access, destination authorization and storage encryption must be verified separately.

The ingester branch configures a queue size of 500 and retries for up to 60 seconds. Destination exporters configure queue size 1000 and retries for up to 300 seconds; no persistent queue storage extension is configured here. These are bounded delivery settings, not durable replay or exactly-once guarantees. A process restart, exhausted retry window, queue pressure or receiver rejection requires investigation; there is no general audit replay button. See background jobs for the separate database-job mechanism rather than treating it as the telemetry queue.

The local health endpoint at http://127.0.0.1:13134/ establishes collector responsiveness only. Sink mode can pass it. Inspect bounded collector errors and destination receipt for the same approved event before reporting export as working.

4. Establish retention for each copy

Copy	Source-backed boundary
PostgreSQL audit rows	AUDIT_LOG_CLEANUP deletes rows older than its computed cutoff when the job executes. AUDIT_LOG_RETENTION_DAYS overrides payload retentionDays; otherwise the default is 180 days. Parsed values below 1 or nonnumeric values fail. The implementation uses parseInt, so this is not strict whole-string numeric validation.
SigNoz records	Separate storage/retention policy. The audit cleanup setting does not set SigNoz retention or delete its copies. Inspect the actual deployment policy.
External destination	Separate ingestion, access, retention and deletion controls. No inherited Delphi 180-day policy is established by forwarding.
Backups or manually retained evidence	Separate lifecycle and access obligations; deleting a live audit row is not proof all retained copies disappeared.

The cleanup cutoff is current time minus the configured day count, and the delete comparison is strictly earlier than that cutoff. There is no category, tenant or legal-hold exclusion in this handler. Do not interpret the default as a guaranteed retention SLA or run cleanup as a verification step. Check the actual enabled schedule, effective environment/payload, last job result and approved policy using background jobs.

No tamper-evident signing, Object Lock, immutable archive or compliance certification is established by these query, forwarding and cleanup implementations. If such preservation is required, resolve its storage/control design before relying on the feed as an immutable evidence archive.

Check the outcome and recover

Record the selected source/release, event/time and timezone, applicable filter, active destination, observed receipt and separate retention owners. Exclude credentials and unnecessary personal data from the handoff. If Access Logs has the row but SigNoz does not, investigate producer transport/collection; if SigNoz has it but the destination does not, inspect branch enablement, sink mode, classification, exporter validity and receiver response. Preserve evidence before changing filters or retrying a test. A documentation update or a passing source-level test is not live ingestion acceptance.