Signal Forge ADR-012: Isolate observability tenants by environment

Status: Accepted

Context

Signal Forge promotes the same immutable release through DEV, QA, and PROD. Mimir rule groups and Alertmanager configuration are tenant-wide resources: loading a complete configuration for one environment into a shared tenant can overwrite another environment’s configuration. SLI recordings that omit deployment_environment can also combine traffic before either alerts or release gates evaluate it.

Treating tenant separation as an undocumented secret convention is unsafe. A wrong credential can silently turn environment isolation into shared state.

Decision

DEV, QA, and PROD use separate Grafana stacks or tenant identities for Mimir, Loki, Tempo, and Alertmanager. Each GitHub Environment declares the expected tenant identity alongside its query and write credentials.
CD verifies the configured tenant identity before reading or mutating rules, routes, dashboards, or collector destinations. A missing or unexpected identity fails closed.
Physical separation does not replace logical identity. Operational metrics, logs, traces, SLI recordings, alerts, dashboard queries, and synthetic checks retain the exact deployment.environment=signal-forge-<env> value.
Operational recording rules aggregate by service_name, deployment_environment. Histogram rules additionally retain le. Alertmanager groups and inhibits on environment as well as the service/SLO identity.
CD diffs the live tenant resources against the desired environment bundle, rejects undeclared cross-environment resources, and captures the previous version before mutation.

Alternatives considered

One shared tenant with a merged routing tree. Viable at larger scale, but it requires a single reconciler to own every environment’s rules and routes. Signal Forge currently deploys each environment independently, so separate tenants make the ownership boundary explicit.
Rely only on environment labels. Rejected because labels do not prevent a full Alertmanager configuration load from replacing another environment’s routes.

Consequences

Three environment-specific credential sets and tenant identities must be managed.
Cross-environment queries require an explicit aggregate view outside the release path.
A deployment cannot proceed when tenant identity is unverifiable, even if the endpoints and credentials otherwise work.
Environment labels remain mandatory, which preserves incident context and protects against accidental future tenant consolidation.

Links