OTel Signal Contracts

OTel Signal Contracts

Production-grade OpenTelemetry contracts for all four services in SignalForge. These contracts define the exact signals each service emits — span names, metric instrument names, log field schemas, and propagation mechanisms — so that dashboards, alerts, and the Alloy pipeline can be built and maintained against a stable interface.

OTel semantic conventions: stable HTTP + RPC semconv (http.request.method, http.response.status_code, url.path, url.full, rpc.grpc.status_code) unless noted SDK versions: .NET OTel SDK / AspNetCore + Http instrumentation 1.16.0 (GrpcNetClient 1.9.0-beta.1), Python opentelemetry-sdk 1.44.0 / instrumentation 0.65b0, Grafana Faro Web SDK 1.10

---

Table of Contents

1. Conventions
2. Telemetry data classification
3. Resource Attributes
4. Propagation
5. Collector Enrichment
6. gateway-api
7. order-api
8. notification-svc
9. frontend (Faro RUM)
10. Cross-Service Trace Topology
11. Observability as release policy
12. Validation Queries

---

Conventions

Attribute naming

• Custom attributes follow domain.noun dot-notation (e.g., order.project_id, email.delay_ms).
• OTel semantic convention attributes follow the published semconv spec and are not redefined here — only their presence on specific spans is called out.
• All attribute names are lowercase with underscores as word separators within each segment.

Metric naming

• Instrument names follow service_noun.measurement with a {unit} suffix in the description.
• Prometheus exposition names are generated by the OTel → OTLP → Alloy → Prometheus pipeline: dots become underscores, {unit} suffix is dropped, _total is appended to Counter types.
• Exemplars are emitted on Histograms when OTEL_METRICS_EXEMPLAR_FILTER=TRACE_BASED is set (all .NET services) or when TraceBased exemplar filtering is enabled (Python).

Unit conventions

OTel unit string	Description
ms	Milliseconds (wall-clock duration)
{request}	Dimensionless count of HTTP requests
{order}	Dimensionless count of orders
{notification}	Dimensionless count of notifications
USD	US Dollars (monetary amount)

---

Telemetry data classification

config/observability/data-policy.yaml is the versioned classification source of truth. It is enforced at browser/source and collector layers; values that reach Grafana backends must already comply with this contract.

Classification	Permitted use	Examples
Allowed	Stable operational dimensions and semantic route templates.	service/deployment identity, route template, HTTP status, RPC operation
Restricted	Correlation-only fields; not metric labels and access is limited by the tenant role policy.	trace ID, span ID, project ID
Forbidden	Never persist in telemetry.	authorization/token/cookie, email, client IP, user agent, plant ID, project name, SQL text/parameters, request body, exception message/stack trace

Gateway/order SDKs do not attach client identity and set RecordException=false. QA/PROD disable SQL statement collection; DEV replaces any captured statement with the literal redacted before export. Local Alloy and the cloud destination delete the same forbidden span attributes; pod stdout is regex-redacted before Loki. Browser Faro recursively redacts forbidden keys and secret/PII patterns in beforeSend.

The policy also declares 7/14/30-day DEV/QA/PROD retention and restricted-reader/break-glass roles. scripts/ci/audit_tenant_settings.py reconciles that declaration against each environment’s real Grafana Cloud stack: it reads retention from Loki’s config-self-serve API (GET <loki_url>/loki/api/v1/config/limits/retention_period) and confirms each declared role name exists via Grafana’s own RBAC API (GET <grafana_url>/api/access-control/roles) — the latter proves a role is defined, not who holds it, since Grafana exposes no reverse role-to-assignee lookup. Its output is a sanitized sanitized-grafana-tenant-settings-export evidence object (per-field declared/actual/matched booleans only — no tokens, no raw API responses) embedded directly in the release manifest (.tenant_settings) that .github/workflows/ci.yml assembles and signs; scripts/ci/validate-release-manifest.sh rejects a manifest whose tenant_settings.result isn’t pass or the deliberate day-1 not_yet_configured rollout state (recorded only when no environment has a read-only audit credential provisioned yet — see the script’s own docstring). A passing static policy check on data-policy.yaml alone is still not proof the external tenant configuration matches; this signed, per-release evidence is.

---

Resource Attributes

Resource attributes describe the entity producing telemetry. They are attached to every span, metric data point, and log record. Alloy’s otelcol.processor.k8sattributes enriches all signals with k8s attributes at the collector level, so services do not need to set them.

Common attributes (all .NET services)

Set by ResourceBuilder.CreateDefault().AddService(name).AddTelemetrySdk().AddEnvironmentVariableDetector():

Attribute	Source	Example
service.name	AddService()	"gateway-api"
service.version	Shared runtime ConfigMap via env detector	local: "1.0.0"; CD: full release commit SHA
telemetry.sdk.name	AddTelemetrySdk()	"opentelemetry"
telemetry.sdk.language	AddTelemetrySdk()	"dotnet"
telemetry.sdk.version	AddTelemetrySdk()	"1.7.0"
OTEL_RESOURCE_ATTRIBUTES.*	AddEnvironmentVariableDetector()	any k/v pairs

Common attributes (notification-svc Python)

Set manually in telemetry.py via Resource.create():

Attribute	Value
service.name	"notification-svc"
service.namespace	"otel-lab"
service.version	Local default "1.0.0"; immutable CD release commit SHA
deployment.environment	Shared OTEL_RESOURCE_ATTRIBUTES; local config or CD target (signal-forge-dev, -qa, -prod)

Name and source split for the deployment environment. The OTel resource attribute is deployment.environment (dotted); the Prometheus/Loki label it becomes is deployment_environment (underscored — set as extraLabels on the Grafana Cloud destinations, and depends on resourceToTelemetryConversion for the attribute form). Its value has two sources that must agree: monitoring.deployment_environment in conf.yml for local mode, and signal-forge-<env> derived from the CD --environment input on the promotion path. They line up today only because promotion re-renders the app-env ConfigMap; nothing enforces it.

Alloy-injected k8s attributes (all services, collector-side)

Added by otelcol.processor.k8sattributes in the alloy-receiver pipeline:

Attribute	Example
k8s.pod.name	"gateway-api-6f4b9d-xkpq2"
k8s.namespace.name	"otel-lab"
k8s.deployment.name	"gateway-api"
k8s.node.name	"k3d-otel-lab-server-0"
k8s.container.name	"gateway-api"

Frontend (Faro RUM)

Set by initializeFaro({ app: { ... } }):

Attribute	Value
app.name	"otel-frontend"
app.version	Exact 40-hex release SHA
app.environment	`signal-forge-dev

---

Propagation

HTTP (gateway-api ↔ order-api via gRPC, gateway-api → notification-svc REST)

Format: W3C TraceContext (traceparent, tracestate) + W3C Baggage Mechanism: .NET OTel SDK injects/extracts headers automatically via AddGrpcClientInstrumentation and AddHttpClientInstrumentation. No manual code required for HTTP/gRPC propagation.

RabbitMQ (order-api PRODUCER → notification-svc CONSUMER)

Format: W3C TraceContext only (traceparent header) Inject side (OrderPublisher.cs):

RabbitMQ message header "traceparent" = UTF-8 bytes of W3C traceparent value

The outbox relay restores the persisted request context for its own span, but the publisher writes the original stored traceparent directly as byte[]; this prevents a background retry from silently replacing request identity with worker identity.

Extract side (consumer.py):

ctx = extract(headers, getter=HeadersGetter())  # decodes bytes → str
token = attach(ctx)
# ... create CONSUMER span with links=[Link(parent_span_ctx)] ...
detach(token)  # always in finally block

The CONSUMER span uses a SpanLink (not parent-child) to the PRODUCER span context, per OTel messaging semconv for async operations.

Browser → Backend (Faro → gateway-api)

Format: W3C TraceContext (traceparent) injected into XHR/fetch request headers Scope: Applied to all requests matching environment.apiBaseUrl regex and /http:\/\/localhost/ Mechanism: Faro TracingInstrumentation with propagateTraceHeaderCorsUrls configuration

---

Collector Enrichment

All OTLP signals flow through alloy-receiver (Helm-managed DaemonSet, monitoring namespace). The receiver pipeline applies:

Stage	Effect
k8sattributes processor	Adds k8s.pod.name, k8s.namespace.name, k8s.deployment.name, k8s.node.name to all signals
Data-policy transform	Removes forbidden trace attributes before Tempo/Jaeger; pod stdout is redacted before Loki
Span filter	Drops spans where http.route == "/healthz" or url.path == "/healthz"
Tail sampling	errors=100%, slow (>2s)=100%, remaining=25%
Spanmetrics connector	Generates RED metrics (calls_total, duration_milliseconds_bucket) per service/operation/status
OTLP export	Traces → Jaeger/Tempo, Metrics → Prometheus/Mimir, Logs → Loki

---

gateway-api

Service name: gateway-api Stack: .NET 8 Minimal API, MySQL 8 (EF Core + Pomelo), gRPC client (to order-api), HTTP client (to notification-svc)

Traces

Auto-instrumented spans

HTTP attribute names are OTel .NET 1.16 stable HTTP semconv (http.request.method, http.response.status_code, url.path, url.full) — the pre-1.9 http.method / http.status_code / http.url names are not emitted. url.full is removed by the source/collector data policy; client IP, user agent, and plant ID are not added to spans.

Span name pattern	Kind	Key attributes	Notes
HTTP {METHOD} {route}	SERVER	http.request.method, http.route, http.response.status_code, url.path	ASP.NET Core; /healthz excluded by filter
orders.OrderService/CreateOrder	CLIENT	rpc.system=grpc, rpc.service, rpc.method, rpc.grpc.status_code	gRPC client to order-api
orders.OrderService/GetOrdersByProject	CLIENT	same as above	gRPC client to order-api
orders.OrderService/GetOrder	CLIENT	same as above	gRPC client to order-api
HTTP GET / HTTP POST	CLIENT	http.request.method, http.response.status_code, server.address	HTTP client to notification-svc; full URL removed
{db operation} {table}	CLIENT	db.system=mysql, db.name, db.operation; DEV only may show db.statement=redacted	SQL text/parameters never exported

RecordException=false prevents exception text/stack events from being created. Error status and safe exception type remain available for operational diagnosis.

Custom spans

All custom spans use ActivityKind.Internal unless otherwise noted.

Span name	Parent	Attributes	Status
gateway.get_projects	HTTP SERVER span	(none)	OK or ERROR on exception
gateway.get_project	HTTP SERVER span	project.id	ERROR + description on 404
gateway.create_project	HTTP SERVER span	project.id (set after DB write)	OK or ERROR on exception
gateway.delete_project	HTTP SERVER span	project.id	ERROR + description on 404
gateway.fanout (orders)	HTTP SERVER span	order.project_id, order.id	ERROR with a fixed safe status description
gateway.fanout (project orders)	HTTP SERVER span	project.id	OK; gateway.downstream.duration recorded with downstream=order-api, operation=GetOrdersByProject
gateway.slow	HTTP SERVER span	delay.ms (2000–5000ms random)	OK
gateway.error	HTTP SERVER span	(none)	ERROR without raw exception payload

Metrics

Custom instruments

Instrument name	Type	Unit	Dimensions	Description
gateway.requests.inflight	UpDownCounter<long>	{request}	(none)	Active in-flight HTTP requests, excluding /healthz
gateway.downstream.duration	Histogram<double>	ms	downstream, operation	Latency of downstream service calls (gRPC or HTTP)

Prometheus exposition names:

• gateway_requests_inflight
• gateway_downstream_duration_milliseconds_{bucket,count,sum}

Standard instruments

Source	Instrument prefix	Notes
AddAspNetCoreInstrumentation()	http.server.*	Request count, duration, active requests
AddRuntimeInstrumentation()	dotnet.*	GC collections, heap size, thread pool
AddProcessInstrumentation()	process.*	CPU time, working set, virtual memory, file descriptors

Exemplars

OTEL_METRICS_EXEMPLAR_FILTER=TRACE_BASED is set in the K8s Deployment. Histograms emit exemplars carrying trace_id and span_id when the recording happens inside a sampled trace. This links Prometheus metric data points to their source trace in Jaeger/Tempo.

Logs

Format: JSON via AddJsonConsole() on stdout OTel export: AddOpenTelemetry() on ILoggingBuilder with IncludeFormattedMessage=true, IncludeScopes=true, ParseStateValues=true

Log records emitted in request handlers include a manually injected TraceId structured parameter alongside the message template parameters. This is in addition to the OTel SDK’s automatic TraceId/SpanId injection on exported log records.

Structured log fields (JSON keys on stdout):

Field	Type	Example	Source
Timestamp	ISO-8601 string	"2024-11-15T10:23:45.123Z"	.NET logging
Level	string	"Information"	.NET logging
Message	string	"Created project 42"	formatted message
SourceContext	string	"GatewayApi.Endpoints.ProjectEndpoints"	logger category
TraceId	string (32-hex)	"4bf92f3577b34da6..."	manual injection
SpanId	string (16-hex)	"00f067aa0ba902b7"	OTel SDK (via OTLP export)
{ParameterName}	varies	ProjectId: 42	ParseStateValues=true

Log-to-trace correlation: Alloy’s loki.process pipeline extracts TraceId and SpanId from the JSON log line and promotes them as Loki structured metadata (traceID, spanID). Grafana uses these to surface “Logs for this span” in the Tempo trace view.

---

order-api

Service name: order-api Stack: .NET 8 gRPC server, PostgreSQL 16 (EF Core + Npgsql 8), RabbitMQ publisher

Traces

Auto-instrumented spans

Span name pattern	Kind	Key attributes	Notes
orders.OrderService/CreateOrder	SERVER	rpc.system=grpc, rpc.service=orders.OrderService, rpc.method=CreateOrder	ASP.NET Core gRPC middleware; caller identity suppressed
orders.OrderService/GetOrdersByProject	SERVER	same; streaming RPC; span covers full stream duration
orders.OrderService/GetOrder	SERVER	same
{db operation} {table}	CLIENT	db.system=postgresql, db.name, db.operation; DEV only may show db.statement=redacted	SQL text/parameters never exported
Npgsql driver spans	CLIENT	db.system=postgresql, connection attributes	Source: "Npgsql" ActivitySource; registered via AddSource("Npgsql")

Client IP, user agent, and plant ID are forbidden fields and are not added to gRPC SERVER spans.

Custom spans

Span name	Kind	Parent	Attributes	Status / Events
order.create	INTERNAL	gRPC SERVER span	order.project_id (set on entry), order.amount, order.id (set after SaveChangesAsync)	ERROR without raw exception payload
order.get_by_project	INTERNAL	gRPC SERVER span	order.project_id	OK
order.get	INTERNAL	gRPC SERVER span	order.id	ERROR with description "Order not found" on 404
outbox.relay	INTERNAL	persisted order.create context	outbox.message_id, order.id	Restores the stored parent context and adds an ActivityLink; each retry is separately observable
order.publish	PRODUCER	outbox.relay span	messaging.system=rabbitmq, messaging.destination=orders, messaging.destination_kind=exchange, messaging.rabbitmq.routing_key=order.created, order.id	OK

Outbox propagation detail: CreateOrder stores the original W3C traceparent with the order and outbox row. A later relay poll restores that context as outbox.relay’s parent and adds an ActivityLink, then order.publish writes the stored traceparent into AMQP headers as UTF-8 byte[]. The Python consumer’s HeadersGetter decodes those values before it creates its linked consumer span. The publisher does not call Propagators.Inject() from a detached worker context.

Metrics

Custom instruments

Instrument name	Type	Unit	Dimensions	Prometheus name
orders.created.total	Counter<long>	{order}	none	orders_created_total
orders.amount.total	Counter<double>	USD	none	orders_amount_total
orders.processing.duration	Histogram<double>	ms	none	orders_processing_duration_milliseconds_{bucket,count,sum}

None of these carry a project_id dimension — it’s unbounded (grows with every project ever created), and this project’s own engineering principles flag unbounded/high-churn labels as an automatic stop for metrics. Per-project drill-down uses the order.project_id span attribute (set on order.create) and this histogram’s trace-based exemplar instead — both can safely carry high-cardinality IDs.

orders.processing.duration measures wall-clock time from the start of order.create to the DB write committing — not to completion of OrderPublisher.Publish(), which no longer happens inline (see the outbox pattern: OutboxRelayWorker publishes later, out-of-band, in its own poll loop). Because this recording happens inside a sampled trace, it carries a trace exemplar.

Standard instruments

Source	Instrument prefix
AddAspNetCoreInstrumentation()	http.server.* (gRPC calls appear as HTTP/2)
AddRuntimeInstrumentation()	dotnet.*
AddProcessInstrumentation()	process.*

Logs

Same format and OTel configuration as gateway-api (JSON console, ParseStateValues=true). Structured parameters per log site:

Log site	Parameters	Level
CreateOrder completion	OrderId, ProjectId, Amount, TraceId	Information
GetOrdersByProject completion	Count, ProjectId, TraceId	Information
OrderPublisher.Publish completion	OrderId, TraceId	Information

---

notification-svc

Service name: notification-svc Stack: Python 3.11, FastAPI, pika (RabbitMQ consumer), redis-py, pythonjsonlogger

Traces

Auto-instrumented spans

Span name pattern	Kind	Key attributes	Notes
{METHOD} {route}	SERVER	http.request.method, http.route, http.response.status_code, url.path	FastAPI via FastAPIInstrumentor (stable HTTP semconv); /healthz excluded
redis {command}	CLIENT	db.system=redis, db.statement={COMMAND args}, net.peer.name, net.peer.port	RedisInstrumentor().instrument() — all commands: EXISTS, HSET, EXPIRE, SET, LPUSH, LTRIM, LRANGE, HGETALL

Custom spans

Span name	Kind	Relationship	Attributes	Events / Status
notification.process	CONSUMER	SpanLink to PRODUCER span context	messaging.system=rabbitmq, messaging.operation=receive, messaging.source.name=orders, messaging.rabbitmq.routing_key=order.created, order.id, order.project_id	Exception recorded on unhandled error; span ends after basic_ack or basic_nack
notification.send_email	INTERNAL	Child of notification.process	email.order_id, email.delay_ms	OK always (mock)

Link relationship on notification.process: The span link (links=[Link(parent_span_ctx)]) preserves the async trace relationship without forcing a parent-child timing dependency. In Jaeger, linked spans render as dashed arrows from the PRODUCER, making the queue-crossing visible without falsely representing the CONSUMER as temporally subordinate to the PRODUCER.

Context lifecycle:

token = attach(ctx)       # install extracted context on this thread
# ... span and all children automatically inherit context ...
finally: detach(token)    # always; prevents context leak across message deliveries

Metrics

Custom instruments

Instrument name	Type	Unit	Dimensions	Prometheus name
notifications.processed.total	Counter	{notification}	status (success | duplicate | failed)	notifications_processed_total
notifications.processing.duration	Histogram	ms	(none)	notifications_processing_duration_milliseconds_{bucket,count,sum}
notifications.email.send.duration	Histogram	ms	(none)	notifications_email_send_duration_milliseconds_{bucket,count,sum}

notifications.processing.duration covers the full handle_order_created lifecycle: dedup Redis check + Redis write + mock email send. Recorded unconditionally (including for duplicates, where it measures only the dedup check).

Export configuration: PeriodicExportingMetricReader with export_interval_millis=15_000 — aligns with Prometheus default scrape interval to avoid gaps in dashboards.

Logs

Format: pythonjsonlogger JsonFormatter on logging.StreamHandler(sys.stdout) Format string: "%(asctime)s %(name)s %(levelname)s %(message)s %(otelTraceID)s %(otelSpanID)s"

After LoggingInstrumentor().instrument() is applied, every log record produced while a span is active gains:

Field	Type	Example	Source
asctime	string	"2024-11-15 10:23:45,123"	LogRecord
name	string	"app.consumer"	logger name
levelname	string	"INFO"	LogRecord
message	string	"Processed notification for order 42"	formatted message
otelTraceID	string (32-hex)	"4bf92f3577b34da6..."	LoggingInstrumentor
otelSpanID	string (16-hex)	"00f067aa0ba902b7"	LoggingInstrumentor
otelServiceName	string	"notification-svc"	LoggingInstrumentor

OTLP log export: Disabled (OTEL_LOGS_EXPORTER=none). Logs reach Loki exclusively through Alloy’s loki.source.kubernetes pod log tailing pipeline. The loki.process stage extracts otelTraceID and otelSpanID from JSON lines and promotes them as Loki structured metadata for trace correlation.

Exception logging: logger.exception("Failed to process order.created event") — used instead of logger.error(..., exc) to include the full traceback without interpolating potentially sensitive exception message text into the log message string.

---

frontend (Faro RUM)

App name: otel-frontend Stack: Angular 17, Grafana Faro Web SDK, Faro Web Tracing (OTel-based), nginx

RUM Signal Types

Faro collects four signal types and ships them to the Grafana Faro collector endpoint (FARO_URL):

Signal type	Trigger	Example
log	console.log/warn/error calls captured by Faro console instrumentation	{level: "error", message: "..."}
exception	Unhandled JS exceptions, Angular ErrorHandler	{type: "TypeError", value: "...", stacktrace: [...]}
event	Page load, navigation, custom events	{name: "page_view", domain: "browser"}
trace	XHR/fetch spans created by TracingInstrumentation	OTel-format spans

Session Tracking

Property	Value	Notes
samplingRate	1 (100%)	Reduce to 0.1 in high-traffic production
persistent	true	Session ID survives page reloads via sessionStorage

Tracing (XHR/fetch spans)

TracingInstrumentation creates OTel-format CLIENT spans for every XMLHttpRequest and fetch call made by the Angular app and injects W3C traceparent into request headers.

Propagation scope — traceparent is injected only for URLs matching:

• environment.apiBaseUrl (regex-escaped, e.g., http://localhost:8080/api)
• /http:\/\/localhost/ (any localhost URL)

Span attribute	Value	Notes
http.method	"GET" / "POST" / "DELETE"	HTTP verb
http.url	not persisted	full URL/query is redacted by beforeSend
http.status_code	integer	set on response
component	"xml-http-request" or "fetch"	Faro tracing convention

API calls and their trace linkage

Angular method	HTTP call	Linked backend span
getProjects()	GET /api/projects	gateway.get_projects
getProject(id)	GET /api/projects/{id}	gateway.get_project
createProject(...)	POST /api/projects	gateway.create_project
deleteProject(id)	DELETE /api/projects/{id}	gateway.delete_project
getOrdersByProject(id)	GET /api/projects/{id}/orders	gateway.fanout → order-api
createOrder(...)	POST /api/orders	gateway.fanout → order.create → asynchronous outbox.relay → order.publish
getNotifications()	GET /api/notifications	notification-svc FastAPI span
triggerError()	GET /api/error	gateway.error

beforeSend filter

Only LOG items whose actual message contains "/healthz" are dropped. Every other Faro payload is recursively cloned and redacted before transport: forbidden keys (credentials, cookies, email, IP, plant/project names, SQL/request-body/exception fields) become [redacted]; bearer tokens, email, IP, and secret assignments embedded in arbitrary strings are replaced. Exception value and stacktrace are always replaced, not pattern-matched.

---

Cross-Service Trace Topology

The complete 5-hop trace for a POST /api/orders request:

%%{init: {'theme':'base','themeVariables':{'background':'#0f172a','primaryColor':'#bae6fd','primaryTextColor':'#0f172a','primaryBorderColor':'#7dd3fc','secondaryColor':'#bbf7d0','tertiaryColor':'#fde68a','lineColor':'#cbd5e1','clusterBkg':'#1e293b','clusterBorder':'#94a3b8','titleColor':'#f8fafc','edgeLabelBackground':'#1e293b'}}}%%
flowchart TD
    subgraph Browser["Browser (Faro)"]
        A["[CLIENT] XHR fetch span<br/>TracingInstrumentation, W3C traceparent → HTTP header"]
    end

    subgraph GW["gateway-api"]
        B["[SERVER] HTTP POST /api/orders"]
        C["[INTERNAL] gateway.fanout<br/>tags: order.project_id, order.id"]
        D["[CLIENT] orders.OrderService/CreateOrder (gRPC)"]
    end

    subgraph OA["order-api"]
        E["[SERVER] orders.OrderService/CreateOrder"]
        F["[INTERNAL] order.create<br/>tags: project_id, amount, id"]
        G["[CLIENT] EF Core INSERT Orders<br/>db.system=postgresql"]
        R["[INTERNAL] outbox.relay<br/>later poll; restored parent + ActivityLink"]
        H["[PRODUCER] order.publish<br/>W3C traceparent → RabbitMQ header"]
    end

    subgraph NS["notification-svc"]
        I["[CONSUMER] notification.process"]
        J["[CLIENT] redis EXISTS (dedup)"]
        K["[CLIENT] redis HSET (store)"]
        L["[CLIENT] redis EXPIRE"]
        M["[CLIENT] redis SET (dedup key)"]
        N["[CLIENT] redis LPUSH + LTRIM"]
        O["[INTERNAL] notification.send_email"]
    end

    A --> B --> C --> D --> E --> F
    F --> G
    F -.->|stored traceparent; asynchronous relay| R
    R --> H
    H -.->|SpanLink, async queue crossing| I
    I --> J --> K --> L --> M --> N
    I --> O
    classDef app fill:#bae6fd,stroke:#7dd3fc,color:#0f172a;
    classDef store fill:#bbf7d0,stroke:#4ade80,color:#0f172a;
    class A,B,C,D,E,F,R,H,I,O app;
    class G,J,K,L,M,N store;

Trace continuity rules:

• Hops 1–4 (Browser → gateway-api → order.create) share the same traceId via HTTP/gRPC W3C header propagation — they form a connected parent-child tree.
• The later outbox relay restores the stored request context as its parent and adds an ActivityLink. This retains one trace query across the asynchronous DB boundary while making a retry attempt separately visible; it is not an inline RabbitMQ publish.
• The RabbitMQ → notification-svc hop shares the same traceId: notification.process is created as a child of the producer span carried in the message traceparent header and carries an explicit SpanLink to it, so the async boundary stays visible (dashed reference in Jaeger) without splitting the trace. See ADR-002.
• notification.process therefore has a non-nil parentSpanId (the producer span’s spanId) and links[0] = producer span context. Each NACK+redelivery adds another notification.process child of the same producer span.

---

Observability as release policy

This repository implements a static, blocking observability-as-policy gate in CI. The policy script validates the contract this page describes before a release image can be published:

• backend OTEL_SERVICE_NAME values must match service identity;
• service.namespace, service.version, and deployment.environment must be present in the shared runtime resource attributes;
• local Alloy and cloud Helm inputs must render as YAML/River, Prometheus rules must pass promtool + mimirtool rules check, the rendered Alloy file must pass the real Alloy validator, and helm template on the pinned chart must emit the spanmetrics connector, the /healthz drop, and the integrations/alloy self-scrape;
• the local River span-metric connector and the cloud chart’s applicationObservability.connectors.spanMetrics must carry the same dimensions (the shared k8s/monitoring/grafana/shared/span-dimensions.txt) and pin namespace: traces.spanmetrics / unit: ms;
• every environment must have an Alertmanager routing template that renders, parses, and routes only to declared receivers (amtool check-config in the same job);
• committed local dashboards must be valid JSON with a title and panel array, and SLO/runbook assets must exist;
• explicitly configured span-metric dimensions cannot include known unbounded request/user/session identifiers such as trace_id, request_id, user_id, raw URL, or email.

This is intentionally a focused release contract, not a fabricated runtime health check. It does not query Grafana, prove dashboard query semantics, measure telemetry arrival, or calculate a full cardinality/cost budget.

For an enabled DEV/QA/PROD deployment, CD runs scripts/ci/observability_gate.py (the observability-gate phase), preceded by a tagged synthetic journey through gateway → order → RabbitMQ → notification. The gate polls Grafana Cloud for the candidate’s live metrics, logs, traces, resource attributes, collector health, and SLO burn rate — keyed on the exact service.version and deployment.environment. block fails every environment; warn is visible in DEV/QA but fails PROD. Its queries and thresholds are versioned, unit-tested code; only the OBS_GATE_* query credentials are environment configuration. See Immutable CI/CD Promotion and ADR-011.

Validation Queries

Traces — Jaeger

Verify the 5-hop async trace exists:

1. In Jaeger UI, search service=gateway-api, operation=HTTP POST /api/orders
2. Open a trace — confirm it contains spans from all four services
3. Confirm order.publish has a dashed link arrow to notification.process
4. Confirm order.publish and notification.process share the same traceId

Metrics — Prometheus / PromQL

# Orders per second by project
rate(orders_created_total[5m])

# p99 order processing latency
histogram_quantile(0.99, rate(orders_processing_duration_milliseconds_bucket[5m]))

# Gateway downstream latency p95 per downstream service
histogram_quantile(0.95,
  rate(gateway_downstream_duration_milliseconds_bucket[5m]))
by (downstream, operation, le)

# Notification failure rate
rate(notifications_processed_total{status="failed"}[5m])
  /
rate(notifications_processed_total[5m])

# Inflight requests (gauge)
gateway_requests_inflight

Logs — Loki / LogQL

# Correlate logs for a specific trace ID
{app="gateway-api"} | json | TraceId="<traceId>"

# notification-svc errors with trace context
{app="notification-svc"} | json | levelname="ERROR"
  | line_format "{{.message}} trace={{.otelTraceID}}"

# All services — log volume by level
sum by (app, levelname) (
  count_over_time({namespace="otel-lab"} | json [5m])
)

Exemplar verification — Grafana

1. Open Grafana → Explore → Prometheus datasource
2. Query orders_processing_duration_milliseconds_bucket
3. Enable “Exemplars” toggle
4. Click an exemplar point — it should navigate to the corresponding Tempo trace