Signal Forge ADR-002: Same-trace consumer span plus a SpanLink for async RabbitMQ

Signal Forge ADR-002: Same-trace consumer span plus a SpanLink for async RabbitMQ

Status: Accepted (revised 2026-09-10 — was “SpanLink, not parent-child”; the code has always produced both, and the single end-to-end trace is a deliberate, load-bearing property. See the Observability-as-Policy audit finding M3.)

Decision: The notification-svc notification.process span continues the originating trace: it is created as a child of the order-api producer span whose W3C context travels in the RabbitMQ message traceparent header, and it carries an explicit Link back to that same context. order-api’s outbox.relay span does the equivalent on the .NET side (parented to the stored order.create context, plus a redundant ActivityLink). The same consumer NACKs poison messages to a dead-letter queue — see the instrumentation reference for the full producer/consumer walkthrough.

Mechanically (Python side): extract(headers) → attach(ctx) → tracer.start_as_current_span("notification.process", kind=CONSUMER, links=[Link(producer_ctx)]) with no explicit context=, so the attached remote context becomes the parent. scripts/ci/observability_gate.py and src/notification-svc/tests/test_consumer_trace_parenting.py both assert this shape; changing it is a breaking change, not a refactor.

Rationale:

• One trace, end to end. A single POST /api/orders produces one trace ID that threads browser → gateway → order.create → outbox.relay → order.publish → notification.process. A single Jaeger/Tempo query surfaces the whole chain; log-to-trace correlation works across the async hop without stitching two trace IDs together. This is the product’s headline observability story and is what the cross-language integration test and the CD gate’s synthetic-trace check verify.
• Tail sampling stays coherent. The errors-always policy force-retains a trace when any span in it has status ERROR. Because the consumer span is in the same trace, an error while processing a notification retains the full originating request trace, not just a disconnected consumer fragment. The decision_wait window (10s) exists precisely to let the late async span join before the trace is evaluated.
• The Link still marks the async boundary. It is not redundant noise: it records that producer and consumer are separated by a queue and by arbitrary wall-clock time (the consumer span can start long after the producer span ended), and in Jaeger it renders as a dashed reference, visually distinct from a synchronous call. It also survives if a future change does move the consumer to its own trace.
• Redelivery is fan-out, not corruption. After a NACK, each redelivery re-runs the handler with the same header and produces another notification.process child of the same producer span. Multiple children of one parent is a valid trace tree (fan-out); the per-span Link on each makes the retry relationship explicit. (The earlier version of this ADR called this an “invalid trace tree” — that was wrong.)

Alternatives considered:

• Link only, consumer span as a new trace root (strict reading of the OTel messaging conventions). Rejected: it splits the end-to-end trace in two, breaks the single-query correlation story, and drops the originating request trace out of errors-always retention when the failure is in the consumer. The messaging-semconv guidance is a default, not a constraint — a same-trace child with a Link is explicitly permitted and is the better fit for a system whose async hop is a first-class part of one logical request.
• Parent-child with no Link. Rejected: loses the visible async-boundary marker and the retry relationship.

Consequences:

• The header inject on the producer is deliberately the stored order.create context written as raw traceparent bytes (not Propagators.Inject() from Activity.Current), so a worker retry cannot overwrite request identity with worker identity. See OrderPublisher.cs.
• notification.process is not a root span; any doc or comment saying parentSpanId is nil is stale.
• Moving to link-only later would require updating test_consumer_trace_parenting.py, the integration-cross-language assertions, and observability_gate.py::_check_synthetic_trace, and re-tuning tail sampling — hence this ADR pins the current shape with a test.