feat(tracing): establish W3C trace context at FastACP ingress

[NiteshDhanpal]

Stack (2/3)

1. feat(tracing): correlate business spans with active observability trace ([WIP] feat(tracing): correlate business spans with active observability trace #465)
2. feat(tracing): establish W3C trace context at FastACP ingress ← this PR
3. feat(tracing): propagate trace context into Temporal via OTel interceptor

Based on obs/01-span-obs-correlation. Review/merge after #465.

What this PR does

Extends RequestIDMiddleware in base_acp_server.py to extract the inbound W3C trace context (traceparent/tracestate) and attach it as the current OpenTelemetry context for the lifetime of the request.

This gives obs_correlation() (PR 1) a real emit-time context: business spans created while handling an RPC — and any Temporal workflow started from that request (PR 3) — now adopt the caller's observability trace_id.

Why it's needed

PR 1 only reads the active obs context; without this, there is no context to read on the ACP request path (the SGP /v5/spans/batch transport doesn't carry it). This is the client-side propagation half.

Safety

Best-effort and lazy-imported: no traceparent header, or OTel unavailable, is a safe no-op and never breaks a request. Context is detached in a finally.

Verification

py_compile + functional smoke: a traceparent header yields a valid current span context with the expected trace_id; no header → invalid context (no-op).

Greptile Summary

This PR extends RequestIDMiddleware in base_acp_server.py to extract the inbound W3C trace context (traceparent/tracestate) and attach it as the active OpenTelemetry context for the lifetime of each HTTP request, enabling downstream business spans and Temporal workflows to share the caller's observability trace ID.

• Two small helper functions (_attach_incoming_trace_context, _detach_trace_context) are added; both are best-effort with lazy OTel imports so a missing opentelemetry package or a missing header is always a safe no-op.
• RequestIDMiddleware.__call__ is refactored from a guard-and-fall-through pattern to an early-return for non-HTTP scope types, and the trace context attach/detach is wrapped in a try/finally block alongside the existing request-ID logic.

Confidence Score: 4/5

Safe to merge — the change is additive, never raises, and the existing request-handling path is functionally unchanged in the absence of OTel.

The attach/detach lifecycle is correct and properly guarded with try/finally; asyncio.create_task inherits the contextvars copy at creation time so background tasks correctly retain the trace context. The one imprecision is the plain-dict carrier that silently drops all but the last value when a caller forwards multiple tracestate headers, which is spec-legal but uncommon in practice.

base_acp_server.py — the multi-value tracestate edge case in _attach_incoming_trace_context is the only area that warrants a second look.

Important Files Changed

Filename	Overview
src/agentex/lib/sdk/fastacp/base/base_acp_server.py	Extends RequestIDMiddleware to extract and attach W3C trace context (traceparent/tracestate) from inbound ASGI headers using lazy OTel imports and a try/finally detach; one edge case where duplicate tracestate headers are silently collapsed by the plain-dict carrier.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Caller as Caller (SGP/upstream)
    participant MW as RequestIDMiddleware
    participant OTel as OpenTelemetry
    participant App as FastAPI App
    participant BG as Background Task

    Caller->>MW: HTTP Request (traceparent header)
    MW->>MW: Extract x-request-id → ctx_var_request_id.set()
    MW->>OTel: extract(carrier) → W3C context
    OTel-->>MW: ctx (with trace_id from traceparent)
    MW->>OTel: context.attach(ctx) → token
    MW->>App: await self.app(scope, receive, send)
    App->>App: asyncio.create_task(_process_request) copies current context
    App-->>MW: response sent
    BG->>BG: runs with copied context (trace_id still active)
    MW->>OTel: context.detach(token) [finally]
    Note over MW,OTel: original context restored

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Caller as Caller (SGP/upstream)
    participant MW as RequestIDMiddleware
    participant OTel as OpenTelemetry
    participant App as FastAPI App
    participant BG as Background Task

    Caller->>MW: HTTP Request (traceparent header)
    MW->>MW: Extract x-request-id → ctx_var_request_id.set()
    MW->>OTel: extract(carrier) → W3C context
    OTel-->>MW: ctx (with trace_id from traceparent)
    MW->>OTel: context.attach(ctx) → token
    MW->>App: await self.app(scope, receive, send)
    App->>App: asyncio.create_task(_process_request) copies current context
    App-->>MW: response sent
    BG->>BG: runs with copied context (trace_id still active)
    MW->>OTel: context.detach(token) [finally]
    Note over MW,OTel: original context restored

Loading

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
src/agentex/lib/sdk/fastacp/base/base_acp_server.py:62-67
**Duplicate `tracestate` headers silently dropped**

The plain-dict carrier overwrites duplicate keys, so if an intermediary forwards multiple `tracestate` headers (which the W3C Trace Context spec explicitly permits), all but the last entry are silently discarded. The OTel `DefaultGetter` calls `carrier.get(key)` on a plain dict, which can only return a single string value, so combined trace-state from several systems in the propagation chain would be partially lost.

To handle multi-value headers correctly, the carrier should concatenate repeated values with a comma separator before inserting them into the dict, or use a multi-value mapping with a custom `Getter`.

_{Reviews (1): Last reviewed commit: "feat(tracing): establish W3C trace conte..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[greptile-apps]

Duplicate tracestate headers silently dropped

The plain-dict carrier overwrites duplicate keys, so if an intermediary forwards multiple tracestate headers (which the W3C Trace Context spec explicitly permits), all but the last entry are silently discarded. The OTel DefaultGetter calls carrier.get(key) on a plain dict, which can only return a single string value, so combined trace-state from several systems in the propagation chain would be partially lost.

To handle multi-value headers correctly, the carrier should concatenate repeated values with a comma separator before inserting them into the dict, or use a multi-value mapping with a custom Getter.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agentex/lib/sdk/fastacp/base/base_acp_server.py
Line: 62-67

Comment:
**Duplicate `tracestate` headers silently dropped**

The plain-dict carrier overwrites duplicate keys, so if an intermediary forwards multiple `tracestate` headers (which the W3C Trace Context spec explicitly permits), all but the last entry are silently discarded. The OTel `DefaultGetter` calls `carrier.get(key)` on a plain dict, which can only return a single string value, so combined trace-state from several systems in the propagation chain would be partially lost.

To handle multi-value headers correctly, the carrier should concatenate repeated values with a comma separator before inserting them into the dict, or use a multi-value mapping with a custom `Getter`.

How can I resolve this? If you propose a fix, please make it concise.