Designing Observability for Agent Workflows

Verify the claim with a paired fault-injection matrix, not span counts. Run the same workflows with task/turn spans enabled and omitted, then inject: generation timeout or error, function failure, rejected guardrail, and missing or failed handoff. For each run, require the retained trace to expose a trace ID, surviving parent-child path, boundary type, duration, and outcome classification sufficient to select the same next action as the full hierarchy: retry or inspect generation, route to the tool owner, resolve the guardrail outcome, or investigate delegation.

The decisive negative control is ambiguity: if two injected failures produce indistinguishable retained paths or outcomes after task/turn omission, the reduced configuration may still detect failure but does not diagnose it. Measure export separately from observability: only score traces after the worker’s required delivery point, since the documented exporter is asynchronous. The SDK documents that task and turn spans may be omitted while lower-level agent, generation, function, guardrail, and handoff spans remain available; that is a configuration capability, not evidence that those retained spans preserve every diagnostic decision. Agents SDK tracing documentation

Make the adapter’s failure mode part of the interoperability contract. Consider a convention upgrade or a newly encountered SDK span_data variant: if the adapter silently emits a partial destination span, the trace graph may remain intact while agent identity, conversation correlation, operation semantics, or failure classification disappear. That is a monitoring regression that can be hard to detect from successful exports alone.

For every fixture, assert a three-part recovery path: detection through a mapping-version and unmapped-field signal; containment by marking or quarantining the affected export rather than presenting it as conformant; and recovery by replaying the retained fixture or captured source representation through the corrected, version-pinned adapter and comparing the resulting graph and attributes. Include malformed or absent source fields as negative cases, not only generation, function, handoff, guardrail, and ordinary-failure examples.

This matters because the SDK trace model provides identifiers, timestamps, parentage, and type-specific span_data, while the OpenTelemetry registry explicitly says GenAI attributes have moved to a separate semantic-conventions repository. A passing transport test is therefore not evidence that a changed mapping remains semantically recoverable. Require the fixture suite before enabling a new adapter or convention version, and retain a rollback to the previously verified mapping if that suite fails. SDK tracing reference OpenTelemetry GenAI registry

The distinction should also apply to deployment posture. The documented facts are that tracing is enabled by default, can be disabled globally or per run, and is unavailable under Zero Data Retention; none of those facts establishes what a particular processor exports, which fields are redacted, or what retention/access controls apply after export. See the tracing documentation.

A reviewable claim boundary would be:

• Supported: SDK tracing availability and documented enable/disable controls.
• Requires deployment evidence: the effective tracing setting, processor destination, and an inspected exported payload for each span type.
• Not established by these excerpts: suitability for a given security, privacy, retention, or access-control requirement.

A reproducible check can run one fixture with tracing enabled and one with RunConfig.tracing_disabled=True, capture the processor output in both cases, and retain the configuration plus raw payload as evidence. That separates “the SDK offers a control” from “this deployment prevented export of the relevant data.”

A bounded export-and-flush path needs an explicit delivery contract at the worker boundary. The SDK documentation says batch export is asynchronous and recommends flush_traces() after leaving the trace context when a long-lived worker needs immediate end-of-unit export; it does not make worker completion itself a delivery acknowledgement. Treat these as separate outcomes: work_completed, flush_requested, and trace_observed.

complete unit(trace_id)
exit trace context
flush_traces()
verify trace_id and ended spans by deadline
else record export_delivery_unknown and reconcile

The important contract boundary is that a timeout or failed lookup must not be silently converted into “no trace” or “worker succeeded with delivered telemetry.” Preserve the trace_id, deadline, lookup result, and any export error as distinct fields so a reconciler can retry observation or raise an export-delivery incident. This keeps the synchronous business-result contract separate from the asynchronous observability-delivery contract while still giving callers a bounded, testable path.