Go's context.Context for Distributed Tracing Across Multi-Agent Workflows

One typed argument at every function signature. The discipline becomes visible the moment the trace tree assembles.

§ IFrame

Today's Ops lesson named three observability primitives. The first of them, distributed tracing, asked the engineer to propagate a span context across every agent boundary, every tool call, every goroutine. The propagation discipline is what makes the trace tree assemble at the collector. Skip one boundary and the trace splinters: two disconnected sub-trees instead of one rooted causal graph.

Go made a choice about this propagation problem that no other major language made. The choice has a name: context.Context. The package was added to the standard library in Go 1.7 (August 2016) after years of pattern-prototype work across Google's production Go codebases. It carries four things: a deadline, a cancellation signal, request-scoped values, and (by convention since OpenTelemetry's Go SDK stabilized in 2021) the active trace span. Every function that takes part in a request takes a context.Context as its first argument. Functions that spawn goroutines pass the context to those goroutines. Functions that make network calls pass the context to the network library.

The result is a discipline visible at every function signature. A reviewer reading a Go service can tell, by reading the function signatures alone, which functions participate in the request's tracing graph. The compile-time signal is the discipline. There is no equivalent in Python, where context propagation hides inside contextvars.ContextVar or async-task locals; no equivalent in Java, where Thread-Local Storage is technically present but invisible at the method signature. Go made tracing propagation a typed argument. The decision shapes how every Go observability library is built.

This lesson takes that choice and shows what it costs and what it earns when the system is a multi-agent workflow.

§ IILanguage Idiom

The context.Context interface has four methods. The interface itself:

type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}
    Err() error
    Value(key any) any
}

Deadline returns the time at which the context expires. Done returns a channel that is closed when the context is cancelled (whether by timeout, by parent cancellation, or by explicit cancel() call). Err returns the cancellation reason. Value retrieves a request-scoped value by key. The interface is small by design.

Contexts are created from a root (context.Background() for top-level work or context.TODO() for not-yet-decided cases), then derived by wrapping. The four derivation functions are WithCancel, WithDeadline, WithTimeout, and WithValue. Each takes a parent context and returns a child context that inherits the parent's properties while adding its own. Cancellation propagates downward through the derivation tree: cancel the parent, every child is cancelled too. Values look upward through the tree: a Value(key) call walks the parent chain until a match is found.

The tree shape is critical for tracing. When function A calls function B with ctx, and B calls function C, all three operations are linked through the context tree. When B opens a span with tracer.Start(ctx, "operation-B"), the OpenTelemetry SDK stores the new span as a value in a child context and returns that child to B. B passes the child to C. C reads the active span from the context, opens its own child span, and the trace tree mirrors the function-call tree exactly.

§ IIICode Worked Example

A single agent-call function, instrumented end-to-end. The function signature, the span open, the deferred close, and the structured log records all wire together through one ctx value.

package agent

import (
    "context"
    "fmt"
    "log/slog"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/trace"
)

type AgentRequest struct {
    AgentName string
    Input     string
    Tools     []string
}

type AgentResult struct {
    Output    string
    TokenIn   int
    TokenOut  int
    ToolCalls int
}

func CallAgent(ctx context.Context, req AgentRequest) (AgentResult, error) {
    start := time.Now()
    tracer := otel.Tracer("hedronite/agent")
    ctx, span := tracer.Start(ctx, "agent.call",
        trace.WithAttributes(
            attribute.String("agent.name", req.AgentName),
            attribute.Int("agent.tool_count", len(req.Tools)),
        ),
    )
    defer span.End()

    logger := slog.Default().With(
        slog.String("agent_name", req.AgentName),
        slog.String("trace_id", span.SpanContext().TraceID().String()),
        slog.String("span_id", span.SpanContext().SpanID().String()),
    )

    logger.InfoContext(ctx, "agent invocation begin",
        slog.Int("input_length", len(req.Input)),
    )

    result, err := dispatchToWorker(ctx, req)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        logger.ErrorContext(ctx, "agent invocation failed",
            slog.String("error", err.Error()),
            slog.Duration("elapsed", time.Since(start)),
        )
        return AgentResult{}, fmt.Errorf("dispatch failed: %w", err)
    }

    span.SetAttributes(
        attribute.Int("agent.token_in", result.TokenIn),
        attribute.Int("agent.token_out", result.TokenOut),
        attribute.Int("agent.tool_calls", result.ToolCalls),
    )
    span.SetStatus(codes.Ok, "")

    logger.InfoContext(ctx, "agent invocation complete",
        slog.Int("token_in", result.TokenIn),
        slog.Int("token_out", result.TokenOut),
        slog.Duration("elapsed", time.Since(start)),
    )

    return result, nil
}

The function takes ctx as its first argument, per convention. It opens a span as the first operation; the returned ctx (now carrying the active span) shadows the parameter for the rest of the function. The defer span.End() ensures the span closes regardless of which return path the function takes. The logger is built once per call with the trace identifier and span identifier already attached, so every subsequent log record carries those fields without the call site repeating them.

The dispatchToWorker call passes the trace-bearing context downward. Inside that function, another tracer.Start call will fire, producing a child span. The parent-child relationship is recorded at the OpenTelemetry SDK layer without any explicit identifier passing. When the trace collector reassembles the trace, the function-call hierarchy and the span hierarchy match exactly.

A second function shows how the discipline scales to a fan-out:

func DispatchSweep(ctx context.Context, configs []SweepConfig) ([]SweepResult, error) {
    tracer := otel.Tracer("hedronite/agent")
    ctx, span := tracer.Start(ctx, "sweep.dispatch",
        trace.WithAttributes(attribute.Int("sweep.cell_count", len(configs))),
    )
    defer span.End()

    results := make([]SweepResult, len(configs))
    errs := make([]error, len(configs))

    done := make(chan int, len(configs))
    for i, cfg := range configs {
        go func(idx int, c SweepConfig) {
            cellCtx, cellSpan := tracer.Start(ctx, "sweep.cell",
                trace.WithAttributes(attribute.Int("sweep.cell_index", idx)),
            )
            defer cellSpan.End()

            r, err := runCell(cellCtx, c)
            results[idx] = r
            errs[idx] = err
            if err != nil {
                cellSpan.RecordError(err)
                cellSpan.SetStatus(codes.Error, err.Error())
            }
            done <- idx
        }(i, cfg)
    }

    for i := 0; i < len(configs); i++ {
        <-done
    }

    span.SetAttributes(attribute.Int("sweep.completed", len(configs)))
    return results, joinErrors(errs)
}

Each goroutine receives the parent ctx by closure and opens its own child span. The N child spans run in parallel and report independently to the collector. The collector reassembles them by parent-identifier into a fan-out tree matching the Ops lesson's §IV worked example exactly. The pattern is identical at the trace layer; only the function-level details differ. Note that the ctx referenced inside the goroutine is the parent context (the one with the sweep span as active); the derivation to a per-cell context happens inside the goroutine via the tracer.Start call.

Cancellation propagates the same way. If the calling function's context is cancelled (the principal hit the kill switch on the nightly run), every cell's cellCtx.Done() channel fires. Each runCell invocation checks cellCtx.Done() between long operations and returns early with cellCtx.Err(). The cancellation event is recorded on each cell's span before the span closes. The trace shows exactly which cells got how far before the cancellation arrived.

§ IVConnection to Today's Ops Lesson

The Ops lesson named tracing, metrics, and structured logging as the three primitives of multi-agent observability. The Go realization:

Tracing is OpenTelemetry's Go SDK, with context.Context as the propagation carrier. Every tracer.Start(ctx, name) call mints a child span and returns a context whose Value chain holds the new span. The deferred span.End() closes the span and ships it to the configured exporter (OTLP-over-gRPC to a collector by default). The collector reassembles the trace tree from the parent-identifier on each span.

Metrics are Prometheus' Go client (github.com/prometheus/client_golang/prometheus) or the OpenTelemetry metrics SDK. The Prometheus library has won the practitioner side for raw metric emission; the OTel metrics SDK is winning where teams want one library for both signals. Both register metric collectors at package init and expose an HTTP endpoint that the Prometheus server scrapes on a 15-second cadence.

Structured logging is Go 1.21's slog package. slog.InfoContext(ctx, msg, attrs...) accepts a context, and the configured handler can extract the trace identifier from the context for inclusion on every record. The slog.Logger is built once per request with the trace identifier already attached, so every subsequent log call inherits it automatically.

Paired lesson → Archmagus-Stack/01-Earth-DevOps/Synthesis-Lessons/2026-05-19-observability-for-multi-agent-llm-systems

§ VPrior-Lesson Reach

Two earlier artifacts and one prior Dev lesson feed this one.

Monday's Python iterator-protocol lesson. Monday's Dev lesson showed how Python's yield-driven generators implement the Pipeline pattern as a chain of streaming functions. The Go equivalent is a chain of context-bearing functions, each opening a child span and passing the context to the next. Python's iterator protocol carries iteration state through the generator's frame; Go's context.Context carries trace state through the function call. Different language idioms, same architectural payoff: state that flows with the work without manual reconstruction at each stage.

Khairallah multi-agent canon-asset. Khairallah's Pipeline pattern is a chain of agent calls; in Go, each call is a function taking ctx as first argument, opening a child span, and passing the context downward. Khairallah's Fan-Out pattern is a parent dispatching N parallel workers; in Go, each worker is a goroutine receiving the parent context by closure and opening its own child span. The patterns map cleanly into the language idiom because both Khairallah's pattern descriptions and Go's context design come from the same Google-engineering tradition of explicit dependency propagation.

Yesterday's Ops lesson. Monday's lesson named the three patterns and called the audit-at-each-seam essential. Today's Ops lesson took one of those audit primitives (distributed tracing) and named what it does. This Dev lesson takes that primitive and shows its Go-language realization through context.Context. The three together are the operator's first complete tour of observable-multi-agent-system construction: Monday sets the architecture; Tuesday's Ops names the observation primitives; Tuesday's Dev hands the primitives their Go-language vocabulary.

The convergence: Python's iterator protocol gives streaming pipelines; Go's context gives traced pipelines; Khairallah names the shapes both languages realize.

§ VIClosing

The Go programmer who reads this should hold three things.

First, that context.Context is not just a cancellation signal. The community treats it as the carrier of request-scoped state, with the trace span as the most important value it carries in any observable system. A function signature without ctx context.Context as the first argument is a function that has opted out of the observation graph. Don't opt out without a clear reason.

Second, that the goroutine-context-closure pattern is the canonical way Go expresses parallel sub-operations within a traced parent. The parent opens a sweep span; each goroutine receives the parent context, opens its own child span at the top of the goroutine body, and ends the span at the bottom. The fan-out tree at the trace collector mirrors the goroutine fan-out at the code site. Read the §III sweep example until the pattern feels natural.

Third, that the cross-pivot between traces, metrics, and logs is engineering work. The trace identifier must be attached to every log record (via slog's handler reading the context). The metric labels must match the trace attribute keys so a high-latency metric query can find the slow spans. The discipline at emission time pays back at debugging time.

Read the §III code samples until the context-passing pattern feels like reflex. Open one of your Go agent calls. Add a span at the top, defer the close at the bottom, pass the context to every function the call invokes. Then run the trace and read the tree. The discipline becomes visible at the moment the tree assembles.