Go Channels and Pipeline Patterns for Multi-Signal Filter Chains

Fan-out, fan-in, and selectivity in streaming market data.

§ IFrame

Today's Ops lesson named a five-stage signal-pipeline architecture: ingest, normalize, filter, score, size. The architecture is conceptual. The implementation question is how to render it in code such that each stage runs concurrently with the others, communicates with its neighbors through a typed boundary, propagates a single shutdown signal cleanly when the operator cancels the run, and exposes the per-stage instrumentation the operator needs to detect divergence between live and backtest paths.

Go offers a concurrency model fitted to that question. Goroutines are independently-scheduled units of work, cheap enough that thousands can run inside a single process without overhead. Channels are typed conduits between goroutines that the language compiler will not let two stages misuse. Together they implement the pipeline pattern in a form that reads, on the page, very close to the architectural diagram of the system.

This lesson takes the five-stage Ops architecture and shows the Go-specific implementation choices the language asks the engineer to make. The choices are not arbitrary. Each one is the place where Go's runtime model meets the production discipline the Ops lesson named.

§ IILanguage Idiom

Channels in Go are typed first-in-first-out queues that goroutines use to coordinate. A goroutine that holds the send side of a channel transmits values; a goroutine that holds the receive side waits for values. The send-receive pair is synchronizing by default. When a goroutine sends on an unbuffered channel, the send blocks until another goroutine receives. When a goroutine receives on a channel with nothing in it, the receive blocks until something is sent. The synchronization is the coordination.

A buffered channel allows up to N items to wait inside the channel before the send-side blocks. Choosing the buffer size is a backpressure decision: a buffer of zero forces upstream and downstream to lock-step; a buffer of N lets upstream get N events ahead of downstream before the upstream blocks. In a signal pipeline, the buffer choice is the operator's statement about how much queueing the system should tolerate before it slows ingest. Larger buffers absorb burst loads at the cost of staler in-flight data.

A select statement waits on multiple channel operations and proceeds with whichever one is ready first. Select is how a goroutine handles more than one concurrent input or output without busy-polling. In a signal pipeline, select implements both the cancellation-aware receive (wait for an event from upstream OR a cancellation signal from the context, whichever arrives first) and the live-vs-backtest divergence detector (consume from the live feed and the replay feed in parallel and compare events as they arrive).

The Go standard library's context.Context (covered in Tuesday's Go lesson) integrates with the channel model through ctx.Done(). The Done() method returns a channel that closes when the context is canceled. Stages that receive on ctx.Done() inside their main select loop will unblock cleanly when the operator cancels the run. The cancellation signal propagates through the pipeline without each stage having to know about the cancellation mechanism explicitly.

§ IIICode Worked Example — Five-Stage Pipeline

The pipeline below shows the five stages from today's Ops lesson rendered in Go. The shape is one goroutine per stage, channels between stages, a single context propagating through every stage.

Stage 1: Ingest

The first stage is ingest. The function reads events from an external feed (a venue WebSocket, an exchange snapshot poll, an on-chain log subscription) and emits them on an outbound channel. The function takes the context so it can stop cleanly when the operator cancels.

type RawEvent struct {
    Venue     string
    Symbol    string
    Timestamp time.Time
    Payload   []byte
}

func ingest(ctx context.Context, feed FeedReader, out chan<- RawEvent) error {
    defer close(out)
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        default:
        }
        ev, err := feed.Read(ctx)
        if err != nil {
            return fmt.Errorf("ingest: %w", err)
        }
        select {
        case out <- ev:
        case <-ctx.Done():
            return ctx.Err()
        }
    }
}

Two patterns repeat in this stage and every stage downstream. The first is the cancellation-aware send: the goroutine attempts to send the event on out, but is willing to give up the send and exit if the context cancels first. Without that pattern, a stage with a backed-up downstream consumer would deadlock against the cancellation signal and never exit. The second is defer close(out): when ingest exits (cleanly or with an error), it closes its outbound channel. Closing a channel signals to downstream goroutines that no more events will arrive; they can drain remaining events and exit themselves. The closure cascade is how the pipeline shuts down without explicit per-stage coordination code.

Stage 2: Normalize

The normalize stage reads RawEvent values and emits typed Feature values that are computable identically in backtest and live paths. The stage holds whatever state the feature computation needs (rolling windows, last-known book state, the previous candle's close).

type Feature struct {
    Symbol            string
    Timestamp         time.Time
    MidPrice          float64
    SpreadBps         float64
    RollingVolatility float64
    BookImbalance     float64
}

func normalize(ctx context.Context, in <-chan RawEvent, out chan<- Feature, state *NormalizerState) error {
    defer close(out)
    for {
        select {
        case raw, ok := <-in:
            if !ok {
                return nil
            }
            f, err := state.Compute(raw)
            if err != nil {
                continue
            }
            select {
            case out <- f:
            case <-ctx.Done():
                return ctx.Err()
            }
        case <-ctx.Done():
            return ctx.Err()
        }
    }
}

The ok value on the channel receive distinguishes a real event from a closed-channel signal. When the upstream stage closes its outbound channel, downstream receives return zero-value events with ok == false. The stage drains, exits, and closes its own outbound channel. The closure cascade carries the shutdown through every stage in series without explicit coordination.

Stage 3: Filter (the selectivity stage)

The filter stage is where the selectivity geometry from the Ops lesson lives. The stage runs each candidate through a sequence of single-purpose predicates and emits only those candidates that pass every filter. Each filter logs its decision through a structured logger so the operator can audit rejected candidates later.

type FilterChain []func(Feature) (pass bool, reason string)

func filter(ctx context.Context, in <-chan Feature, out chan<- Feature, chain FilterChain, log Logger) error {
    defer close(out)
    var seen, passed atomic.Int64
    for {
        select {
        case f, ok := <-in:
            if !ok {
                log.Info("filter exit", "seen", seen.Load(), "passed", passed.Load(),
                    "selectivity", float64(passed.Load())/float64(max(1, seen.Load())))
                return nil
            }
            seen.Add(1)
            rejected := false
            for _, pred := range chain {
                pass, reason := pred(f)
                if !pass {
                    log.Info("filter reject", "symbol", f.Symbol, "reason", reason)
                    rejected = true
                    break
                }
            }
            if !rejected {
                passed.Add(1)
                select {
                case out <- f:
                case <-ctx.Done():
                    return ctx.Err()
                }
            }
        case <-ctx.Done():
            return ctx.Err()
        }
    }
}

The selectivity ratio (passed / seen) is the operator-visible metric from the Ops lesson. Logging it at stage exit gives the operator the per-run number; logging it incrementally on a tick interval (omitted here for brevity) gives the operator the time-series view the Ops lesson called for. The atomic counters allow concurrent observability scrape goroutines to read the values safely without locking.

Stage 4: Score (with internal fan-out)

The score stage runs the surviving candidates through the strategy's quantitative voice. The Ops lesson made the case that scoring must be deterministic given identical features. The stage is also the natural fan-out point: independent signal-family scorers run in parallel goroutines and fan back into a combiner.

type ScoredCandidate struct {
    Feature
    Score float64
    Components map[string]float64
}

func score(ctx context.Context, in <-chan Feature, out chan<- ScoredCandidate,
    families []SignalFamily, combiner func(map[string]float64) float64) error {
    defer close(out)
    for {
        select {
        case f, ok := <-in:
            if !ok {
                return nil
            }
            results := make(chan struct {
                name  string
                value float64
            }, len(families))
            var wg sync.WaitGroup
            for _, fam := range families {
                wg.Add(1)
                go func(family SignalFamily) {
                    defer wg.Done()
                    v := family.Compute(f)
                    results <- struct {
                        name  string
                        value float64
                    }{family.Name, v}
                }(fam)
            }
            wg.Wait()
            close(results)
            components := make(map[string]float64, len(families))
            for r := range results {
                components[r.name] = r.value
            }
            sc := ScoredCandidate{Feature: f, Score: combiner(components), Components: components}
            select {
            case out <- sc:
            case <-ctx.Done():
                return ctx.Err()
            }
        case <-ctx.Done():
            return ctx.Err()
        }
    }
}

The fan-out is internal to the score stage. For each candidate, the stage launches one goroutine per signal family, waits for all to complete via a sync.WaitGroup, and assembles the results into a combined score. The fan-in happens through the buffered results channel of capacity exactly equal to the number of families: every family's contribution gets a slot. Because the buffer matches the producer count, the family goroutines never block on send.

Stage 5: Size (the kill-switch surface)

The size stage is downstream of score and upstream of execution. Under the Capital-Zero-INVARIANT discipline the Ops lesson cited, the stage's behavior switches on a paper-mode flag.

type OrderIntent struct {
    ScoredCandidate
    Side      string
    QuantityUSD float64
    Mode      string
}

func size(ctx context.Context, in <-chan ScoredCandidate, out chan<- OrderIntent,
    sizer Sizer, paperMode bool, journal Journal) error {
    defer close(out)
    for {
        select {
        case sc, ok := <-in:
            if !ok {
                return nil
            }
            side, qty := sizer.Compute(sc)
            mode := "live"
            if paperMode {
                mode = "paper"
            }
            intent := OrderIntent{ScoredCandidate: sc, Side: side, QuantityUSD: qty, Mode: mode}
            journal.Record(intent)
            if paperMode {
                continue
            }
            select {
            case out <- intent:
            case <-ctx.Done():
                return ctx.Err()
            }
        case <-ctx.Done():
            return ctx.Err()
        }
    }
}

In paper mode, the stage records the intent to a journal and emits nothing downstream. The execution goroutine downstream of size sees an idle channel and runs no orders. In live mode, the intent both records to the journal and emits downstream. The journal records every intent in both modes so the operator can later compare paper-mode intents to live-mode actuals on the same data.

Wiring the five stages

The wiring of the five stages happens in one function that creates the channels and launches each stage in its own goroutine.

func RunPipeline(ctx context.Context, cfg PipelineConfig) error {
    rawCh := make(chan RawEvent, cfg.IngestBuffer)
    featCh := make(chan Feature, cfg.NormalizeBuffer)
    passCh := make(chan Feature, cfg.FilterBuffer)
    scoreCh := make(chan ScoredCandidate, cfg.ScoreBuffer)
    intentCh := make(chan OrderIntent, cfg.SizeBuffer)

    g, gctx := errgroup.WithContext(ctx)
    g.Go(func() error { return ingest(gctx, cfg.Feed, rawCh) })
    g.Go(func() error { return normalize(gctx, rawCh, featCh, cfg.NormalizerState) })
    g.Go(func() error { return filter(gctx, featCh, passCh, cfg.Chain, cfg.Log) })
    g.Go(func() error { return score(gctx, passCh, scoreCh, cfg.Families, cfg.Combiner) })
    g.Go(func() error { return size(gctx, scoreCh, intentCh, cfg.Sizer, cfg.PaperMode, cfg.Journal) })
    g.Go(func() error { return execute(gctx, intentCh, cfg.Venue) })
    return g.Wait()
}

The errgroup.Group from golang.org/x/sync/errgroup is the standard Go pattern for managing a set of goroutines with shared cancellation. When any goroutine returns an error, the group's context cancels; every other stage observes the cancellation through its select on ctx.Done() and exits cleanly. The shutdown is coordinated through the channel-closure cascade and the context-cancellation propagation simultaneously.

§ IVConnection to Today's Ops Lesson

The five-stage architecture from today's Ops lesson is the conceptual model; the Go implementation above is one way to render it. The pairing is exact: each stage in the architecture becomes one goroutine in the implementation, each transition between stages becomes one channel, the cross-cutting operator disciplines (selectivity logging, paper-mode switching, observability metrics) become explicit code in the stage they belong to.

Three of today's Ops disciplines map directly onto Go's runtime model. The first is per-stage observability: structured logs and counters live at every stage's exit and tick boundaries. Go's log/slog package emits the structured logs; sync/atomic counters cover the per-stage rate metrics. The second is single-purpose filters: the FilterChain slice is a sequence of independent functions, each a single predicate. The compiler will not let one filter accidentally couple to another. The third is the paper-mode kill-switch: the size stage's paperMode parameter changes one boolean in one function and the rest of the pipeline behaves identically. Capital-Zero-INVARIANT discipline at the Go layer is literally a flag.

§ VPrior-Lesson Reach

Tuesday's Go lesson on context.Context covered the propagation discipline: every function in a request takes a context as its first argument, every spawned goroutine receives the same context, every network call passes the context to the network library. The pipeline above honors that discipline at every stage. The single context.Context threads through RunPipeline into each of the six goroutines, and each stage's select includes ctx.Done() as one of its cases. When the operator cancels the run (or when one goroutine returns an error and the errgroup cancels its derived context), every stage exits cleanly within one event loop iteration.

Thursday's Python lesson on contextvars named the per-task state primitive in Python's asyncio runtime. Go's equivalent is context.Context's WithValue, but the Go community generally avoids stashing request-scoped state inside the context; the Go idiom is to thread the state through the goroutine's local variables and channel messages instead. The two languages reach the same observability and tracing goals through different language-level primitives. The signal pipeline above sits closer to the Python contextvars story than to most Go programs because the strategy state lives inside each stage's local variables and only the order intent crosses stage boundaries via channels.

Monday's α-Cognition Ops lesson on multi-agent orchestration named the broker-worker pattern that the pipeline above implements at language-primitive scale. The orchestration shape (one controller routing work to specialized workers) is the same shape this pipeline expresses, with the difference that the workers here are stages and the controller is the channel topology rather than an explicit broker process.

Paired Ops lesson → Archmagus-Stack/γ-Adversarial-Markets/Synthesis-Lessons/2026-05-22-production-signal-pipelines-...

§ VIClosing

Go's channel-and-goroutine model is unusually well-fitted to the signal-pipeline shape. The five stages from today's Ops lesson render as six small goroutines, six channels, and one shared context. The shape on the page is close to the architectural diagram, the compiler enforces the type boundary at every stage transition, and the language's standard concurrency idioms (the closure cascade, the cancellation-aware select, the errgroup) handle the lifecycle coordination that production deployments quietly demand.

The next Go lesson on Praetor cycle (Mon 2026-05-25 under the Lang-W2 rotation, when Mon+Thu shifts from Python to Go) builds on this pipeline shape with execution-side discipline: venue API integration, retry-and-idempotency patterns, partial-fill reconciliation, and the metrics surface that turns a working pipeline into an operations-ready deployment. The substrate accreted today (cancellation-aware select, fan-out for parallel scoring, fan-in for combiner, paper-mode kill-switch) will recur in every Go lesson that follows under the γ-Adversarial-Markets curriculum.