Continuous Training Pipelines for Multi-Agent Cognition

Drift-triggered retraining, the promotion gate, and the shadow-eval discipline.

§ IFrame

Last Monday the curriculum named evaluation as the layer that watches the answers. Eval detects when a model's quality-surface bends: a regression on the labeled set, a drift on the canary set, a freeze-worthy failure on the adversarial set. Eval names the problem. It does not fix it.

This lesson takes the next stage. The model that drifted has to be replaced by a better one, and the better one rarely arrives by accident. It arrives because a pipeline noticed the drift, gathered fresh training data, re-ran the training workflow, scored the candidate against the same eval discipline that raised the alarm, and promoted it only after the candidate proved itself. That pipeline is continuous training.

The word continuous misleads beginners. It does not mean a model retrains every hour on a clock. A model that retrains on a fixed schedule with no trigger wastes compute when nothing changed and lags reality when something did. Continuous training means the retraining loop is always armed and fires when a condition the operator chose deliberately becomes true. The clock is one possible condition. Drift is a better one.

An operator who treats retraining as a quarterly project inherits a model that is always three months behind the data. An operator who treats retraining as an armed loop inherits a model that follows the data and a record of every time it had to.

§ IIFoundations

Four parts carry a continuous-training pipeline. Name them; the design follows.

The four compose into a loop with a property worth stating plainly: the loop's output is never a model that serves traffic directly. The loop's output is a candidate that the gate either admits or rejects.

§ IIIMechanism

The trigger, and why drift beats the clock

A schedule-only trigger has one virtue: it is simple. It also retrains a fraud model at 3 a.m. every Sunday whether or not fraud patterns moved, and it does nothing when a new fraud pattern appears on a Tuesday. The schedule answers has time passed when the operator needs to answer has the world changed.

A drift-driven trigger reads the canary-set scores the eval pipeline already produces. When the canary metric for a model version falls below threshold across a sustained window, the trigger fires. The eval pipeline that last Monday's lesson built becomes the input to this Monday's loop without any new instrumentation. Drift detection and retraining are the same loop seen from two ends.

The mature pattern carries both: drift as the primary trigger, a long-horizon schedule as a floor so the model never goes stale even in a quiet regime, and a manual override so an operator who knows a regime shift is coming can retrain ahead of the drift instead of after it.

Data assembly is where retraining quietly fails

The training run gets the attention. The data assembly step earns the failures.

Fresh data arrives with the distribution of the present, which is the point, and with the defects of the present, which is the danger. A new upstream service starts emitting a field in a different unit. A labeling vendor changes its guidelines mid-week. A bot floods one request category and skews the class balance. Each of these produces training data that looks fine row by row and is wrong in aggregate.

The discipline is to validate the assembled set against the prior set before training, not after. Compare feature distributions between this run's data and the last run's data; flag any feature whose distribution moved more than a chosen distance. Check label balance against the historical range. Re-run the schema contract the serving layer enforces. A retraining run that begins only after its input passes these checks fails far less often, and when it does fail, fails for a reason the operator can name.

The training run and the reproducibility floor

A retraining run is the original training workflow with new data and a new version stamp. The original orchestration lesson named multi-agent training workflows as a coordinated graph of steps; retraining re-runs that graph. The one rule that separates a useful retraining run from a confusing one is reproducibility: every input that is not the data must be pinned. Same code commit, same dependency versions, same hyperparameters, same random seeds where the framework honors them.

The reason is attribution. When the new candidate scores differently from the prior model, the operator needs to know whether the data moved the score or the recipe moved the score. Pin everything except the data, and a score change is a data effect the operator can trust. Let the recipe drift between runs, and every score change is a mystery.

The promotion gate and the shadow-eval discipline

The candidate exists. It must not serve a single real request until it has proven it is at least as good as the model it would replace. The pattern that earns this proof is shadow evaluation.

A shadow endpoint receives a copy of live traffic and produces outputs that no user ever sees. The candidate runs in shadow while the prior model continues to serve. The eval pipeline scores both on the same inputs: the same regression set, the same canary window, the same adversarial bank. Because both models see identical inputs, the comparison is clean — any score difference is the model, not the traffic.

The gate then applies the same three verdict patterns the eval lesson named. The candidate must hold the regression set within tolerance of the prior model. It must match or beat the prior model on the canary metric that triggered the retraining in the first place; a retraining that does not fix the drift it was summoned to fix has failed its own purpose. And it must show no adversarial regression: no previously-refused prompt now answered, no previously-handled edge case now broken.

§ IVWorked Example

A support-routing agent classifies incoming tickets into one of forty queues and drafts a first-response suggestion. It runs on a fine-tuned classification model behind the routing layer plus a generation model for the draft.

For three weeks the canary metric holds near 0.91 accuracy on the routing classification. Then a new product line ships. Tickets about the new product arrive in volume, and the model — trained before the product existed — routes them to a catch-all queue. The canary accuracy slides to 0.78 across a four-day window. The eval pipeline's drift detector crosses threshold and emits a verdict.

The trigger reads that verdict and fires. Data assembly gathers the last thirty days of tickets, including the new-product tickets now labeled by the support team during their manual triage of the misrouted catch-all queue. The validation step compares the new training set against the prior one: the queue-label distribution has a new mode (the new product), which is expected and within the allowed range; no feature distribution moved unexpectedly; the schema contract passes. Assembly completes.

The training run re-executes the original fine-tuning workflow — same base model, same hyperparameters, same code commit — on the assembled data, and registers the candidate as version 14 alongside the live version 13.

Shadow evaluation runs both versions on the same inputs. The regression set, 600 historical tickets labeled by senior agents, returns 0.90 for the candidate against 0.91 for the prior; the bootstrap interval on the difference crosses zero, so the gate counts it as no regression. The canary set, the recent four-day window that drifted, returns 0.93 for the candidate against the live 0.78 — the candidate fixes the drift it was summoned to fix. The adversarial set, 50 deliberately ambiguous tickets and a handful of prompt-injection attempts hidden in ticket bodies, returns no new failures.

The candidate promotes. It takes one percent of routing traffic while the eval pipeline watches, then ten, then full. The canary accuracy recovers to 0.92. The whole loop ran without a human deciding anything except the thresholds, and every step left a record the operator can read.

§ VConnection to Prior Lessons

The 2026-06-01 lesson on evaluation pipelines built the layer that watches the answers and named three verdict patterns: promotion gating, rollback triggering, freeze on adversarial regression. This lesson consumes all three. The drift verdict that eval raises is the trigger that fires retraining; the promotion gating eval defined is the gate the candidate clears; the adversarial freeze is the third check shadow evaluation enforces. Eval and continuous training are one loop: eval is the half that notices, retraining is the half that acts.

The 2026-05-25 lesson on model-serving topology named routing, pooling, versioning, and cost. Continuous training is the supplier of the versions that topology routes between. Versioning makes shadow endpoints and canary deployment possible; this lesson is what fills the shadow endpoint and decides whether the canary advances.

The 2026-05-18 lesson on multi-agent orchestration patterns named the training workflow as a coordinated graph of steps. A retraining run is that graph re-executed under the reproducibility discipline. Orchestration built the workflow once; continuous training is the discipline of running it again, correctly, on demand.

§ VIConnection to Today's Dev Lesson

The Python Dev lesson today encodes the four-part pipeline as a typed graph of stages. Where this Ops lesson named the trigger, data assembly, training run, and promotion gate as four parts that compose into a loop, the Python lesson shows how Python's structural typing makes each stage a step whose input and output contract the type checker verifies before anything runs.

A retraining pipeline is a sequence of stages where the output of one is the input of the next. The Python lesson builds that chain with Protocol for the stage contract, generics for the data flowing between stages, and ParamSpec for preserving stage signatures through composition. A stage wired to the wrong neighbor fails the type check instead of failing at 3 a.m. on real data. The shape this Ops lesson named in prose becomes a graph the type checker walks before the pipeline ever fires.

§ VIIClosing

Continuous training is the loop that keeps a model honest against a world that moves. Eval notices the drift; retraining answers it; the gate makes sure the answer is real. Four parts: a trigger that fires on drift rather than on a clock, an assembly step that validates before it trains, a training run pinned everywhere except the data, and a promotion gate that admits a candidate only after shadow evaluation proves it.

The output of the loop is never a model that serves. The output is a candidate the gate judges. Retraining proposes; the gate disposes.

Examine the loop well. The model serving traffic next week is the one this week's gate let through, and the gate is only as honest as the eval discipline behind it.