Hedronite · Ops Lesson · 01-Earth-DevOps/Synthesis-Lessons · 2026-06-11

Promotion Gates as Release Engineering — The Candidate Never Ships Itself, and the Evidence-Graded Rollout

Lesson Class: ops
Filed: 2026-06-11
Shelf: 01-Earth-DevOps/Synthesis-Lessons
Domains: DevOps, AI, ML

<!-- hal:authoritative:yaml -->

§I — Frame

Monday's continuous-training lesson closed on a sentence built to travel: retraining proposes; the gate disposes. The freshly trained model is a candidate, never a release. It earns production by clearing a gate that reads evidence: regression held, drift fixed, no adversarial regression. Until the gate says yes, the live model keeps serving and the candidate waits in the registry with its scores on record.

Strip the words "model" and "training" out of that paragraph and read what remains: an artifact is built; the artifact is a candidate; the candidate carries evidence; a gate reads the evidence; promotion is the gate's verdict, not the builder's. That is release engineering, whole. A container image, a Terraform plan, a frontend bundle, a schema migration, a fine-tuned model: five artifact classes, one shape. The Cognition arc named the shape; the foundational tier is where it belongs, because every domain above the foundation ships something, and everything shipped should pass through a gate it did not write.

The principle to commit to before the mechanics: the candidate never ships itself.

§II — Foundations

Four parts make a promotion pipeline, whatever the artifact.

The candidate is an immutable, identified artifact: an image digest, a versioned bundle, a registered model, a plan file. Mutability disqualifies. A candidate that can change after its evidence was gathered carries evidence about something else.

The evidence is the accumulated record attached to the candidate: tests passed, scan results, eval scores, staging soak time, canary metrics. Evidence accretes as the candidate moves; each environment it survives adds to the record. The DevOps canon's telemetry discipline exists to make this evidence automatic rather than ceremonial.

The gate is the decision point that reads evidence against criteria someone chose deliberately and wrote down. A gate has three honest outputs: promote, refuse, or escalate to a human. A gate whose criteria live in a reviewer's head is a mood, not a gate.

The graded exposure is promotion's refusal to be binary. Between zero traffic and full traffic sits the canary: a small, real slice of production whose telemetry is the final and best evidence. The SRE canon treats canarying as the standard answer to the cost of a bad push; the gate pattern treats the canary as the last gate's evidence-producer, watched by the same criteria that govern every earlier stage.

§III — Mechanism

The environment ladder is a gate sequence

Build, test, staging, canary, full production: read the familiar ladder as a sequence of gates, each consuming the evidence the prior rung produced. The build gate reads compilation and unit results. The staging gate reads integration runs and soak. The canary gate reads live telemetry against the incumbent's baseline. Each rung answers one question: has this candidate earned the next, larger audience? The ladder's value is exactly the gate discipline; an environment a candidate passes through without a criteria-bearing gate is a hallway, not a stage.

Criteria are written, versioned, and owned

The continuous-training gate held three written checks. The release gate generalizes the pattern: criteria live in version control beside the pipeline, criteria changes are reviewed like code, and each gate names its owner — the human accountable for what the gate admits. Write the criteria as comparisons against the incumbent where possible (error rate within tolerance of current, latency p99 no worse than current, business metric held), because the incumbent is the only baseline that is always current. Absolute thresholds rot; relative ones travel.

The canary is shadow evaluation with stakes

The shadow-eval discipline ran the model candidate on copied traffic where outputs reached no user. Release engineering's canary runs the candidate on a real slice with stakes attached, which is why the slice starts small: one percent, then ten, then full, each step a gate-read of the canary's telemetry against the incumbent fleet's. The two patterns are one family. Shadow answers "does the candidate behave" with zero blast radius; canary answers "does the candidate behave under reality" with bounded blast radius. Use shadow where outputs can be copied safely; graduate to canary for the evidence shadow cannot produce.

Refusal and rollback are outcomes, not failures

A gate that never refuses is decoration. The registry keeps refused candidates with their evidence, and the refusal record is the cheapest engineering education the team owns: every refused candidate is a defect caught at the gate price instead of the incident price. Rollback is the gate pattern run in reverse: the incumbent is re-promoted over a failing release, which is only cheap if releases are immutable and the prior artifact is one verdict away. The frontend arc's rollback-surface lesson built exactly this for bundles; the shape is artifact-agnostic.

§IV — Worked Example

A team ships a payments service. The candidate is image sha256:a41f…, built from a reviewed merge. Gate 1 (build): unit suite green, dependency scan clean, image signed; promote to staging. Gate 2 (staging): integration suite green, twelve-hour soak with zero error-budget burn; promote to canary. Gate 3 (canary, 2% of traffic, 60-minute window): error rate within 0.1% of incumbent, p99 latency within 5%, decline-rate on payment authorizations no worse than incumbent. At minute 40 the decline-rate check trips: the candidate declines 0.4% more authorizations than the fleet. The gate refuses; traffic returns to the incumbent automatically; the candidate lands in the registry with the canary evidence attached.

The defect: a stricter timeout against the card-network client, correct in staging where the network is fast, wrong against the long tail of real authorizations. No incident was declared, because no incident occurred; two percent of traffic saw an extra 0.4% declines for forty minutes, and the gate caught what the suite could not. The fix re-enters the same ladder as a new candidate the following day. The on-call engineer slept through the whole release. That sentence is the pattern's value, stated plainly.

§V — Connection to Prior Lessons

The 2026-06-08 continuous-training lesson built this lesson's source: candidate, evidence, gate, graded exposure, for model artifacts. This lesson removes the artifact-class assumption and lands the shape on the foundational tier, where Cognition, Trust, Chain, and Adversarial-Markets all inherit it: a model, a policy bundle, an indexer release, and a strategy parameter set are all candidates, and none of them ships itself. The 2026-05-31 frontend release-engineering lesson is the shape's frontend instance: atomic rollouts and canary slicing are gates and graded exposure wearing web clothes. The 2026-06-01 eval lesson remains the evidence-producer; gates are only as good as the evidence they read.

§VI — Harmonic Ripples

This lesson ripples into AIOps-Magus (model promotion is now one instance of the team's standard release ladder rather than a parallel bespoke process), QuantOps-Magus (strategy-parameter changes enter production through a gate that reads paper-mode and shadow evidence, never by hand-edit, which is the Capital-Zero posture expressed as pipeline), SecOps-Magus (gates are admission control for the software supply chain; signature and provenance checks are gate criteria like any other), and MLOps-Virtual-Machinist (the registry habit: every candidate recorded with its evidence, promoted or refused, nothing shipped from a laptop).

§VII — Closing

Inventory what your system shipped this month. For each artifact class, name the gate it passed, the written criteria the gate read, and the owner who answers for it. Where you find an artifact class that ships on a human's say-so with no recorded evidence, you have found your next incident's front door. Build the gate before the incident builds it for you. Examine each gate well; the candidate never ships itself.

Related