Hedronite · Curriculum · DevOps · Rust · Mon 2026-11-09

A release is on probation until the window closes A gated rollback in Rust

Passing readiness once is not the same as being good. Count the exits, close the window, then confirm.

Lesson Class: Curriculum · B-DevOps · rung A
Tested with: Rust 1.98 · serde_json 1
Lab: lab/ · sh check.sh (8 RESULT lines) · CPU only, no network
Source: Kubernetes Deployments · Pod Lifecycle · The Site Reliability Workbook ch. 16 · Armstrong thesis ch. 6

Lab download. Starter is the tree you edit. Worked answer is a separate file; open it after the starter passes. Rust 1.85 or newer, tar, and jq. One online cargo vendor vendor. The zip has no vendored crates. All lab requirements.

Soak Window
A promoted release is not good until 30 s pass with its restart budget intact.
Restart Budget
Two restarts are allowed; the third exit inside the window rolls the deploy back.
Last Confirmed
The rollback target is the last release that finished a window, never a claimed previous.
Ready once is a guess. Ready for the whole window is evidence.

B-DevOps · A · arc: self-healing-edge · langs: rust, go · CKA-workloads-rollout, DOP-1.4

By Evan Eugene Pincham.

Concept

Some vocabulary first, because the rest of the lesson leans on it.

A readiness check asks one question at one instant: is the candidate answering right now? A release can say yes, be promoted, and then crash and restart every few seconds. The check passed, and the deploy is still bad. Readiness at one instant is a weak test.

The remedy in this lesson is a time-boxed probation. After promotion the candidate enters a soak window, a fixed span of time during which the deployer watches it. The deployer also holds a restart budget, the number of restarts it will tolerate inside that window. This lesson uses a 30 000 ms window and a budget of 2, so two restarts are allowed and the third exit inside the window fails the deploy.

Three rules follow from that, and the program enforces all of them.

  1. A promoted release is not "good" yet. It becomes confirmed, and becomes the rollback target, only when the window closes with the budget intact.
  2. If the budget is exceeded, the deployer rolls back to the last confirmed release (last_good) and writes down why.
  3. The rollback target comes from the deployer's own state. It never comes from a previous field on an incoming event, because that field is a claim made by whoever wrote the event, and it can be stale. It is also never a release that was itself rolled back.

The program also keeps a ledger: an append-only record of every action the deployer took. The ledger is JSONL, meaning one JSON object per line, so each line can be read on its own with jq or loaded into a table later. Every line carries ts (a timestamp in milliseconds), event, and version. Other fields depend on the event.

Honesty line: this is a deploy decision kernel over a fixture plan. A plan is a JSONL file of events, each with a millisecond ts. The program uses the CPU only. It starts no processes, reads no clock, and talks to no cluster. Time moves only when an event's ts says so. What is real is the soak window, the restart budget, the rollback target rule, and the honest ledger. What is simulated is actually starting releases and probing them.

The same two knobs exist in Kubernetes, separately. A Pod is the smallest unit Kubernetes runs, one or more containers together. A Deployment is the Kubernetes object that manages a set of Pods and rolls them from one version to the next. The Deployment documentation describes .spec.minReadySeconds as "an optional field that specifies the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available." That is probation. It defaults to 0, "the Pod will be considered available as soon as it is ready", which is the weak test described above. The second knob, progressDeadlineSeconds, covers stalls. Neither knob is a restart budget with a rollback attached. See Other paths.

The idea of "N restarts inside a window" is older than Kubernetes. Joe Armstrong's thesis describes supervisor restart intensity written as {one_for_one,5,1000}: if a supervised process restarts more than 5 times in 1000 seconds, the supervisor itself gives up and fails. The release-level version here applies the same cap to a release instead of a worker.

The Site Reliability Workbook makes the matching point about canaries: a canary has to last long enough, and be large enough, to be representative, and ending it after a handful of requests gives no useful signal. A soak window is the duration half of that advice.

Worked example

The kernel is a small state machine, a program that is always in exactly one named state and moves between states only when an event arrives. (A decision kernel is the part of a deployer that decides what to do, kept apart from the code that would start and stop processes.) Deploy holds four states: Serving, Soaking, Confirmed, RolledBack. It also holds current, last_good, candidate, the window start, and two counters. Four events drive it:

Here is the whole program.

use serde::Deserialize;
use serde_json::{json, Value};
use std::{env, fs, process::exit};

const SOAK_MS: u64 = 30_000;
const MAX_RESTARTS: u32 = 2;

#[derive(Deserialize)]
struct Event {
    ts: u64,
    event: String,
    version: Option<String>,
}

#[derive(Clone, Copy, PartialEq)]
enum State {
    Serving,
    Soaking,
    Confirmed,
    RolledBack,
}

struct Deploy {
    current: String,
    last_good: String,
    candidate: String,
    state: State,
    start: u64,
    restarts: u32,
    exits: u32,
}

impl Deploy {
    fn new() -> Self {
        Deploy {
            current: String::new(),
            last_good: String::new(),
            candidate: String::new(),
            state: State::Serving,
            start: 0,
            restarts: 0,
            exits: 0,
        }
    }

    fn step(&mut self, e: &Event, out: &mut Vec<Value>) {
        // The soak window closes when time passes its end with the candidate still alive.
        if self.state == State::Soaking && e.ts >= self.start + SOAK_MS {
            self.last_good = self.candidate.clone();
            self.state = State::Confirmed;
            out.push(json!({"event":"confirmed","ts":e.ts,"version":self.candidate,"restarts":self.restarts}));
        }
        let version = e.version.clone().unwrap_or_default();
        match e.event.as_str() {
            "serving" => {
                self.current = version.clone();
                self.last_good = version;
                self.state = State::Serving;
            }
            "promoted" => {
                self.candidate = version.clone();
                self.current = version;
                self.state = State::Soaking;
                self.start = e.ts;
                self.restarts = 0;
                self.exits = 0;
            }
            "exited" => {
                if self.state == State::Soaking && version == self.candidate {
                    self.exits += 1;
                    if self.exits <= MAX_RESTARTS {
                        self.restarts += 1;
                        out.push(json!({"event":"restarting","ts":e.ts,"version":version,"attempt":self.restarts,"max":MAX_RESTARTS}));
                    } else {
                        // Roll back to the last confirmed release, never to a release that was itself rolled back.
                        self.current = self.last_good.clone();
                        self.state = State::RolledBack;
                        out.push(json!({"event":"rolled_back","ts":e.ts,"version":version,"from":version,"to":self.last_good,"reason":"restart_budget_exceeded","exits_in_window":self.exits,"max":MAX_RESTARTS,"window_ms":SOAK_MS}));
                    }
                } else if self.state == State::Confirmed && version == self.current {
                    out.push(json!({"event":"restarted_post_soak","ts":e.ts,"version":version}));
                }
            }
            _ => {}
        }
    }

    fn deploy(&self) -> &'static str {
        match self.state {
            State::Serving => "serving",
            State::Soaking => "soaking",
            State::Confirmed => "confirmed",
            State::RolledBack => "rolled_back",
        }
    }
}

fn run(path: &str) -> Result<(Deploy, Vec<Value>), String> {
    let text = fs::read_to_string(path).map_err(|e| format!("{path}: {e}"))?;
    let mut d = Deploy::new();
    let mut out = Vec::new();
    for (i, line) in text.lines().enumerate() {
        if line.trim().is_empty() {
            continue;
        }
        let e: Event = serde_json::from_str(line).map_err(|e| format!("{path}:{}: {e}", i + 1))?;
        d.step(&e, &mut out);
    }
    Ok((d, out))
}

fn main() {
    let args: Vec<String> = env::args().collect();
    if args.len() != 3 || !(args[1] == "apply" || args[1] == "report") {
        eprintln!("usage: soak apply|report PLAN");
        exit(2);
    }
    let (d, out) = match run(&args[2]) {
        Ok(v) => v,
        Err(m) => {
            eprintln!("soak: {m}");
            exit(2);
        }
    };
    if args[1] == "apply" {
        for a in out {
            println!("{a}");
        }
    } else {
        println!("soak_ms: {SOAK_MS}");
        println!("max_restarts: {MAX_RESTARTS}");
        println!("current: {}", d.current);
        println!("last_good: {}", d.last_good);
        println!("candidate: {}", d.candidate);
        println!("candidate.restarts: {}", d.restarts);
        println!("deploy: {}", d.deploy());
    }
}

The binary has two commands. soak apply PLAN prints the ledger, one JSON line per action. soak report PLAN prints the final state: soak_ms, max_restarts, current, last_good, candidate, candidate.restarts, and deploy, which is one of serving, soaking, confirmed, rolled_back. A missing or unreadable plan file exits with code 2.

Trace the storm by hand. This is the plan the checker calls flap.plan. The release v2 exits three times inside its window.

{"ts":0,"event":"serving","version":"v1"}
{"ts":1000,"event":"promoted","version":"v2","previous":"v1"}
{"ts":3000,"event":"exited","version":"v2"}
{"ts":6000,"event":"exited","version":"v2"}
{"ts":9000,"event":"exited","version":"v2"}

The reference run on the lab image, with the network disabled, printed:

flap apply: restarting=2 rolled_back=1 to=v1 reason=restart_budget_exceeded
flap report: current=v1 last_good=v1 deploy=rolled_back

Three details in the code decide the harder cases.

The ledger schema is the Data half of the lesson. Every line, whatever its event, has a numeric ts, a string event, and a string version. On a rolled_back line version names the release that failed, and to names the release restored. A query tool such as jq, or a table loader later, can read the file with no special parsing. The checker tests the schema with this filter, which prints any line that breaks it:

jq -c 'select((.ts|type)!="number" or (.event|type)!="string" or (.version|type)!="string")'

Lab

You get a starter project with the same layout as the reference: Cargo.toml, src/main.rs, a vendored vendor.tgz of serde and serde_json, and check.sh. The checker is POSIX sh and needs cargo, tar, and jq. It builds offline, so no network is needed. Run it from the directory that holds check.sh:

sh check.sh

It prints one RESULT line per check, eight in all, each ending in PASS or FAIL: build, storm_rolls_back, survivor_confirmed, no_confirm_while_soaking, window_closes, rollback_target_last_good, ledger_schema, and missing_file_exit_2. The checker writes its own fixture plans (flap, stable, soaking, slowburn, chain) and runs soak against them.

The starter has three planted bugs. On it, four checks fail (the CI gate log has the full starter run):

RESULT survivor_confirmed FAIL
RESULT no_confirm_while_soaking FAIL
RESULT window_closes FAIL
RESULT rollback_target_last_good FAIL

The bugs, described by behavior:

  1. Promotion counts as good at once. On promoted, the starter sets last_good to the candidate and moves straight to Confirmed. A release that has only just been promoted becomes the rollback target and shows as confirmed. After the storm in flap.plan, the starter's gate run reported flap report: current=v1 last_good=v2 deploy=rolled_back, which names the failed release as the last good one. In the reference, the promoted arm sets State::Soaking and leaves last_good alone.
  2. The window is ignored. Exits count forever, so a slow, healthy release eventually trips the budget. The fix is the block at the top of step, which closes the window when ts reaches the end:
        // The soak window closes when time passes its end with the candidate still alive.
        if self.state == State::Soaking && e.ts >= self.start + SOAK_MS {
            self.last_good = self.candidate.clone();
            self.state = State::Confirmed;
            out.push(json!({"event":"confirmed","ts":e.ts,"version":self.candidate,"restarts":self.restarts}));
        }
  1. **The rollback target is the event's previous.** The starter reads previous from the plan, so on chain.plan its gate run printed chain apply: rolled_back=2 last_to=v2, rolling back to a release that had already failed. The fix is to take the target from self.last_good, as in the reference:
                        // Roll back to the last confirmed release, never to a release that was itself rolled back.
                        self.current = self.last_good.clone();
                        self.state = State::RolledBack;

Work through the checks in this order: no_confirm_while_soaking, survivor_confirmed, window_closes, rollback_target_last_good. After each fix, re-run sh check.sh and read the diagnostic lines the checker prints above each RESULT. When all eight lines read PASS, the reference output looks like this:

RESULT build PASS
RESULT storm_rolls_back PASS
RESULT survivor_confirmed PASS
RESULT no_confirm_while_soaking PASS
RESULT window_closes PASS
RESULT rollback_target_last_good PASS
RESULT ledger_schema PASS
RESULT missing_file_exit_2 PASS

Then extend it. Change MAX_RESTARTS to 0 and predict, before running, which plans change outcome and why. Then write your own plan where v2 is confirmed, v3 is promoted and storms, and v2 is the correct rollback target.

Other paths

Kubernetes. A Deployment already has both knobs, as separate fields. minReadySeconds is the probation: a Pod must be ready without crashing for that many seconds before it counts as available. The default of 0 means a Pod counts as available as soon as it is ready. progressDeadlineSeconds handles the case where a rollout stalls. When it expires, the Deployment gains a condition with type: Progressing, status: "False", and reason: ProgressDeadlineExceeded. Rolling back is a separate act. The Deployment documentation shows kubectl rollout undo deployment/nginx-deployment, with --to-revision to choose a specific revision. The documentation also says that for a stalled Deployment, Kubernetes takes no action other than to report a status condition.

So the declarative part is the probation and the deadline: you state them in the Deployment, and Kubernetes evaluates them. What is still missing is a restart budget wired to a rollback. Something above the Deployment has to read the condition and run the undo. That something can be a person, or a controller or pipeline step that you write. Seen that way, the kernel in this lesson is the decision step such a controller would call. Separately, the Pod Lifecycle documentation notes that a Pod which keeps failing to start may show CrashLoopBackOff in the Status field of some kubectl commands. That is a display field, not a Pod phase, so it is a symptom you read, not a state you can rely on to trigger anything.

Go. The same decision step can be written in Go as a function that takes the current deploy state and one event and returns the new state plus any ledger lines. The branches are the same: close the window, then promote, exit, or ignore. The Go version is a sketch for comparison. It was not compiled or run for this lesson.

jq. Because the ledger is JSONL, questions about it are one-line filters. The schema check in the worked example is one. A table loader can read the same files without any conversion.

Citations