Hedronite · Curriculum · DevOps · Rust · Mon 2026-11-02

Keep the old release until the new one proves it A readiness-gated deploy that rolls back on its own

Two ports, one deadline, one rename, and a log that never says promoted for a release that failed.

Lesson Class: Curriculum · B-DevOps · rung A
Tested with: Rust 1.98 · axum 0.8.9 · tokio 1
Lab: lab/ · sh check.sh (6 checks) · no network
Source: Kubernetes Deployments docs · The Site Reliability Workbook ch. 16 · POSIX rename()
Test the Release
Readiness 200 before the deadline, and the process still alive.
Swap Atomically
Write a temp file, then rename it over the pointer.
Say What Happened
A failed release ends in rolled_back, with the reason.
The old release keeps the lights on until the new one earns the switch.

B-DevOps · A · arc: self-healing-edge · langs: rust, go · CKA-workloads-rollout, DOP-1.4

By Evan Eugene Pincham.

Concept

A release is one specific build of a service, running as a process. A deploy replaces the running release with a new one. The usual mistake is to treat a deploy as a command that succeeds when the new process starts. A better model is that a deploy is a hypothesis: "the new release can do the job." Hypotheses need a test, and while the test runs the old release has to keep serving.

The test used here has two parts, both of which must hold:

  1. Readiness. The candidate answers GET /readyz with status 200 before a deadline. Readiness is a service's own statement that it can take traffic right now. It is different from liveness, which only says the process can answer at all. A service that is still warming up is alive and not ready; it returns 503 on /readyz.
  2. Still alive. The candidate process has not exited. A release that crashes at boot never listens, so "no answer yet" and "dead" must be told apart. Otherwise you wait out the whole deadline for a corpse.

If the test passes, the deployer promotes the candidate: it records the new release as the live one, then asks the old process to stop. If the test fails, the deployer kills the candidate, checks that the old release still answers, and logs rolled_back with the reason. It never logs promoted for a release that failed. An honest log is part of the design. A deploy tool that reports success it did not achieve is worse than one that crashes.

The pointer to the live release is one small file named current, holding <port> <pid> <version>. Anything that needs to know which release is live reads that file, so updating it must never be observable half-done. The deployer writes the new line to current.tmp and then calls rename to move it over current. POSIX specifies rename() so that "a link named new shall remain visible to other threads throughout the renaming operation and refer either to the file referred to by new or old before the operation began." A reader gets the old line or the new line, never half of one. The temp file must be on the same filesystem as the target: when the two names are on different file systems, POSIX allows rename to fail with EXDEV rather than move the file. Writing it next to current is the simple way to guarantee that. This lab does not call fsync, so it does not claim the swap survives a power cut; it claims only that no reader sees a torn line.

Kubernetes expresses the same gate declaratively. A Deployment with a readinessProbe and maxUnavailable: 0 will not remove old Pods until new ones are Ready. progressDeadlineSeconds marks a stalled rollout with the reason ProgressDeadlineExceeded. The Deployments documentation is explicit about what happens next: "Kubernetes takes no action on a stalled Deployment other than to report a status condition with reason: ProgressDeadlineExceeded." Rolling back is done by kubectl rollout undo or by a tool above the controller. The program below is that missing step in miniature: it gates, and it also acts on the verdict.

The scope is deliberately small. Everything runs on one machine, on two local ports, and nothing routes client traffic; the current file stands in for the router. The gate also judges a release only once, at the moment it becomes ready. A release that passes readiness and fails ten minutes later is not caught here. That is the restart supervisor's job from the earlier lesson in this arc, and a later lesson in this arc joins the two.

Why bother to make rollback cheap and automatic? The Site Reliability Workbook, chapter 16, "Canarying Releases", says smaller artifacts are cheaper to roll back (section "Release Engineering Principles") and that having no rollback option prolongs user impact (section "Balancing Release Velocity and Reliability"). In its blue/green discussion, rollback is "a trivial reversal of the router change". The current pointer is that router in the smallest form.

Worked example

The service under deploy is edge-svc, a small HTTP service written in Rust (axum 0.8.9 on tokio 1) that the earlier lessons in this arc built. It serves /healthz (liveness), /readyz (readiness, 503 until a warm-up timer ends), and /version (which release is answering). Two environment variables let us stage bad releases: READY_AFTER_MS sets the warm-up time, and CRASH_ON_START makes the process exit with code 78 before it listens. It stops cleanly on SIGTERM, the signal an orchestrator sends to ask a process to finish and exit.

The deployer is a second Rust program, deployer <svc-binary> <new-port> <new-version> [ready-deadline-ms]. It runs old and new releases side by side on two ports. First it reads current and starts the candidate on the new port:

let mut child = Command::new(svc)
    .env("PORT", new_port).env("VERSION", new_ver)
    .stdin(Stdio::null()).stdout(Stdio::null()).stderr(Stdio::null())
    .spawn().expect("spawn candidate");
let pid = child.id().unwrap_or(0);
log("candidate_started", new_ver, &format!("pid {pid} port {new_port}"));

The gate, the probe, the swap

The gate is a loop that polls every 100 ms and ends in one of three ways: ready, exited, or out of time.

// The gate: readiness 200 before the deadline, and the process still alive.
let start = Instant::now();
let verdict: Result<(), String> = loop {
    if let Ok(Some(code)) = child.try_wait() {
        break Err(format!("candidate exited with {code} before ready"));
    }
    if status(new_port, "/readyz").await == Some(200) { break Ok(()); }
    if start.elapsed() >= deadline {
        break Err(format!("not ready after {} ms", deadline.as_millis()));
    }
    tokio::time::sleep(Duration::from_millis(100)).await;
};

Child::try_wait in tokio::process checks whether the child has already exited without blocking: Ok(None) means still running, Ok(Some(status)) means it is gone. That is the "still alive" half of the test. The order matters: check for exit first, then probe, then check the clock, so a crashed candidate is reported as a crash and not as a timeout.

The readiness probe is hand-written HTTP/1.0 over a raw TCP socket, so the deployer needs no HTTP client crate. HTTP/1.0 is convenient because the server closes the connection after the response, so read_to_end terminates without parsing Content-Length. The first line of the response is HTTP/1.0 200 OK (or similar), and the status code is its second whitespace-separated field:

/// Minimal HTTP/1.0 GET: returns the status code, or None if nothing answered.
async fn status(port: &str, path: &str) -> Option<u16> {
    let mut s = tokio::net::TcpStream::connect(format!("127.0.0.1:{port}")).await.ok()?;
    s.write_all(format!("GET {path} HTTP/1.0\r\nHost: localhost\r\n\r\n").as_bytes()).await.ok()?;
    let mut buf = Vec::new();
    s.read_to_end(&mut buf).await.ok()?;
    String::from_utf8_lossy(&buf).split_whitespace().nth(1)?.parse().ok()
}

Every failure (connection refused, closed early, garbage) becomes None, and the gate treats None as "not ready yet". That is the right default for a gate: anything other than a clean 200 keeps the old release in charge.

On success, the swap is write-temp-then-rename, as described above:

fn write_current(port: &str, pid: u32, version: &str) -> std::io::Result<()> {
    std::fs::write("current.tmp", format!("{port} {pid} {version}\n"))?;
    std::fs::rename("current.tmp", "current")
}

The Ok arm swaps the pointer first, then retires the old release, then logs. The order is deliberate: the new release is recorded as live before the old one is told to stop, so there is no moment where current names a process that is already going away. One gap remains and is worth knowing: the deployer sends SIGTERM and exits without waiting for the old process to finish its in-flight requests. Here the old release stops quickly because with_graceful_shutdown lets axum finish open connections, but nothing checks that it did.

Ok(()) => {
    write_current(new_port, pid, new_ver).expect("write current");
    send_term(old_pid);
    log("promoted", new_ver, &format!("replaced {old_ver} on port {old_port}"));
    // Let the candidate outlive the deployer: it is now the release.
    std::mem::forget(child);
}

On failure, the candidate is killed and reaped. In tokio, Child::kill sends SIGKILL on Unix and then waits for the child, so its exit status is collected and no zombie remains; the wait that follows is redundant but harmless, and it keeps the intent readable. The deployer does not trust that the old release is fine; it asks it. By default tokio does not kill a child when the Child handle is dropped (kill_on_drop is false), so the explicit kill is required.

Err(reason) => {
    let _ = child.kill().await;
    let _ = child.wait().await;
    let still = status(old_port, "/readyz").await == Some(200);
    log("rolled_back", new_ver,
        &format!("{reason}; {old_ver} still serving on port {old_port}: {still}"));
    std::process::exit(1);
}

The process exit code is part of the contract: 0 for promoted, 1 for rolled back. A CI job or a shell script above the deployer can branch on it.

The log is a record

Each decision is one JSON object on one line with exactly four keys: ts (Unix seconds), event, version, detail.

/// One JSON line per decision. Fixed keys: ts, event, version, detail.
fn log(event: &str, version: &str, detail: &str) {
    let ts = SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs();
    println!(r#"{{"ts":{ts},"event":"{event}","version":"{version}","detail":"{detail}"}}"#);
}

A fixed schema is what makes a log queryable later: any line can be read without guessing which fields exist. One limit to know: this log function builds JSON by string formatting and does not escape " or \. The reasons produced here contain neither, and the checker's schema test would catch a line that broke. A production version should use a JSON serializer. A later Data lesson in this track queries this log as-is.

What a run looks like

The lab's reference solution was run inside a container with no network, deploying three releases in a row onto a live v1: v2 (ready after 300 ms, deadline 3000 ms), v3 (ready after 60 s, deadline 1500 ms), and v4 (crashes at boot). This is the log it produced:

{"ts":1793632142,"event":"candidate_started","version":"v2","detail":"pid 997 port 18092"}
{"ts":1793632142,"event":"promoted","version":"v2","detail":"replaced v1 on port 18091"}
{"ts":1793632143,"event":"candidate_started","version":"v3","detail":"pid 1034 port 18093"}
{"ts":1793632144,"event":"rolled_back","version":"v3","detail":"not ready after 1500 ms; v2 still serving on port 18092: true"}
{"ts":1793632145,"event":"candidate_started","version":"v4","detail":"pid 1068 port 18094"}
{"ts":1793632145,"event":"rolled_back","version":"v4","detail":"candidate exited with exit status: 78 before ready; v2 still serving on port 18092: true"}

Read it as a story. v2 passed the test and replaced v1. v3 never became ready, so after 1500 ms it was killed, and the line records that v2 was still answering. v4 died with exit status 78 before it could be ready, and the reason says so. Neither failed release appears next to the word promoted.

Lab

You start from a deployer that does what many first deploy scripts do: start the candidate, sleep 300 ms, switch over, report success. It does not check anything.

The starter's src/deployer.rs has this at its core:

tokio::time::sleep(Duration::from_millis(300)).await; // TODO: replace with a readiness gate
std::fs::write("current", format!("{new_port} {pid} {new_ver}\n")).expect("write current");
send_term(old_pid);
log("promoted", new_ver, &format!("replaced {old_ver} on port {old_port}"));
std::mem::forget(child);

Setup. Copy lab/starter/ somewhere you can edit, and copy lab/check.sh into it. The starter holds the edge-svc service (src/main.rs, already finished), the starter src/deployer.rs, and a Cargo.lock pinning axum 0.8.9 and tokio 1. The checker needs Rust (1.98 was used here), curl, jq and pkill, and it builds with cargo build --offline, so fetch the crates once while you have a network:

cargo fetch
sh check.sh

The lab's own gate ran the checker inside a Rust 1.98 (Debian bookworm) container with curl, jq and procps installed, the crates fetched in advance, and networking disabled. Any container with those tools works the same way; nothing in the checker leaves 127.0.0.1.

What the starter prints. The starter passes the three checks that a "promote anything" deployer cannot get wrong (it builds, a good release still gets promoted, and its log lines have the right keys) and fails the three that need a gate. This is the output from the lab's CI gate for the starter tree:

RESULT build PASS
good deploy: exit=0 serving_18092=v2 old_18091=000 current=18092 977 v2
RESULT good_release_promoted PASS
never-ready deploy: exit=0 took=0s serving_18092= candidate_18093=200 current=18093 1014 v3
RESULT never_ready_rolled_back FAIL
crash-at-boot deploy: exit=0 serving_18092= current=18094 1044 v4
RESULT crash_rolled_back FAIL
--- deploy.log
{"ts":1793632152,"event":"candidate_started","version":"v2","detail":"pid 977 port 18092"}
{"ts":1793632152,"event":"promoted","version":"v2","detail":"replaced v1 on port 18091"}
{"ts":1793632153,"event":"candidate_started","version":"v3","detail":"pid 1014 port 18093"}
{"ts":1793632153,"event":"promoted","version":"v3","detail":"replaced v2 on port 18092"}
{"ts":1793632153,"event":"candidate_started","version":"v4","detail":"pid 1044 port 18094"}
{"ts":1793632154,"event":"promoted","version":"v4","detail":"replaced v3 on port 18093"}
RESULT log_is_honest FAIL
RESULT log_schema PASS

Look at what the failure lines show. After the never-ready deploy, serving_18092= is empty: the good v2 was shut down and nothing took its place. After the crash-at-boot deploy, current points at 18094 1044 v4, a process that is already dead. The log says promoted for both. That is the outage this lab exists to prevent.

Your task. Edit only src/deployer.rs.

  1. Make child mutable and replace the 300 ms sleep with a loop that polls the candidate's /readyz every 100 ms. Write the probe as a small function that returns the HTTP status code, using tokio::net::TcpStream and the HTTP/1.0 request shown above.
  2. In the same loop, call child.try_wait() each time round so a crashed candidate ends the wait immediately.
  3. Take the deadline from the fourth argument (milliseconds, default 3000) and end the loop with an error when it passes.
  4. Only on success: write current.tmp, rename it over current, send SIGTERM to the old pid, log promoted.
  5. On failure: kill and wait on the candidate, probe the old release's /readyz, log rolled_back with the reason and whether the old release still answers, and exit with a non-zero code. Do not touch current.

Run sh check.sh until the output ends with every check passing:

RESULT build PASS
RESULT good_release_promoted PASS
RESULT never_ready_rolled_back PASS
RESULT crash_rolled_back PASS
RESULT log_is_honest PASS
RESULT log_schema PASS

The checker's schema test is one jq expression over the whole log, which is worth reading as a pattern for validating any line-delimited JSON record:

jq -e -s 'all(.[]; has("ts") and has("event") and has("version") and has("detail"))' deploy.log

-s slurps all lines into one array, all(...) checks every element, and -e turns a false result into a non-zero exit status.

Stretch questions. (a) The log function would emit invalid JSON if a reason contained a quote. Make the checker catch it by adding a release whose failure text includes one, then fix log. (b) The rolled_back line says whether the old release is still serving, but the deployer exits 1 either way. Should "rolled back and the old release is also down" be a different event or exit code? Argue for one, and note what a reader of the log gains.

Other paths

The gate-then-act idea does not depend on Rust or on a hand-written program. Two other forms are worth comparing: the same deployer in Go, and the same gate written as Kubernetes configuration, where the rollback step turns out to be missing.

Go (sketch, not run in this lab)

The same deployer in Go has the same shape. What changes:

// Sketch only. Not compiled or run in this lab.
func gate(cmd *exec.Cmd, port string, deadline time.Duration) error {
	exited := make(chan error, 1)
	go func() { exited <- cmd.Wait() }()
	client := &http.Client{Timeout: 500 * time.Millisecond}
	stop := time.Now().Add(deadline)
	for {
		select {
		case err := <-exited:
			return fmt.Errorf("candidate exited (%v) before ready", err)
		default:
		}
		if resp, err := client.Get("http://127.0.0.1:" + port + "/readyz"); err == nil {
			resp.Body.Close()
			if resp.StatusCode == 200 {
				return nil
			}
		}
		if time.Now().After(stop) {
			return fmt.Errorf("not ready after %d ms", deadline.Milliseconds())
		}
		time.Sleep(100 * time.Millisecond)
	}
}

func promote(port string, pid int, version string) error {
	line := fmt.Sprintf("%s %d %s\n", port, pid, version)
	if err := os.WriteFile("current.tmp", []byte(line), 0o644); err != nil {
		return err
	}
	return os.Rename("current.tmp", "current")
}

On failure you would call cmd.Process.Kill() and log rolled_back, as in the Rust version. The gate-then-act structure carries over unchanged. Only the plumbing differs.

The declarative version: a Kubernetes Deployment

Kubernetes is a system that runs containers across machines. In it, a Deployment declares how many copies (Pods) of a service should run and how to replace them when the image changes. The same gate can be written as configuration:

# Sketch only. No cluster is used in this lab; this was not applied anywhere.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: edge-svc
spec:
  replicas: 2
  progressDeadlineSeconds: 60
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels: { app: edge-svc }
  template:
    metadata:
      labels: { app: edge-svc }
    spec:
      containers:
        - name: edge-svc
          image: example/edge-svc:v3
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 1

How the pieces map to the Rust program:

This labKubernetes
poll /readyz for a 200readinessProbe on /readyz
old release serves until the test passesmaxUnavailable: 0: old Pods are not removed until new ones are Ready
a second port for the candidatemaxSurge: 1: one extra Pod may exist during the rollout
the deadline argumentprogressDeadlineSeconds
log rolled_back and restorereport ProgressDeadlineExceeded only; undo is separate

The last row is the important one. The controller reports the stall and leaves the Deployment as it is. You watch a rollout and reverse it with two commands:

kubectl rollout status deployment/edge-svc
kubectl rollout undo deployment/edge-svc

rollout status waits for the rollout and reports whether it finished or stalled. rollout undo returns the Deployment to the previous revision. Putting these in a script or pipeline, so that a stalled status triggers the undo with no human, is the same job deployer does here. This lab uses no cluster; the YAML is shown so you can compare the two forms of the same idea.

Citations