Hedronite · Curriculum · SWE · Rust · Sun 2026-11-08

Restart only the worker that failed A one-for-one supervisor in Rust

Independent workers share a supervisor, not a fate. Empty siblings_touched. Ensemble never healthy after gave_up.

Lesson Class: Curriculum · B-SWE · rung A
Tested with: Rust 1.98 · serde_json 1
Lab: lab/ · sh check.sh (7 RESULT lines) · CPU only, no network
Source: Armstrong thesis §5.2.3 · Erlang/OTP Supervisor Behaviour · Kubernetes Pod Lifecycle · ReplicaSet · CKA Workloads

Lab download. Starter is the tree you edit. Worked answer is a separate file; open it after the starter passes. Rust 1.85 or newer, tar, and jq. One online cargo vendor vendor. The zip has no vendored crates. All lab requirements.

One-for-One
An independent worker's exit restarts only that worker.
Per-Worker Cap
Three restarts, then gave_up for that worker alone.
Honest Ensemble
The pair is never healthy if any worker gave up.
A shared supervisor is not a shared fate.

B-SWE · A · arc: self-healing-edge · langs: rust, gleam · CKA-WORKLOADS-self-healing

By Evan Eugene Pincham.

Concept

A supervisor is a program whose only job is to watch other programs, called workers or children, and start them again when they fail. Self-healing systems are built from this idea. A worker crashes, the supervisor notices, and the service comes back without a person logging in.

With one worker, the supervisor has one decision: restart it or stop trying. With two or more workers there is a second decision, called the restart strategy: when one worker dies, which workers get restarted?

Take a service made of two independent workers. api is an HTTP server. scorer is a sidecar, a helper process that runs next to the main one and does a separate job, here computing scores for requests. They share a machine and nothing else. If scorer crashes and the supervisor also restarts api, every in-flight HTTP request is dropped for no reason. The right rule is to restart only the worker that failed.

The rule in Erlang/OTP. Erlang is a programming language built for systems that must keep running through failures. OTP (Open Telecom Platform) is its standard library of ready-made patterns, and its supervisor module is the library that names the restart strategies this lesson uses. OTP names the rule above one_for_one, and it is the default. The Supervisor Behaviour documentation puts it in one sentence: "If a child process terminates, only that process is restarted." The opposite strategy, one_for_all, terminates and restarts every child when any one of them dies. A third, rest_for_one, restarts the failed child plus the children that were started after it.

The rule in Armstrong's thesis. Joe Armstrong designed Erlang's fault-tolerance model and described it in his 2003 PhD thesis at KTH. In section 5.2.3 he calls the one_for_one rule an OR supervision tree. An OR supervisor restarts the child that died, and the activities under it are assumed to be independent, so the failure of one child has no consequence for the other children. The dependent case is the AND tree, where the processes are co-ordinated and one failure means all of them restart (that is one_for_all). In chapter 6, Armstrong configures the supervisor as {one_for_one,5,1000} and says this builds an "or" tree because the three activities being supervised are considered unrelated. The numbers also set a limit: more than 5 restarts in 1000 seconds and the supervisor itself fails.

That limit matters. A worker that crashes the moment it starts will be restarted forever unless something stops the loop. Every real supervisor therefore pairs a strategy with a restart cap. This lesson's version is per worker: after three restarts, a further exit produces a gave_up event for that worker only, and the supervisor stops restarting it. The other worker is not affected. The pair as a whole, which the report calls the ensemble, is never healthy if any worker has given up.

The same line in Kubernetes. Kubernetes is a system that runs containers (packaged programs) across a cluster of machines. Its smallest scheduling unit is a Pod, a group of one or more containers that run together. It draws the one-for-one versus one-for-all line at two layers.

Inside a Pod, the Pod Lifecycle documentation says: "If one of the containers in the Pod fails, then Kubernetes may try to restart that specific container." Sidecar containers "can restart independently of the main application container." That is one_for_one. The documentation also describes RestartAllContainers, an explicit opt-in that terminates and restarts the entire Pod in place. That is the cluster analog of one_for_all, and you have to ask for it.

Above the Pod, a ReplicaSet exists "to maintain a stable set of replica Pods running at any given time." The ReplicaSet documentation compares it directly to this lesson's topic: "Think of it similarly to a process supervisor, only it supervises multiple Pods across multiple nodes instead of individual processes on a single node." It also says a ReplicaSet "delegates local container restarts to some agent on the node such as Kubelet." The kubelet is the agent that runs on each machine and starts and stops that machine's containers. A Deployment is the higher-level object people usually write, and it manages ReplicaSets. The practical consequence: if two workloads are unrelated, put them in separate Deployments, so replacing a Pod of one does not take the other service down. The Certified Kubernetes Administrator (CKA) curriculum lists the primitives for building self-healing application deployments under Workloads and Scheduling, and restart behaviour is part of that.

What this lesson builds. The lab is a decision kernel named isolate, written in Rust. It is not a live process supervisor. It reads a plan, which is a JSONL file (JSON Lines: one JSON object per line) of started, heartbeat, and exited events. It decides what a one-for-one supervisor would do and prints an action ledger, one JSON line per decision. A second subcommand prints a status report. Simulated here: spawning processes, OTP, and the Kubernetes API. Nothing in the lab starts a real process or talks to a cluster. Separating the decision from the side effects is deliberate, because it lets you test the policy with plain text files. The scorer in the examples stands for the ML side of a pipeline, an independent scoring sidecar. No model is trained or run.

Worked example

The reference program is one Rust file, main.rs, built with edition 2024, serde and serde_json for the JSON, and no other dependencies. Each plan line deserializes into a record with a worker name and an event name:

#[derive(Debug, Deserialize)]
struct Record {
    worker: String,
    event: String,
}

Extra keys such as ts and code are ignored, and lines that are not valid JSON are skipped. The state each worker accumulates is two fields:

#[derive(Debug, Default, Clone)]
struct Worker {
    restarts: u32,
    gave_up: bool,
}

The strategy is one small function. In the reference, one_for_one means "no siblings":

fn siblings_to_touch(_failed: &str, _known: &[String]) -> Vec<String> {
    Vec::new()
}

Every restarting line in the ledger records what this function returned in a field called siblings_touched. For one_for_one that field must always be []. A sibling here is any worker other than the one that failed.

The per-worker state and the pair's state come from two more functions:

fn worker_state(w: &Worker) -> &'static str {
    if w.gave_up {
        "gave_up"
    } else if w.restarts > 0 {
        "recovering"
    } else {
        "healthy"
    }
}

fn ensemble_state(states: &[&str]) -> &'static str {
    if states.iter().any(|s| *s == "gave_up") {
        "degraded"
    } else if states.iter().any(|s| *s == "recovering") {
        "recovering"
    } else {
        "healthy"
    }
}

The order of the checks is the policy. gave_up is tested first, so a single exhausted worker makes the whole pair degraded regardless of what the other worker is doing.

The main loop, shortened here, handles each exited event. A worker that has already given up is ignored. Otherwise the loop compares the worker's restart count to the cap of three:

if snapshot.gave_up {
    continue;
}
if snapshot.restarts < MAX_RESTARTS {
    let attempt = snapshot.restarts + 1;
    workers.get_mut(&failed).expect("just inserted").restarts = attempt;
    let touched = siblings_to_touch(&failed, &known);
    actions.push(Action {
        event: "restarting",
        worker: failed.clone(),
        attempt,
        max: MAX_RESTARTS,
        siblings_touched: touched.clone(),
    });
    // ... restarts for any siblings in `touched`
} else {
    workers.get_mut(&failed).expect("just inserted").gave_up = true;
    // ... pushes a "gave_up" action with attempt: snapshot.restarts
}

**Four exits of api.** The plan starts both workers, then has api exit four times. Timestamps are plain counters, and the kernel does not read them: the cap counts exits, not elapsed time.

{"ts":1,"worker":"api","event":"started"}
{"ts":2,"worker":"scorer","event":"started"}
{"ts":3,"worker":"api","event":"exited","code":70}
{"ts":4,"worker":"api","event":"exited","code":70}
{"ts":5,"worker":"api","event":"exited","code":70}
{"ts":6,"worker":"api","event":"exited","code":70}

Running isolate apply on that plan prints the ledger. The reference binary produced these lines:

{"event":"restarting","worker":"api","attempt":1,"max":3,"siblings_touched":[]}
{"event":"restarting","worker":"api","attempt":2,"max":3,"siblings_touched":[]}
{"event":"restarting","worker":"api","attempt":3,"max":3,"siblings_touched":[]}
{"event":"gave_up","worker":"api","attempt":3,"max":3,"siblings_touched":[]}

The first three exits each earn a restart, numbered 1, 2, 3. The fourth finds restarts already at the cap of 3, so the worker is marked gave_up and the ledger records attempt: 3, the number of restarts that were spent. There is no line at all for scorer. Running isolate report on the same plan prints:

strategy: one_for_one
max_restarts: 3
worker.api.state: gave_up
worker.api.restarts: 3
worker.scorer.state: healthy
worker.scorer.restarts: 0
siblings_restarted: 0
ensemble: degraded

api is gave_up with three restarts used. scorer is healthy with none, because it never failed and was never touched. siblings_restarted: 0 counts restart actions that touched a sibling. The ensemble is degraded, not healthy, because one worker gave up. A dashboard that read only scorer would say everything is fine, and the ensemble line exists to prevent that.

**One exit of api.** With a plan that has the same two started lines and a single api exit, the ledger is one line and only api is restarted:

{"event":"restarting","worker":"api","attempt":1,"max":3,"siblings_touched":[]}

The report for that plan:

strategy: one_for_one
max_restarts: 3
worker.api.state: recovering
worker.api.restarts: 1
worker.scorer.state: healthy
worker.scorer.restarts: 0
siblings_restarted: 0
ensemble: recovering

recovering means a worker has been restarted at least once and has not exhausted its cap. The lab has a mirror-image plan where scorer exits once and api only starts. By the same rules the ledger should contain one restarting line for scorer and none for api. The checker runs that case, though this lesson does not show its output.

Lab

The starter project is the reference with three deliberate bugs. You fix them so the checker passes.

  1. siblings_to_touch returns every other known worker. That is one_for_all: the first api exit would also list scorer in siblings_touched and add a restart line for it.
  2. worker_state always returns healthy, so no worker is ever reported as recovering or gave_up.
  3. ensemble_state always returns healthy, so the pair never reports degraded.

Prerequisites. Rust with edition 2024 support. The dependencies serde and serde_json are vendored in vendor.tgz, so the build runs with cargo build --offline and needs no network. The checker also uses tar and jq. Everything is simulated: no cloud account, no cluster.

Interface. The binary is isolate. It takes a subcommand and a plan path.

Checker. From the project directory, the one containing Cargo.toml, src/, vendor.tgz, and check.sh, run:

sh check.sh

The script extracts the vendored crates if needed, builds offline, writes its own four plan files, runs the binary against them, and prints one result line per check. On the reference tree the containerised gate printed:

RESULT build PASS
RESULT isolate_api_crash PASS
RESULT isolate_gave_up PASS
RESULT ensemble_not_healthy_after_gave_up PASS
RESULT isolate_scorer_crash PASS
RESULT both_healthy PASS
RESULT missing_file_exit_2 PASS

On the starter tree the same checker compiled, then failed the isolation and honesty checks (the last two still pass, because a plan with no exits is healthy even when the state functions always say healthy, and a missing file still exits 2):

RESULT build PASS
RESULT isolate_api_crash FAIL
RESULT isolate_gave_up FAIL
RESULT ensemble_not_healthy_after_gave_up FAIL
RESULT isolate_scorer_crash FAIL
RESULT both_healthy PASS
RESULT missing_file_exit_2 PASS

The seven checks are:

Fix the three functions, run the checker, and read any failing line against the list above. Fixing siblings_to_touch alone is not enough to pass the state checks, since the two state functions are still wrong. Once all seven pass, change MAX_RESTARTS to 5 and predict the report for the four-exit plan before running it. The api worker should now show recovering with four restarts instead of gave_up.

Other paths

The same isolation rule shows up as a flag in OTP and as a function you write yourself in Go. The teaching point is what changes, not a survey of languages.

Gleam and OTP, the original. Gleam is a statically typed language that runs on the same virtual machine as Erlang and can use OTP's supervisors. In OTP you do not write the decision at all. You pick a strategy with a flag. The Supervisor Behaviour page describes supervisor flags including strategy, which takes one_for_one, one_for_all, or rest_for_one, plus intensity and period, which together set the maximum restart intensity: too many restarts within the period stops the supervisor itself. An Erlang flags map looks like this (illustrative, not run for this lesson):

#{strategy => one_for_one, intensity => 5, period => 1000}

This lines up with Armstrong's {one_for_one,5,1000}. Two things differ from the Rust kernel. OTP counts restarts over a time window for the supervisor as a whole, while the kernel counts exits per worker with no clock. And when the limit is hit, OTP's supervisor fails and escalates to its own supervisor, while the kernel emits gave_up and carries on with the other worker.

Go, a hypothetical sketch. The same decision kernel in Go would be a loop over decoded events with the strategy as an ordinary function. This sketch was not compiled or run:

// hypothetical, not run
func siblingsToTouch(failed string, known []string) []string {
	return nil // one_for_one: independent workers
}

What changes is where the strategy lives. In Rust or Go you write the match yourself, as the three functions above show, and the compiler only checks that your code is well-typed, not that your policy is right. In OTP the strategy is a flag on a tested library, so choosing one_for_all instead of one_for_one is a one-word change. The ledger-and-report approach in this lab is how you would test a hand-rolled policy without starting any processes.

Citations