Alive is not ready Liveness, readiness, and a supervisor that knows when to stop

One tiny service with two health endpoints, one supervisor that restarts it, and a log whose last line admits it gave up.

B-SWE · A · arc: self-healing-edge · langs: rust, gleam, go · CKA-workloads-probes

By Evan Eugene Pincham.

Concept

A long-running service can be unhealthy in two different ways, and one health check cannot tell them apart.

Liveness asks whether the process is able to answer at all. A deadlocked or wedged process fails this. The usual remedy is to kill it and start a new one. A liveness endpoint is a cheap HTTP route (here /healthz) that returns 200 if the process can respond. It should not depend on anything slow, such as a database.

Readiness asks whether traffic should be sent to this instance right now. A process that has just started may be alive but still loading caches, opening connections or reading config. Sending requests to it produces errors. A readiness endpoint (here /readyz) returns 503 until the service is warm and 200 afterwards. A failed readiness check should stop traffic to the instance. It should not restart the instance, because restarting a service that is only warming up starts the warm-up over and can loop forever.

The SRE book draws the same line. In its chapter on cascading failures it calls process health checking (“is this binary responding at all?”) and service health checking (“is this binary able to respond to this class of requests right now?”) “two conceptually distinct operations”: the first matters to whatever restarts processes, the second to the load balancer. Mixing them up lets health checking itself make a service unhealthy, for example by killing tasks that are only starting up. (Beyer et al., Site Reliability Engineering, ch. 22, “Stop Health Check Failures/Deaths”.)

Kubernetes, a container orchestrator, uses the same split. In its documentation a probe is a periodic check run against a container. A livenessProbe failure makes the kubelet (the per-node agent) restart the container. A readinessProbe failure removes the pod from the endpoints that receive Service traffic, and the container keeps running. A third kind, the startupProbe, holds the other two off while a slow application boots. This lesson uses no cluster. It builds both endpoints and a tiny restarter by hand so the mapping to those probe fields is visible in the code. (Kubernetes documentation, "Liveness, Readiness, and Startup Probes" and "Configure Liveness, Readiness and Startup Probes".)

A supervisor is a process whose only job is to start another process (the child), watch it, and restart it when it dies. Joe Armstrong's thesis describes supervision trees as the structure of fault-tolerant Erlang systems. Workers are allowed to crash ("let it crash") and a separate supervisor decides what to do next. In the thesis's supervisor example the restart policy is data handed to the supervisor, {one_for_one,5,1000}: if the supervisor has to restart its children more than 5 times in 1000 seconds, the supervisor itself fails and the problem moves up the tree. The workers contain no restart logic at all.

The restart policy needs a limit. The SRE book defines a cascading failure as “a failure that grows over time as a result of positive feedback”: one replica fails, its load moves to the others, and they fail in turn. Its “Restart Servers” section recommends restarting servers that are wedged (deadlocked, or stuck in a garbage-collection spiral), then warns that you should find the source of the failure first, because a restart “may amplify an existing cascading failure if the outage is actually due to an issue like a cold cache.” A restarted process starts with empty caches and no warm connections, which is exactly the state the readiness endpoint exists to report. A supervisor that restarts forever keeps feeding cold instances into the problem and never tells a person. So the supervisor here is bounded. It restarts a fixed number of times, then stops and says so. (Beyer et al., Site Reliability Engineering, ch. 22, “Restart Servers”.)

That leads to the last rule, honest logging. When the supervisor gives up, its final line must say gave_up, not "healthy" or "recovering". Chapter 6 of the same book, "Monitoring Distributed Systems", separates symptoms (what is broken) from causes (why) and asks that anything which wakes a person be actionable. A supervisor that has stopped trying is a symptom a person must act on, so the log has to name it. Newman's Building Microservices (2nd ed., ch. 12 "Resiliency") makes the surrounding point that failure is a normal state to design for. A signal that says clearly what state the system is in is part of that design.

The supervisor writes one JSON object per line for every action it takes (started, exited, unhealthy, restarting, gave_up). Each line has a timestamp and a named event, so the log is labelled event data. A later lesson can learn from such data, for example to predict which exits precede a give-up. This lesson only produces the data.

Worked example

The reference project, edge-svc, is two small Rust binaries in one Cargo package: the service and its supervisor. It uses axum 0.8, a web framework, on tokio, the async runtime. It was built and checked with Rust 1.98 and axum 0.8.9 (the version pinned in Cargo.lock). Edition 2024 needs Rust 1.85 or later.

**Cargo.toml.**

[package]
name = "edge-svc"
version = "0.1.0"
edition = "2024"

[dependencies]
axum = "0.8"
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "signal", "time", "process", "io-util"] }

[[bin]]
name = "edge-svc"
path = "src/main.rs"

[[bin]]
name = "supervisor"
path = "src/supervisor.rs"

The two [[bin]] tables produce two executables. The process feature of tokio provides tokio::process::Command, the supervisor's way to spawn a child. The io-util feature provides read_to_end and write_all, which the supervisor uses to talk HTTP over a raw socket, so it needs no HTTP client dependency.

The service: shared state.

#[derive(Clone)]
struct App { ready: Arc<AtomicBool> }

The only state is one flag. An AtomicBool can be read and written from many tasks without a lock, and Arc (an atomically reference-counted pointer) lets the handlers and the warm-up task share it. axum requires state to be Clone, and cloning an Arc copies the pointer, not the flag.

The service: the three handlers.

async fn healthz() -> &'static str { "ok" }

async fn readyz(State(app): State<App>) -> (StatusCode, &'static str) {
    if app.ready.load(Ordering::SeqCst) { (StatusCode::OK, "ready") }
    else { (StatusCode::SERVICE_UNAVAILABLE, "not ready") }
}

async fn crash() -> &'static str { std::process::exit(70) }

healthz touches no state. It returns 200 whenever the runtime is scheduling the handler, and that is exactly the liveness claim: the process can answer. readyz returns 503 or 200 from the flag. That status code is what a load balancer or a Kubernetes readinessProbe reads. crash exits the whole process with code 70, which stands in for a fatal fault so the supervisor has something to heal. Real services would not expose it.

The service: warm-up and startup.

let warm_ms: u64 = std::env::var("READY_AFTER_MS").ok().and_then(|v| v.parse().ok()).unwrap_or(1000);
let app = App { ready: Arc::new(AtomicBool::new(false)) };
// ... router and listener ...
let flag = app.ready.clone();
tokio::spawn(async move {
    tokio::time::sleep(Duration::from_millis(warm_ms)).await;
    flag.store(true, Ordering::SeqCst); // warm-up finished: accept traffic
});

The flag starts false. A spawned task sleeps for READY_AFTER_MS (default 1000) and then sets it. In a real service the sleep would be the actual work: filling a cache, waiting for a connection pool. The listener is already bound by then, so /healthz answers 200 from the first moment while /readyz answers 503 for about a second. The gap between the two answers is the lesson.

axum::serve(listener, router)
    .with_graceful_shutdown(async { let _ = tokio::signal::ctrl_c().await; })
    .await
    .unwrap();

Serve::with_graceful_shutdown takes a future. When that future completes, the server stops accepting new connections and lets in-flight requests finish before serve returns (axum 0.8 API docs, serve::Serve::with_graceful_shutdown). Here the future waits for Ctrl-C. The supervisor below stops a sick child with a hard kill, not a signal, so this path matters for a human at a terminal or for an orchestrator that sends a polite stop first.

The lab checker shows the gap. It starts edge-svc with READY_AFTER_MS=1500, asks both endpoints for a status code as soon as /healthz answers, waits two seconds, and asks /readyz again:

healthz=200 readyz_during_warmup=503 readyz_after_warmup=200

The supervisor: logging.

const MAX_RESTARTS: u32 = 3; // restart intensity: give up and escalate after this many

fn now() -> u64 { SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() }
fn report(event: &str, detail: &str) { println!(r#"{{"ts":{},"event":"{event}","detail":"{detail}"}}"#, now()); }

report is the whole logging layer: one println! that writes {"ts":…,"event":…,"detail":…} to standard output. The raw string r#"…"# avoids escaping the quotes, and {{ }} print literal braces. It does not escape event or detail, so a detail containing a double quote would produce invalid JSON. The strings used here are fixed or are numbers and exit statuses, so the lab can ignore that. A production version should use a JSON library.

The supervisor: probing liveness.

async fn healthy(port: &str) -> bool {
    let Ok(Ok(mut s)) = tokio::time::timeout(Duration::from_millis(500), TcpStream::connect(format!("127.0.0.1:{port}"))).await else { return false };
    let req = "GET /healthz HTTP/1.1\r\nHost: x\r\nConnection: close\r\n\r\n";
    if s.write_all(req.as_bytes()).await.is_err() { return false; }
    let mut buf = Vec::new();
    let _ = tokio::time::timeout(Duration::from_millis(500), s.read_to_end(&mut buf)).await;
    buf.starts_with(b"HTTP/1.1 200")
}

This is a hand-written HTTP/1.1 GET. It connects with a 500 ms timeout, sends the request, reads the response for at most 500 ms, and reports healthy only if the reply starts with HTTP/1.1 200. The timeouts matter: a wedged process may accept a connection and never answer, and a probe without a deadline would hang the supervisor along with it. Kubernetes probes have the same idea in timeoutSeconds. The supervisor probes /healthz, not /readyz. A child still warming up is alive and must not be killed for it.

The supervisor: the bounded loop.

let mut restarts = 0;
loop {
    let mut child = Command::new(&bin).env("PORT", &port).stderr(Stdio::null()).spawn().expect("spawn");
    report("started", &format!("pid {}", child.id().unwrap_or(0)));
    sleep(Duration::from_millis(300)).await;
    loop {
        sleep(Duration::from_millis(200)).await;
        if let Ok(Some(status)) = child.try_wait() { report("exited", &format!("{status}")); break; }
        if !healthy(&port).await { report("unhealthy", "healthz failed; killing"); let _ = child.kill().await; break; }
    }
    restarts += 1;
    if restarts > MAX_RESTARTS { report("gave_up", &format!("{MAX_RESTARTS} restarts; escalating to a human")); std::process::exit(1); }
    report("restarting", &format!("attempt {restarts}/{MAX_RESTARTS}"));
}

The outer loop starts one child. The inner loop watches it every 200 ms after a 300 ms grace period, which gives the new child time to bind its port before the first probe. Two things end the inner loop. try_wait returning a status means the child has already exited, so the supervisor logs exited with the status. A failed liveness probe means the child is alive but not answering, so the supervisor logs unhealthy and kills it. child.kill().await also waits for the process, so no zombie process (an exited child whose exit status nobody has collected) is left behind.

After the inner loop ends, the counter goes up. The test is restarts > MAX_RESTARTS, so with MAX_RESTARTS = 3 the first three failures log restarting and the fourth logs gave_up and calls exit(1). That non-zero exit code is the supervisor's own signal to whoever started it. Two details are worth seeing. The counter never resets, so this is a lifetime cap, not "3 per minute". The Erlang answer to that is in Other paths. And gave_up is the final line the supervisor writes. Nothing after it can suggest recovery.

Here is a captured run. The supervisor was started with READY_AFTER_MS=100, and /crash was requested four times, each time once the new child answered /healthz. The last line was appended by the shell after the supervisor exited:

{"ts":1793545396,"event":"started","detail":"pid 960"}
{"ts":1793545397,"event":"exited","detail":"exit status: 70"}
{"ts":1793545397,"event":"restarting","detail":"attempt 1/3"}
{"ts":1793545397,"event":"started","detail":"pid 979"}
{"ts":1793545397,"event":"exited","detail":"exit status: 70"}
{"ts":1793545397,"event":"restarting","detail":"attempt 2/3"}
{"ts":1793545397,"event":"started","detail":"pid 1000"}
{"ts":1793545398,"event":"exited","detail":"exit status: 70"}
{"ts":1793545398,"event":"restarting","detail":"attempt 3/3"}
{"ts":1793545398,"event":"started","detail":"pid 1018"}
{"ts":1793545398,"event":"exited","detail":"exit status: 70"}
{"ts":1793545398,"event":"gave_up","detail":"3 restarts; escalating to a human"}
supervisor exit status: 1

exit status: 70 is the child's own exit code, passed through from /crash. Three restarts follow, and the fourth exit produces gave_up and the supervisor's exit status 1. ts is Unix time in whole seconds, which is why several events share a value; a log meant for later analysis would want milliseconds.

Lab

You get a starter project with the same layout as the reference: Cargo.toml, Cargo.lock, src/main.rs, src/supervisor.rs, and the checker check.sh beside them. It differs in two ways:

• In src/main.rs, /readyz always returns "ready" with status 200. There is no warm-up flag.
• In src/supervisor.rs, there is no MAX_RESTARTS. The loop logs restarting attempt N and runs forever.

Task.

1. Make /readyz return 503 until warm-up completes, then 200. Warm-up lasts READY_AFTER_MS milliseconds, default 1000. /healthz must return 200 the whole time.
2. Cap restarts at MAX_RESTARTS = 3. When the child fails a fourth time, write a JSON line whose event is gave_up and exit with a non-zero status.
3. Keep every supervisor log line a single JSON object.

You are allowed to read the walkthrough above, but type the changes yourself and think about which flag, constant or comparison each one touches.

Checker. Run it from the project root with sh check.sh. It needs cargo, curl and jq on your path and builds with cargo build --offline, so run cargo fetch once first while you have a network connection. It listens on 127.0.0.1:18081, so nothing else may hold that port. The lab is simulated: no cloud account and no cluster. It prints one line per check: the word RESULT, the check name, then PASS or FAIL. The six checks and what they are meant to establish:

Run against the untouched starter, the checker prints:

RESULT build PASS
healthz=200 readyz_during_warmup=200 readyz_after_warmup=200
RESULT healthz_200 PASS
RESULT readyz_gates_traffic FAIL
child pid before kill -9: 976, after: 1016
RESULT restart_after_kill PASS
supervisor exit: running
last line: {"ts":1793545365,"event":"started","detail":"pid 1076"}
RESULT bounded_gives_up FAIL
RESULT log_is_json PASS

Read the failures as diagnostics. readyz_during_warmup=200 means /readyz answers 200 before warm-up ends: either there is no flag, the flag starts true, or the handler ignores it. supervisor exit: running means the supervisor was still alive after four failures and the checker had to stop it, so there is no cap. restart_after_kill and log_is_json already pass on the starter; your changes must keep them passing.

Run against the finished reference code, it prints:

RESULT build PASS
healthz=200 readyz_during_warmup=503 readyz_after_warmup=200
RESULT healthz_200 PASS
RESULT readyz_gates_traffic PASS
child pid before kill -9: 981, after: 1021
RESULT restart_after_kill PASS
supervisor exit: 1
last line: {"ts":1793545354,"event":"gave_up","detail":"3 restarts; escalating to a human"}
RESULT bounded_gives_up PASS
RESULT log_is_json PASS

The process IDs and the timestamp in the gave_up line will differ on your machine. The RESULT lines will not.

Stretch, optional. Add a second environment variable for the restart cap and read it in the supervisor. Then decide whether MAX_RESTARTS should count lifetime failures or failures inside a time window, and write down which one you chose and why.

Other paths

Gleam on the BEAM. Gleam is a statically typed language that compiles to run on the BEAM, the Erlang virtual machine. Here the supervisor is not code you write. It is part of the platform. Armstrong's thesis describes the Erlang design in which a supervisor process starts workers and applies a restart strategy when one dies, and OTP (Open Telecom Platform, the standard Erlang libraries for building such systems) ships that as a ready-made behaviour. Restart intensity (a maximum number of restarts within a time period, like the thesis's 5 restarts in 1000 seconds, after which the supervisor itself terminates and passes the failure up the tree) is a setting on that supervisor, so the "give up" rule from supervisor.rs becomes configuration. It is also windowed, which fixes the lifetime-counter limitation noted above. Gleam has an OTP package for this, but the exact API names and option types here are hypothetical; they were not built or checked for this lesson. The HTTP side (a liveness route and a readiness route) would use a Gleam web server library and the same two-status logic.

Go. In Go, the service is net/http with two handlers and an atomic.Bool for the readiness flag, and the supervisor is a loop around os/exec's exec.Command(...).Start() with cmd.Wait() running in a goroutine, plus a ticker that hits /healthz with an http.Client that has a timeout. The loop, the counter, the restarts > max comparison and the gave_up line are the same as in the Rust version, written by hand. Go gives you no built-in supervisor, so every part of the restart policy is your own code. This sketch is hypothetical: it was not built or run for this lesson.

Citations

• Joe Armstrong, Making reliable distributed systems in the presence of software errors, PhD thesis, KTH (Royal Institute of Technology), 2003. Supervision trees, "let it crash", and the supervisor behaviour example with restart frequency {one_for_one,5,1000}.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.), Site Reliability Engineering, O'Reilly, 2016. Chapter 6, "Monitoring Distributed Systems"; Chapter 22, "Addressing Cascading Failures" sections "Stop Health Check Failures/Deaths" and "Restart Servers".
• Sam Newman, Building Microservices, 2nd ed., O'Reilly, 2021. Chapter 12, "Resiliency".
• Kubernetes documentation, "Liveness, Readiness, and Startup Probes" (Concepts) and "Configure Liveness, Readiness and Startup Probes" (Tasks), kubernetes.io.
• axum 0.8 API docs, serve::Serve::with_graceful_shutdown, docs.rs/axum.