Hedronite · Curriculum · SWE · Ops · Sun 2026-11-01

Restart policy is a decision Backoff, a cap, and who gets told

Pull the restart policy out of the code, make the config refuse a bad one, and ask the log two questions: how many restarts, and when did it give up.

Lesson Class: Curriculum · O-SWE · rung A
Tested with: Nickel 1.18.0 · Nushell 0.116.1
Lab: lab/ · sh check.sh (6 checks) · no network
Source: Kubernetes Pod Lifecycle · systemd.service(5) · SRE ch. 22 · Armstrong 2003
Name the Policy
How many restarts, how long between them, and what happens after the last one.
Refuse Early
A contract rejects a bad policy when it is exported, before anything runs.
Read the Log
Count restarts, not starts, and report the moment it gave up.
A restart loop with no ceiling is an outage that has not been reported yet.

O-SWE · A · arc: self-healing-edge · langs: nickel, nushell, go · CKA-workloads-probes, DOP-2.3

By Evan Eugene Pincham.

Concept

The companion lesson Liveness, readiness, and a bounded supervisor in Rust built edge-svc, a tiny Rust HTTP service. It has three endpoints:

A supervisor program starts the service and restarts it when it exits. It stops after MAX_RESTARTS = 3, a Rust constant. After that it writes a gave_up line and exits with status 1. It also writes one JSON line per action:

{"ts":<unix seconds>,"event":"started|exited|restarting|unhealthy|gave_up","detail":"..."}

That constant is a decision, and an operator should own it, not whoever last edited the source. A restart policy answers five questions:

  1. How many restarts are allowed?
  2. How long should the supervisor wait before the first restart?
  3. How fast does that wait grow?
  4. What is the ceiling on the wait?
  5. When the supervisor gives up, who is told, and what happens next: escalate to a human or roll back?

The wait and the ceiling matter because restarting is not free. The SRE book's chapter on cascading failures notes that restarting servers can make a cascade worse, because restarted tasks come back with cold caches. It also notes that health-check failures can kill tasks that are healthy but slow. Backoff with a cap exists so that restarts do not pile on top of each other.

The idea of "cap, then escalate" is older than containers. In Joe Armstrong's 2003 thesis, an Erlang/OTP supervisor carries a restart frequency such as {one_for_one,5,1000}. If more than 5 restarts happen in 1000 seconds, the supervisor terminates itself and escalates to its parent. A cap is a statement that, past this point, a person or a higher-level component should decide.

Two more operator decisions sit next to the restart policy, and they are about probes:

Those are different questions with different costs. A failed liveness check costs a restart. A failed readiness check only takes the instance out of rotation.

This lesson does two jobs:

The machine-readable JSON log is also a small piece of future labelled data. Each line records what happened and when, and the gave_up line marks an outcome. Machine learning only gets a mention here. Nothing in this lesson trains or evaluates a model, and the only point is to keep the log structured so it can be used later.

One limit to state plainly: this lesson produces the policy file and the log query. It does not change the Rust supervisor to read the exported policy. Until that wiring is done, the Rust constant and the Nickel file can disagree.

Worked example

You need nickel and nu on your PATH, nothing else. The code below was written against Nickel 1.18.0 and Nushell 0.116.1.

The contract

# restart-policy.ncl: the contract every service's restart policy must satisfy.
let Bounded = fun lo hi label =>
  std.contract.from_validator (fun n =>
    if std.is_number n && std.number.is_integer n && n >= lo && n <= hi then 'Ok
    else 'Error { message = "%{label} must be an integer from %{std.to_string lo} to %{std.to_string hi}" })
in
{
  RestartPolicy = {
    max_restarts | Bounded 1 10 "max_restarts",
    backoff_ms | Bounded 100 60000 "backoff_ms",
    backoff_factor | Bounded 1 4 "backoff_factor",
    max_backoff_ms | Bounded 100 300000 "max_backoff_ms",
    on_give_up | [| 'escalate, 'rollback |],
  },
}

Why a contract. The Rust compiler checks that MAX_RESTARTS is a u32, and nothing more: it accepts 5000 as happily as 3. A config file with a contract is checked every time someone runs nickel export. A policy with max_restarts = 5000 or backoff_ms = 0 never reaches a deploy, because the export fails first. This is the point of "refuse at export time": the mistake costs a failed command on a laptop, not a restart storm in production.

**What std.contract.from_validator does.** The Nickel manual's chapter on contracts describes a contract as a check on a value. from_validator is the simplest way to write one. You give it a function that takes the value and returns either 'Ok or 'Error { message = ... }. Nickel turns that function into a contract and shows your message if the check fails.

Here, Bounded lo hi label builds one such contract for each field. It accepts only numbers that are integers inside lo..hi, and the message names the field and the allowed range. The ranges are the operator's guard rails:

on_give_up is an enum with exactly two tags, 'escalate and 'rollback. A typo in a tag is an error, not a silent default.

The policy

# policy.ncl: edge-svc's restart policy, checked by the contract, plus the delays it implies.
let { RestartPolicy, .. } = import "restart-policy.ncl" in
let p | RestartPolicy = {
  max_restarts = 3,
  backoff_ms = 200,
  backoff_factor = 2,
  max_backoff_ms = 5000,
  on_give_up = 'escalate,
}
in
{
  edge_svc = p & {
    delays_ms = std.array.generate
      (fun i => std.number.min p.max_backoff_ms (p.backoff_ms * std.number.pow p.backoff_factor i))
      p.max_restarts,
  },
}

The first line imports the contract. let p | RestartPolicy = { ... } binds the policy and checks it against that contract. The values are the ones the Rust constant stood for: 3 restarts, a first wait of 200 ms, doubling, capped at 5000 ms, and escalate to a human on give-up.

The exported object also contains delays_ms. This is the policy stating its own consequences. std.array.generate builds a list of length max_restarts. Entry i is the smaller of the cap and backoff_ms * backoff_factor^i. For this policy that is [200, 400, 800]. A reviewer does not have to do the arithmetic in their head, and a diff to the policy shows the new schedule next to the new inputs.

With only three restarts the cap never applies, because 800 is below 5000. It starts to matter if someone raises max_restarts. At 10 restarts the schedule is 200, 400, 800, 1600, 3200, then 5000 for the remaining entries. The contract allows that, so the cap has to be right now, not later.

Export it:

nickel export policy.ncl

Output, captured in the lab container:

$ nickel export policy.ncl
{
  "edge_svc": {
    "backoff_factor": 2,
    "backoff_ms": 200,
    "delays_ms": [
      200,
      400,
      800
    ],
    "max_backoff_ms": 5000,
    "max_restarts": 3,
    "on_give_up": "escalate"
  }
}

To see the contract refuse something, write a policy with max_restarts = 50 into bad.ncl and export it. The error names the broken field, prints your message, and points at both the contract line and the offending value:

$ nickel export bad.ncl
error: contract broken by the value of `max_restarts`
       max_restarts must be an integer from 1 to 10
  ┌─ /lab/restart-policy.ncl:9:20
  │
9 │     max_restarts | Bounded 1 10 "max_restarts",
  │                    --------------------------- expected type
  │
  ┌─ /lab/bad.ncl:2:18
  │
2 │ { max_restarts = 50, backoff_ms = 200, backoff_factor = 2, max_backoff_ms = 5000, on_give_up = 'escalate } | RestartPolicy
  │                  ^^ applied to this expression

exit status: 1

The exit status is non-zero, so any script or CI step that runs the export stops here.

The log

This is the real supervisor log captured in the companion Rust lesson's lab. The last line is plain text appended by the capture, not a JSON event.

{"ts":1793545396,"event":"started","detail":"pid 960"}
{"ts":1793545397,"event":"exited","detail":"exit status: 70"}
{"ts":1793545397,"event":"restarting","detail":"attempt 1/3"}
{"ts":1793545397,"event":"started","detail":"pid 979"}
{"ts":1793545397,"event":"exited","detail":"exit status: 70"}
{"ts":1793545397,"event":"restarting","detail":"attempt 2/3"}
{"ts":1793545397,"event":"started","detail":"pid 1000"}
{"ts":1793545398,"event":"exited","detail":"exit status: 70"}
{"ts":1793545398,"event":"restarting","detail":"attempt 3/3"}
{"ts":1793545398,"event":"started","detail":"pid 1018"}
{"ts":1793545398,"event":"exited","detail":"exit status: 70"}
{"ts":1793545398,"event":"gave_up","detail":"3 restarts; escalating to a human"}
supervisor exit status: 1

Look at the timestamps. All three restarts happened inside about two seconds, because that supervisor has no backoff: it starts the next copy as soon as it sees the exit. That is what "no backoff" looks like in a log, and it is the behaviour the new policy is meant to replace.

The query

# restarts.nu: summarise a supervisor JSON-lines log for an operator.
# Usage: nu restarts.nu <log>   prints one JSON object.
def main [log: path] {
  let events = (open --raw $log | lines | where {|l| $l starts-with '{' } | each {|l| $l | from json })
  let gave = ($events | where event == 'gave_up')
  {
    restarts: ($events | where event == 'restarting' | length)
    gave_up: (not ($gave | is-empty))
    gave_up_at: (if ($gave | is-empty) { null } else {
      $gave | first | get ts | $in * 1_000_000_000 | into datetime | format date '%Y-%m-%dT%H:%M:%SZ'
    })
  } | to json --raw
}

Piece by piece:

Run it on the real log and on a constructed one. recovered.log is a constructed four-line fixture, not captured output. It has one restart and no give-up:

{"ts":1793545396,"event":"started","detail":"pid 960"}
{"ts":1793545397,"event":"exited","detail":"exit status: 70"}
{"ts":1793545397,"event":"restarting","detail":"attempt 1/3"}
{"ts":1793545397,"event":"started","detail":"pid 979"}
$ nu restarts.nu fixtures/gave-up.log
{"restarts":3,"gave_up":true,"gave_up_at":"2026-11-01T15:03:18Z"}
$ nu restarts.nu fixtures/recovered.log
{"restarts":1,"gave_up":false,"gave_up_at":null}

The real log reports three restarts and a give-up at 15:03:18 UTC. The constructed one reports one restart, gave_up false, and gave_up_at null.

The two numbers in that output are the operator's signal. The restart count says how unhealthy the service is. The give-up time says when a human was supposed to be told.

Lab

This lab is simulated. It needs no cloud account and no cluster, and the checker needs no network. You need nickel and nu.

The starter folder holds the same three files as the worked example, plus the two fixtures, but with these faults:

Your task, in the starter folder:

  1. In restart-policy.ncl, bound the contract. max_restarts must be an integer from 1 to 10. backoff_ms must be an integer of at least 100. The other numeric fields must be integers. A rejected value must produce a readable message that names the field and the allowed range.
  2. In policy.ncl, set the policy to 3 restarts, starting at 200 ms, doubling, capped at 5000 ms. Make delays_ms apply the cap.
  3. In restarts.nu, count restarting events and report gave_up and gave_up_at.

Run the checker from the starter folder:

cd lab/starter
sh ../check.sh

It prints one result line per check: the word RESULT, the check name, then PASS or FAIL. The six checks are:

The two contract_rejects_* checks write small probe files that try a policy with 50 restarts and a policy with a zero-millisecond backoff. They pass only when your contract refuses both. The query checks compare your output to an exact JSON string for each fixture, so keep the field names and the date format as they appear in the worked example.

On the untouched starter, every check fails. Each line before a RESULT is a diagnostic: the starter exports 12 restarts at 50 ms, the contract accepts both bad probes, the uncapped schedule climbs past 100 seconds, and the query counts four restarts where there were three:

exported: max_restarts=12 backoff_ms=50
RESULT policy_exports_bounded FAIL
contract accepted max_restarts=50 backoff_ms=200
RESULT contract_rejects_no_cap FAIL
contract accepted max_restarts=3 backoff_ms=0
RESULT contract_rejects_zero_backoff FAIL
delays_ms=[50,100,200,400,800,1600,3200,6400,12800,25600,51200,102400]
RESULT delays_capped FAIL
gave-up.log   -> {"restarts":4,"gave_up":false,"gave_up_at":null}
recovered.log -> {"restarts":2,"gave_up":false,"gave_up_at":null}
RESULT query_gave_up_log FAIL
RESULT query_recovered_log FAIL

On a finished solution (this is the reference, run in a clean container with networking off), all six pass:

exported: max_restarts=3 backoff_ms=200
RESULT policy_exports_bounded PASS
contract refused max_restarts=50 backoff_ms=200: max_restarts must be an integer from 1 to 10
RESULT contract_rejects_no_cap PASS
contract refused max_restarts=3 backoff_ms=0: backoff_ms must be an integer from 100 to 60000
RESULT contract_rejects_zero_backoff PASS
delays_ms=[200,400,800]
RESULT delays_capped PASS
gave-up.log   -> {"restarts":3,"gave_up":true,"gave_up_at":"2026-11-01T15:03:18Z"}
recovered.log -> {"restarts":1,"gave_up":false,"gave_up_at":null}
RESULT query_gave_up_log PASS
RESULT query_recovered_log PASS

If you finish early, set on_give_up to 'rollback and confirm the export still passes. Then try 'restart_forever. The export fails with `contract broken by the value of on_give_up and tag not in the enum type`: the enum, not a comment, defines the allowed answers.

Other paths

Go. The same policy can be a Go struct with the five fields and a Validate() method that returns an error naming the field and its range. The delay schedule is a small loop that multiplies and clamps. What you lose is export-time refusal inside the config itself. A bad value is caught when the program starts or when a test calls Validate(), not when someone merely exports or reviews the file. All of this is hypothetical, since no Go code was written or compiled for this lesson.

systemd. The decisions map onto unit settings. In systemd.service(5), Restart=on-failure chooses when to restart and RestartSec= is the first wait (default 100 ms). RestartSteps= and RestartMaxDelaySec=, added in version 254, give exponential backoff with a ceiling. The man page's own example, RestartSec=10s, RestartSteps=4, RestartMaxDelaySec=160s, gives 10 s, 20 s, 40 s, 80 s, 160 s, then 160 s again, and it gives the ratio as (RestartMaxDelaySec / RestartSec)^(1 / RestartSteps). For this lesson's policy, RestartSec=200ms and RestartMaxDelaySec=5s give a ratio of 25^(1/RestartSteps). No whole number of steps makes that exactly 2, because 25 is 2 to the power of about 4.64. RestartSteps=5 is the closest, with a ratio of about 1.9: roughly 200, 381, 725, 1380 and 2627 ms, then 5 s from then on. systemd describes the ramp by its end points, where the Nickel file describes it by its factor. The give-up side lives in systemd.unit(5): units started more than StartLimitBurst= times within StartLimitIntervalSec= are not permitted to start any more, and a unit with Restart= that reaches the limit is not restarted. The limit counts every start, including the first and any manual one, so "one launch plus three restarts" is StartLimitBurst=4; StartLimitIntervalSec=infinity makes it a limit on total attempts. OnFailure= names units to activate when this unit enters the failed state, and that is the escalation hook.

Kubernetes (read-only mapping, no cluster used). The Pod Lifecycle page says a Pod's restartPolicy is Always, OnFailure or Never, and defaults to Always. Deployments allow only Always. After containers exit, the kubelet restarts them with an exponential backoff delay of 10 s, 20 s, 40 s and so on, capped at 300 seconds. Once a container has run for 10 minutes without problems, the kubelet resets the backoff timer. CrashLoopBackOff is the status you see while that backoff is in effect for a crash-looping container. There is no Pod field for these numbers. The nearest knob is per node: with the KubeletCrashLoopBackOffMax feature (beta since v1.35), the kubelet configuration's crashLoopBackOff.maxContainerRestartPeriod lowers the 300-second ceiling, set between 1 s and 300 s. So max_backoff_ms has a cluster-operator counterpart, and backoff_ms and max_restarts have none. The important operator point is that, for a Deployment, there is no give-up. The kubelet keeps retrying at the cap. The "cap, then escalate" decision therefore has to come from alerting on restart count, not from restartPolicy.

Probes map to the two endpoints. A failed liveness probe makes the kubelet restart the container, so /healthz is the liveness check. A failed readiness probe removes the Pod from the Service's EndpointSlices without restarting it, so /readyz is the readiness check. A startup probe holds off liveness and readiness until startup succeeds.

Citations