Rollout policy is a decision A deadline, a budget, and who rolls back

Write down how long a release gets to prove itself and how much may be down meanwhile, make the config refuse a bad answer, and ask the log which release is really live.

O-DevOps · A · arc: self-healing-edge · langs: nickel, nushell · CKA-workloads-rollout, DOP-1.4

By Evan Eugene Pincham.

Concept

A deploy is a hypothesis about the new release. Readiness inside a deadline is the test, and the old release stays in place until the test passes. The companion lesson, "A readiness-gated deploy that rolls back on its own, in Rust", built deployer <svc> <new-port> <new-version> [deadline-ms] for a service called edge-svc. It starts the candidate on a second port and polls its /readyz every 100 ms. It stops at the first of three outcomes: the candidate answers 200, the candidate exits, or the deadline passes. If the candidate is ready, the deployer swaps the current pointer file with an atomic rename, sends SIGTERM to the old release, logs promoted, and exits 0. Otherwise it kills the candidate, confirms the old release still answers, logs rolled_back with the reason, and exits 1.

That deployer made its decisions inside the code, and one of them, the deadline, arrived as a command-line argument. This lesson argues that the numbers behind those decisions are an operator's call and belong in a reviewed config file. There are three numbers and one choice:

• Progress deadline: how long a candidate gets to become ready.
• Poll interval: how often readiness is checked.
• Unavailable and surge budgets: how much of the old capacity may be down during the change, and how much extra may run beside it.
• Who acts when the deadline passes: the deployer rolls back by itself, or it stops and escalates to a person.

Software engineering view. The deployer is a small program that reads a policy. The policy file needs a type-like guard that refuses nonsense before any release is touched. Nickel provides that guard as a contract. A contract is a check attached to a value, and Nickel runs it when it evaluates the config. You can write one with std.contract.from_validator, which takes a function that returns 'Ok or 'Error { message = ... }. A field that a record contract declares without a default is required, so a policy with no deadline is rejected rather than silently running with none.

Three refusals matter for edge-svc:

• No deadline. The hypothesis can never fail. A candidate that hangs holds the rollout forever.
• A deadline over 10 minutes. A bad release is found, but far too late.
• An unavailable budget over 25%. The change itself becomes an outage.

The 25% and 10-minute bounds are this project's choice for edge-svc. They sit at Kubernetes' defaults and are not a universal rule. A service with different needs would choose different bounds in its own contract.

Data view. The deploy log is a fixed-schema record, one JSON line per action: ts, event, version, detail. ts is Unix seconds. event is candidate_started, promoted, or rolled_back. Because every line has the same fields, the log is a table. Nushell reads it as one and answers the operator's questions with ordinary filters: how many releases were attempted, how many rolled back and why, and which release is live. One rule keeps the answers honest. Only a promoted event changes which release is live, so a rolled-back release must never be reported live.

Scope. Today's deployer still takes the deadline as an argument. Wiring it to read policy.json is a later step, and it is the first stretch task in the lab. Until then the policy file is a checked statement of what the operator intends, and it already has teeth: the contract refuses bad policies, and the log query reads what actually happened.

Worked example

The example is edge-svc's own rollout policy and the deploy log its deployer wrote in the companion lab. It goes in four steps: write the contract, write the policy and export it, watch the contract refuse a bad policy, then query the log. The last two subsections map the same decisions onto Kubernetes and onto systemd.

The contract

Start with the guard. Bounded builds a validator for an integer in a range and puts the field name in the error message. RolloutPolicy applies it to each numeric field and restricts on_stall to two named choices.

# rollout-policy.ncl: the contract every service's rollout policy must satisfy.
let Bounded = fun lo hi label =>
  std.contract.from_validator (fun n =>
    if std.is_number n && std.number.is_integer n && n >= lo && n <= hi then 'Ok
    else 'Error { message = "%{label} must be an integer from %{std.to_string lo} to %{std.to_string hi}" })
in
{
  RolloutPolicy = {
    progress_deadline_ms | Bounded 100 600000 "progress_deadline_ms",
    poll_ms | Bounded 50 5000 "poll_ms",
    max_unavailable_pct | Bounded 0 25 "max_unavailable_pct",
    max_surge_pct | Bounded 0 100 "max_surge_pct",
    on_stall | [| 'rollback, 'escalate |],
  },
}

The ranges are: a deadline from 100 ms to 600000 ms (10 minutes), a poll interval from 50 ms to 5 s, an unavailable budget of 0 to 25 percent, and a surge budget of 0 to 100 percent. The cap on surge is our own choice, not a Kubernetes default. The 'rollback and 'escalate values are Nickel enum tags, and anything else in on_stall is refused.

The policy

policy.ncl imports the contract and checks one concrete policy against it with let p | RolloutPolicy = .... It then exports the policy plus one derived value, the number of polls that fit inside the deadline.

# policy.ncl: edge-svc's rollout policy, checked by the contract, plus what it implies.
let { RolloutPolicy, .. } = import "rollout-policy.ncl" in
let p | RolloutPolicy = {
  progress_deadline_ms = 1500,
  poll_ms = 100,
  max_unavailable_pct = 0,
  max_surge_pct = 100,
  on_stall = 'rollback,
}
in
{
  edge_svc = p & {
    max_polls = std.number.floor (p.progress_deadline_ms / p.poll_ms),
  },
}

Here edge-svc gets 1500 ms to become ready, is polled every 100 ms, may lose none of its capacity, may run one extra copy beside the old one, and rolls back on a stall. For a single-instance service that is what the Rust deployer already does: the old release keeps serving while one extra candidate runs. Export it:

$ nickel --version
nickel-lang-cli nickel 1.18.0 (rev e6515c4)
$ nu --version
0.116.1
$ nickel export policy.ncl
{
  "edge_svc": {
    "max_polls": 15,
    "max_surge_pct": 100,
    "max_unavailable_pct": 0,
    "on_stall": "rollback",
    "poll_ms": 100,
    "progress_deadline_ms": 1500
  }
}

1500 ms divided by 100 ms gives 15 polls. The deployer would need that number to bound its loop, and the config can compute it so nobody keeps two numbers in sync by hand.

The contract refusing a bad policy

Now take the same record with a 900000 ms deadline, which is 15 minutes, and save it as bad.ncl beside the contract:

let { RolloutPolicy, .. } = import "rollout-policy.ncl" in
{ progress_deadline_ms = 900000, poll_ms = 100, max_unavailable_pct = 0, max_surge_pct = 100, on_stall = 'rollback } | RolloutPolicy

Nickel refuses to export it. The output below was captured in the lab container, whose working directory is /w, so your paths will differ:

$ nickel export bad.ncl
error: contract broken by the value of `progress_deadline_ms`
       progress_deadline_ms must be an integer from 100 to 600000
  ┌─ /w/rollout-policy.ncl:9:28
  │
9 │     progress_deadline_ms | Bounded 100 600000 "progress_deadline_ms",
  │                            ----------------------------------------- expected type
  │
  ┌─ /w/bad.ncl:2:26
  │
2 │ { progress_deadline_ms = 900000, poll_ms = 100, max_unavailable_pct = 0, max_surge_pct = 100, on_stall = 'rollback } | RolloutPolicy
  │                          ^^^^^^ applied to this expression

exit 1

The message names the field and states the allowed range in the words our validator wrote. The first location points at the contract, the second points at the offending value, and the exit status is 1, so a script or CI step that runs nickel export stops there. Nothing about the release has been touched yet. The lab checker also tries a policy with no deadline at all, and a policy with a 50% unavailable budget. Its output for those two cases is in the Lab section.

The log as a table

The query below reads the log, keeps lines that start with {, and parses each one into a record. That gives a table with the columns ts, event, version, and detail. It then keeps only the two outcome events, because candidate_started is not an outcome.

# rollouts.nu: summarise a deployer JSON-lines log for an operator.
# Usage: nu rollouts.nu <log>   prints one JSON object.
def main [log: path] {
  let events = (open --raw $log | lines | where {|l| $l starts-with '{' } | each {|l| $l | from json })
  let outcomes = ($events | where event in ['promoted' 'rolled_back'])
  let promoted = ($outcomes | where event == 'promoted')
  let rolled = ($outcomes | where event == 'rolled_back')
  {
    releases: ($outcomes | length)
    promoted: ($promoted | length)
    rolled_back: ($rolled | length)
    reasons: ($rolled | each {|e| { version: $e.version, reason: ($e.detail | split row '; ' | first) } })
    live: (if ($promoted | is-empty) { null } else { $promoted | last | get version })
  } | to json --raw
}

Three details are worth reading closely:

• **releases counts outcomes, not lines.** One attempt writes a candidate_started line and one outcome line. Counting every line double counts.
• **reasons keeps the text of detail before the first ; .** Only rolled-back entries produce a reason.
• **live comes from the promoted list only.** A rolled-back version is never in that list, so it cannot be reported live. If nothing was ever promoted, live is null.

There are two fixtures. gated.log is the log the gated deployer wrote in the lab image. In it, v1 to v2 was promoted. Then v3, which only becomes ready after 60 s, was tried with a 1.5 s deadline, and v4, which crashes at boot with exit 78, was tried after it. ungated.log is the log the companion lab's starter deployer wrote, which promoted everything after a 300 ms sleep.

$ nu rollouts.nu fixtures/gated.log
{"releases":3,"promoted":1,"rolled_back":2,"reasons":[{"version":"v3","reason":"not ready after 1500 ms"},{"version":"v4","reason":"candidate exited with exit status: 78 before ready"}],"live":"v2"}

$ nu rollouts.nu fixtures/ungated.log
{"releases":3,"promoted":3,"rolled_back":0,"reasons":[],"live":"v4"}

In the gated log, three releases were attempted and one was promoted. Two were rolled back, and each carries its reason. The live release is v2, even though v4 is the last one attempted. The ungated log has no rollbacks at all and v4 is live. The same query shows both histories, and the difference is the point of the gate.

The same decisions in Kubernetes

Nothing here needs a cluster. Kubernetes runs containers across machines; a Deployment is its object for "keep N copies of this release running and replace them gradually", and each copy runs in a Pod. The Kubernetes Deployment page is the reference, and the table maps each field in our policy to the setting the Deployment controller exposes for the same decision.

Our deadline is in milliseconds and Kubernetes' is in seconds. Our 600000 ms cap is Kubernetes' default 600 s, and our 25% cap on unavailable matches its default. For a single Pod, the defaults work out like this: 25% unavailable rounds down to 0, and 25% surge rounds up to 1. That is the shape the Rust deployer already has.

The last row is the interesting one. When the deadline passes, the Deployment status gets a condition with type Progressing, status "False", and reason ProgressDeadlineExceeded. The docs say Kubernetes takes no action on a stalled Deployment other than to report that condition, and that "Higher level orchestrators can take advantage of it and act accordingly, for example, rollback the Deployment to its previous version." The controller also keeps retrying the Deployment. So in Kubernetes the question of who acts on a stall is answered outside the Deployment. Our on_stall field makes that decision explicit and local: 'rollback means the deployer acts, and 'escalate means a person does. When someone does want to go back, the manual route is kubectl rollout undo deployment/<name>, with --to-revision=N for a specific revision and kubectl rollout history to list the revisions.

systemd has no rollback

A systemd service unit names one ExecStart command, and Restart= reruns that same command. systemd keeps no history of earlier versions of a service's binary, so it has no rollback. Going back to a previous release is something you build around it, for example two installed versions and a symlink you switch. The current pointer file in the Rust deployer plays that role. It is the one atomic switch between the old release and the new one, and rolling back means not switching it.

The Site Reliability Workbook makes the same argument at a larger scale. Chapter 16, "Canarying Releases", says in "Release Engineering Principles" that smaller, frequent releases are cheaper to roll back. In "Balancing Release Velocity and Reliability" it says that without a rollback option, user impact lasts longer. It describes a blue/green deployment as one where "rollback is a trivial reversal of the router change".

Lab

Setup. Install Nickel 1.18 and Nushell 0.116, or use any container that has both. Then:

cd lab/starter && sh ../check.sh

check.sh is a POSIX shell script that prints one result line per check (the word RESULT, the check name, then PASS or FAIL) and exits 1 if any check fails. It has six checks:

1. policy_exports_bounded: policy.ncl exports, has a progress_deadline_ms of at most 600000, and has a max_unavailable_pct of at most 25.
2. contract_rejects_missing_deadline: the contract refuses a policy with no deadline.
3. contract_rejects_long_deadline: the contract refuses a 900000 ms deadline.
4. contract_rejects_wide_unavailable: the contract refuses a 50% unavailable budget.
5. query_gated_log: rollouts.nu gives the exact expected summary for fixtures/gated.log.
6. query_ungated_log: rollouts.nu gives the exact expected summary for fixtures/ungated.log.

Before you change anything, all six fail. The starter's contract types each field as a plain Number with an optional deadline and no bounds. Its policy.ncl has no deadline and sets the unavailable budget to 100%. Its rollouts.nu counts every log line as a promoted release and calls the last line live. This is the starter's output for the two log summaries:

gated.log   -> {"releases":6,"promoted":6,"rolled_back":0,"reasons":[],"live":"v4"}
ungated.log -> {"releases":6,"promoted":6,"rolled_back":0,"reasons":[],"live":"v4"}

The starter reports six releases for three attempts, and it reports v4 live on the gated log when v4 was rolled back.

Tasks. Try each one before you compare with the worked example above.

1. Bound the contract and make the deadline required. In rollout-policy.ncl, replace the plain Number fields with bounded ones: a deadline from 100 to 600000, a poll interval from 50 to 5000, an unavailable budget from 0 to 25, and a surge budget from 0 to 100. Remove whatever makes the deadline optional. Put the field name in each error message so a refusal says which number is wrong.
2. Give the policy a deadline and a 0% unavailable budget. In policy.ncl, add progress_deadline_ms and set max_unavailable_pct to 0. The worked example uses 1500 ms.
3. Fix the query. In rollouts.nu, count only promoted and rolled_back events as releases, report the reason for each rollback, and take live from the last promoted event only.

Run sh ../check.sh after each task. Task 1 turns the three contract checks green. It also turns policy_exports_bounded from a quiet failure into a loud one: the starter policy has no deadline, so the export now stops with missing definition for progress_deadline_ms. That error is the contract doing its job on your own file, and task 2 clears it. Task 3 turns the last two checks green. If query_gated_log still fails with "live":"v4", your query is taking the last line of the log, not the last promotion.

Expected result. With all three tasks done, the checker prints these lines:

exported: {"max_polls":15,"max_surge_pct":100,"max_unavailable_pct":0,"on_stall":"rollback","poll_ms":100,"progress_deadline_ms":1500}
RESULT policy_exports_bounded PASS
contract refused { max_unavailable_pct = 0, }: error: missing definition for `progress_deadline_ms`
RESULT contract_rejects_missing_deadline PASS
contract refused { progress_deadline_ms = 900000, max_unavailable_pct = 0, }: progress_deadline_ms must be an integer from 100 to 600000
RESULT contract_rejects_long_deadline PASS
contract refused { progress_deadline_ms = 1500, max_unavailable_pct = 50, }: max_unavailable_pct must be an integer from 0 to 25
RESULT contract_rejects_wide_unavailable PASS
gated.log   -> {"releases":3,"promoted":1,"rolled_back":2,"reasons":[{"version":"v3","reason":"not ready after 1500 ms"},{"version":"v4","reason":"candidate exited with exit status: 78 before ready"}],"live":"v2"}
ungated.log -> {"releases":3,"promoted":3,"rolled_back":0,"reasons":[],"live":"v4"}
RESULT query_gated_log PASS
RESULT query_ungated_log PASS

Stretch.

• (a) Add a Kubernetes rule to the contract. The Deployment docs say maxUnavailable cannot be 0 if maxSurge is 0, since with neither budget the rollout could neither stop an old Pod nor start a new one. Add that rule to the contract, so a policy with max_unavailable_pct = 0 and max_surge_pct = 0 is refused. It involves two fields at once, so it needs a check on the whole record, not on one field. There is no reference solution in the lab. Write a probe like the ones in check.sh and confirm that the contract refuses it and still accepts the real policy.
• (b) Have the deployer read the policy. Export policy.json with nickel export policy.ncl > policy.json, which is what the checker does, and change the Rust deployer from the companion lesson to read progress_deadline_ms from edge_svc instead of taking a deadline argument. Decide what should happen if someone still passes the argument: does the policy win, or does the argument? Write down your choice and why.

Other paths

• Let Kubernetes hold the policy. In a cluster the same decisions live in the Deployment: .spec.progressDeadlineSeconds, maxUnavailable, maxSurge, and a readiness probe. The controller reports a stall and does not roll back. You still need a person or a higher-level orchestrator to act, with kubectl rollout undo deployment/<name>. Our contract plays the role of a review gate that Kubernetes does not have: it refuses a Deployment-shaped policy before it is applied.
• Build rollback around systemd. Install each release in its own directory and point a symlink at the live one. The unit's ExecStart runs through the symlink. Switching the symlink back is the rollback. This is the same structure as the current pointer file, with systemd as the supervisor and not the deployer.
• Blue/green. The Site Reliability Workbook describes running the new release beside the old one and moving traffic with the router. The rollback is then "a trivial reversal of the router change". Our deployer is a one-instance version of the same idea, with the pointer file as the router.
• A different config checker. A JSON Schema could state the same ranges against policy.json. It checks a finished document and cannot compute max_polls, so the derived value would be written by hand or by a second program, and the two could drift. A different log query tool such as jq could produce the same summary. The table view is what makes the Nushell version easy to extend, for example with a column of rollbacks per version.

Citations

• Kubernetes documentation, "Deployments", https://kubernetes.io/docs/concepts/workloads/controllers/deployment/. Sections "Rolling Back a Deployment", "Progress Deadline Seconds", "Failed Deployment", "Max Unavailable", and "Max Surge".
• Kubernetes documentation, "Liveness, Readiness, and Startup Probes", https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/.
• Kubernetes documentation, "Configure Liveness, Readiness and Startup Probes", https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/, the periodSeconds field.
• Beyer, Murphy, Rensin, Kawahara, Thorne (eds.), The Site Reliability Workbook, O'Reilly, 2018, chapter 16, "Canarying Releases", sections "Release Engineering Principles" and "Balancing Release Velocity and Reliability", and the description of blue/green deployment.
• Nickel 1.18 documentation, "Contracts", https://nickel-lang.org/user-manual/contracts, including std.contract.from_validator.
• Nushell documentation, "Working with tables", https://www.nushell.sh/book/working_with_tables.html.
• systemd.service(5), https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html.