Probation as operator policy A soak contract and a rolled-back-for-budget query

Refuse a probation policy with a bad window, a loose budget, or a rollback target other than the last confirmed release, then ask every ledger and report which releases were rolled back for budget and whether a soaking release was ever called good.

Lab download. Starter is the tree you edit. Worked answer is a separate file; open it after the starter passes. Nickel 1.18 and Nushell 0.116. The reference tree's jq-report.sh also needs jq 1.7. All lab requirements.

O-DevOps · A · arc: self-healing-edge · langs: nickel, nushell · CKA-workloads-rollout, DOP-1.4

By Evan Eugene Pincham.

Concept

A release is one versioned build of a service, written here as v1, v2 and v3. Promotion is the moment a newly started release takes over from the one that was serving. A promoted release is not yet trusted, so it stays on probation. For a fixed span called the soak window, a program called the deployer counts how often the new release's process exits. The restart budget is the number of those exits the deployer absorbs by restarting the process, and the exit after the budget is spent fails the deploy. A release that reaches the end of its window with the budget intact is confirmed. The most recently confirmed release is stored as last_good, and last_good is also the rollback target, meaning the release the deployer returns to when probation fails. The release that was already serving before any promotion counts as confirmed by definition.

The companion lesson, "A gated rollback that treats a restart storm as a failed deploy, in Rust", built a tool named soak that makes these decisions. soak apply PLAN reads a plan file of timestamped events and prints a ledger, an append-only list of JSON lines in which each line records one decision. This lesson calls that text apply output. The ledger has four kinds of line. A restarting line carries the version, the attempt number and the maximum. A rolled_back line carries the version, the from and to releases, the reason restart_budget_exceeded, the number of exits in the window, the maximum and the window length in milliseconds. A confirmed line carries the version and its restart count. A restarted_post_soak line marks an exit after the window closed, which never triggers a rollback of the deploy. soak report PLAN prints key: value lines named soak_ms, max_restarts, current, last_good, candidate, candidate.restarts and deploy, and this text is report output. The deploy field is one of serving, soaking, confirmed or rolled_back. In the companion lesson the soak window was 30000 ms and the restart budget was 2, so a third exit inside 30 seconds rolls the release back.

The tool is a decision kernel over fixture plans. It runs on the CPU only, starts no live processes and talks to no cluster. Time moves only when an event's ts field says so. What is real is the rule set: the window, the budget, the rollback target and the honest ledger. Starting and probing releases is simulated. The fixtures used here are real apply and report output from that tool, except forged.apply and forged.report, which were edited by hand to contain lies on purpose.

The numbers 30000 and 2 are choices that somebody makes. A policy is the written record of those choices: a window length soak_ms, a budget max_restarts and a rollback_target rule. This lesson takes the operator's side of the deploy and does two jobs. A Nickel contract refuses a bad policy before any deployer reads it. Nickel is a configuration language, and a contract is a check attached to a value: when the configuration is evaluated, a value that fails its contract stops the evaluation with an error message. A Nushell query then reads the ledger the deployer wrote and reports every violation. Nushell is a shell that treats structured data as tables, and a violation here means a line in a ledger or report that contradicts the policy or a rollback rule.

The design rests on four decisions, and each has a reason.

A policy is checked before use because a bad number fails silently. The Site Reliability Workbook, in its chapter on canarying releases, says a canary should be sizeable and last long enough to be representative, and that ending it after a handful of requests gives no useful signal. A soak window of 500 ms is that mistake written as a number. Kubernetes shows the same trap in its Deployment field .spec.minReadySeconds, which "defaults to 0 (the Pod will be considered available as soon as it is ready)". The budget needs a ceiling for a different reason. Armstrong's thesis introduces restart intensity with the supervisor setting {one_for_one,5,1000}: more than 5 restarts in 1000 seconds and the supervisor itself fails. The idea of N restarts inside a window is old, and it only protects anything when N is small and finite. A bound enforced at export time costs one failed command. The same mistake found after a deploy costs a crash loop that nobody stopped.

The ledger is checked afterwards because the deployer's own account is a claim, not evidence. A deployer built with different constants, a stale policy file and a hand edit all produce a ledger that looks plausible. Reading the ledger against the policy turns it from a claim into a checked record. The Workbook lists, among the requirements of a canary process, an evaluation process that decides whether a change is good or bad, and integration into the release process. A policy plus a ledger check is a small version of that evaluation step.

**The rollback target is the symbolic value last_confirmed, never a version the release names.** The promoted event carries a previous field, and that field is written by the release being promoted. After v2 is rolled back and v3 is promoted, v3 can still claim previous is v2. Following that claim returns the deploy to a release that already failed probation. The value last_good is computed from confirmed lines in the ledger, which are evidence instead of a claim, so the policy pins the rule to that and refuses the alternative.

A soaking release is never good, because readiness at one instant is a weak test. A release can pass its probe and then crash-loop. The exam objective labelled DOP-1.4 on this page is DOP-C02 version 1.6, Task Statement 1.4, "Implement deployment strategies for instance, container, and serverless environments", which covers evaluating canary and blue-green deployments and troubleshooting deployment issues. Probation is the evaluation step that decides whether a promoted release stays, and a report that calls the release good before that step ends has skipped the evaluation.

Worked example

The running example follows one release, v2, through probation under the policy 30000 ms and 2 restarts, in three steps: the Nickel contract and policy that fix the numbers, the apply and report fixtures that record what happened, and the Nushell query that reads them back. Each step touches the two subjects the day is about, a small deploy state machine with a restart budget (software engineering) and a JSON Lines ledger with a fixed schema (data).

The contract

The contract file defines two reusable checks and exports a record that uses them.

# probation-policy.ncl: the contract every probation policy must satisfy.
# `nickel export` refuses a policy whose window or budget is out of range
# or whose rollback target is anything but the last confirmed release.
let Bounded = fun lo hi label =>
  std.contract.from_validator (fun n =>
    if std.is_number n && std.number.is_integer n && n >= lo && n <= hi then 'Ok
    else 'Error { message = "%{label} must be an integer from %{std.to_string lo} to %{std.to_string hi}" })
in
let LastConfirmed = std.contract.from_validator (fun s =>
  if s == "last_confirmed" then 'Ok
  else 'Error { message = "rollback_target must be last_confirmed" })
in
{
  ProbationPolicy = {
    soak_ms | Bounded 1000 600000 "soak_ms",
    max_restarts | Bounded 0 5 "max_restarts",
    rollback_target | LastConfirmed,
  },
}

Bounded is a function that takes a lower bound, an upper bound and a label, and returns a contract. std.contract.from_validator builds that contract from a plain function that answers 'Ok or 'Error with a message. The test requires a number, requires it to be an integer, and requires it to sit between the bounds. A window of 30000.5 ms is a number but has no meaning, so the integer test is part of the contract. The message is assembled with %{...} interpolation, which is why a refusal reads "soak_ms must be an integer from 1000 to 600000" and names the field it refused.

LastConfirmed is an equality test against one string. The final record exports ProbationPolicy, whose fields carry contracts through the | annotation. Applying that record as a contract to another record checks each matching field. The bounds are judgments made by this policy: 1000 ms to 600000 ms for the window, and 0 to 5 for the budget. A budget of 0 is allowed by the contract, and under the companion tool's rule that an exit beyond the budget fails the deploy, it means the first exit rolls back. The contract does not prove those bounds correct. It makes them explicit and enforces them in one place.

The policy itself is a short file that imports the contract and applies it.

# policy.ncl: today's probation policy for a promoted release.
let { ProbationPolicy, .. } = import "probation-policy.ncl" in
{
  soak_ms = 30000,
  max_restarts = 2,
  rollback_target = "last_confirmed",
} | ProbationPolicy

The first line of code imports the contract file and pulls out ProbationPolicy by destructuring, where the .. ignores any other fields. The record literal holds the three chosen values. The trailing | ProbationPolicy applies the contract to the whole record. Running nickel export policy.ncl > policy.json evaluates the file, and evaluation stops with the contract's message if any field fails. On success the command writes the JSON that the Nushell query will read, so a policy that never passed the contract never reaches the query. The checker output below shows the accepted policy and the four probe policies it refuses: soak_ms values of 500 and 700000, a max_restarts of 9, and a rollback_target of "previous". Each probe is today's policy with one field changed, so the refusal message names the one field that broke the contract.

policy accepted: {"max_restarts":2,"rollback_target":"last_confirmed","soak_ms":30000}
RESULT policy_accepted PASS
policy short_window: refused: soak_ms must be an integer from 1000 to 600000
RESULT rejects_short_window PASS
policy long_window: refused: soak_ms must be an integer from 1000 to 600000
RESULT rejects_long_window PASS
policy big_budget: refused: max_restarts must be an integer from 0 to 5
RESULT rejects_big_budget PASS
policy previous_target: refused: rollback_target must be last_confirmed
RESULT rejects_previous_target PASS

The ledgers

The query runs over the apply and report fixtures from the companion lesson. The storm fixture is a release that exits three times in nine seconds.

{"attempt":1,"event":"restarting","max":2,"ts":3000,"version":"v2"}
{"attempt":2,"event":"restarting","max":2,"ts":6000,"version":"v2"}
{"event":"rolled_back","exits_in_window":3,"from":"v2","max":2,"reason":"restart_budget_exceeded","to":"v1","ts":9000,"version":"v2","window_ms":30000}

Two exits were absorbed as restarts, and the third at 9000 ms spent the budget and rolled v2 back to v1. The forged fixture is the same story told twice, with one lie added.

{"attempt":1,"event":"restarting","max":2,"ts":2000,"version":"v2"}
{"attempt":2,"event":"restarting","max":2,"ts":3000,"version":"v2"}
{"event":"rolled_back","exits_in_window":3,"from":"v2","max":2,"reason":"restart_budget_exceeded","to":"v1","ts":4000,"version":"v2","window_ms":30000}
{"attempt":1,"event":"restarting","max":2,"ts":52000,"version":"v3"}
{"attempt":2,"event":"restarting","max":2,"ts":53000,"version":"v3"}
{"event":"rolled_back","exits_in_window":3,"from":"v3","max":2,"reason":"restart_budget_exceeded","to":"v2","ts":54000,"version":"v3","window_ms":30000}

The first three lines are an honest rollback of v2 to v1 at 4000 ms. The next three lines are v3 storming at 52000 ms and 53000 ms, and its rollback at 54000 ms. That last line says "to":"v2", but v2 failed probation fifty seconds earlier, so the only defensible target is v1. This is the chain scenario with the target replaced by the stale previous claim, and it is the kind of line a deployer with the wrong target rule would write.

The query

The Nushell script reads one ledger or one report plus the exported policy and prints an answer with a verdict.

# probation-report.nu: which releases were rolled back for budget, and how many restarts first?
# Reads one `soak apply` JSONL ledger or one `soak report` key: value file, plus the
# exported policy (policy.json).
# Usage: nu probation-report.nu <apply-or-report> <policy.json>
# Exit 0 when the file agrees with the policy and the rollback rules, 1 when a line
# contradicts them, 2 when a file cannot be opened.

def pick [rows: table, name: string] {
  let hit = ($rows | where key == $name)
  if ($hit | is-empty) { "missing" } else { $hit | get value | first }
}

def main [input: path, policy: path] {
  if not ($input | path exists) or not ($policy | path exists) {
    print -e $"probation-report.nu: cannot open ($input) or ($policy)"
    exit 2
  }
  let pol = (open --raw $policy | from json)
  let raw = (open --raw $input)
  let nonempty = ($raw | lines | each {|l| $l | str trim } | where {|l| $l != ""})
  if ($nonempty | is-empty) {
    print -e $"probation-report.nu: cannot open ($input)"
    exit 2
  }
  mut v = []
  if ($nonempty | first | str starts-with "{") {
    let acts = ($nonempty | each {|l| $l | from json })
    let confirmed = ($acts | where {|a| $a.event == "confirmed" } | each {|a| $a.version })
    let post = ($acts | where {|a| $a.event == "restarted_post_soak" } | length)
    mut seen = []
    mut names = []
    mut firsts = []
    mut targets = []
    mut last_confirmed = ""
    for a in $acts {
      if $a.event == "confirmed" { $last_confirmed = $a.version }
      if $a.event == "rolled_back" {
        let from = ($a.from? | default $a.version)
        let n = ($acts | where {|r| ($r.event == "restarting") and ($r.version == $from) and ($r.ts <= $a.ts) } | length)
        $names = ($names | append $from)
        $firsts = ($firsts | append $"($from)=($n)")
        $targets = ($targets | append $"($from)->($a.to)")
        if $a.to in $seen {
          $v = ($v | append $"rollback of ($from) went to ($a.to), which was itself rolled back")
        }
        if ($last_confirmed != "") and ($a.to != $last_confirmed) {
          $v = ($v | append $"rollback of ($from) went to ($a.to), last confirmed is ($last_confirmed)")
        }
        if $n != $pol.max_restarts {
          $v = ($v | append $"($from) rolled back after ($n) restarts, policy allows ($pol.max_restarts)")
        }
        if $a.window_ms? != $pol.soak_ms {
          $v = ($v | append $"($from) ledger window ($a.window_ms?) differs from policy soak_ms ($pol.soak_ms)")
        }
        $seen = ($seen | append $from)
      }
    }
    print $"probation-report: ($input | path basename)"
    print "kind: apply"
    print $"events: ($acts | length)"
    print $"rolled_back_for_budget: (if ($names | is-empty) { 'none' } else { $names | str join ', ' })"
    print $"restarts_first: (if ($firsts | is-empty) { 'none' } else { $firsts | str join ', ' })"
    print $"rollback_targets: (if ($targets | is-empty) { 'none' } else { $targets | str join ', ' })"
    print $"confirmed: (if ($confirmed | is-empty) { 'none' } else { $confirmed | str join ', ' })"
    print $"post_soak: ($post)"
  } else {
    let rows = ($raw | lines | parse -r '^(?<key>[a-z_.]+): (?<value>.*)$')
    let deploy = (pick $rows "deploy")
    let current = (pick $rows "current")
    let last_good = (pick $rows "last_good")
    let candidate = (pick $rows "candidate")
    let restarts = (pick $rows "candidate.restarts")
    if ($deploy == "soaking") and ($last_good == $candidate) {
      $v = ($v | append $"soaking release ($candidate) is already last_good")
    }
    if ($deploy == "confirmed") and ($last_good != $candidate) {
      $v = ($v | append $"deploy confirmed but last_good is ($last_good), not ($candidate)")
    }
    if ($deploy == "rolled_back") and ($current != $last_good) {
      $v = ($v | append $"rolled back but current is ($current), not last_good ($last_good)")
    }
    if (pick $rows "soak_ms") != ($pol.soak_ms | into string) {
      $v = ($v | append $"report soak_ms differs from policy soak_ms ($pol.soak_ms)")
    }
    if (pick $rows "max_restarts") != ($pol.max_restarts | into string) {
      $v = ($v | append $"report max_restarts differs from policy max_restarts ($pol.max_restarts)")
    }
    let good = ($deploy == "confirmed") and ($v | is-empty)
    print $"probation-report: ($input | path basename)"
    print "kind: report"
    print $"deploy: ($deploy)"
    print $"current: ($current)"
    print $"last_good: ($last_good)"
    print $"candidate: ($candidate)"
    print $"restarts: ($restarts)"
    print $"good: (if $good { 'yes' } else { 'no' })"
  }
  print $"violations: ($v | length)"
  for line in $v { print $"  ($line)" }
  if ($v | is-not-empty) { exit 1 }
}

The script has four parts, and each one maps to a decision in the concept section.

The opening is plumbing. pick looks up one key in the parsed report rows and returns the string missing when the key is absent, so a malformed report yields visible "missing" values instead of a crash. def main [input: path, policy: path] makes the script take two command-line arguments. If either path does not exist, or the input holds only blank lines, the script prints to standard error and exits with code 2. That separates "the file could not be read" from "the file was read and is wrong". A first non-blank line that starts with { selects the apply branch, and anything else is treated as a report.

The apply branch walks the ledger in order. It uses mut accumulators and a for loop because Nushell closures cannot capture mutable variables, so the where closures only touch values bound with let. For each rolled_back line, the release is from, falling back to version if from is absent. The variable n counts the restarting lines for that release at or before the rollback timestamp, and that count is the "restarts first" figure. Three lists collect the names, the v2=2 pairs and the v2->v1 targets for printing.

Four comparisons then produce violations. A target already in $seen was itself rolled back earlier. A target that differs from the most recent confirmed line breaks the last-confirmed rule, but the comparison only starts once a confirmed line has appeared, because the baseline release has no such line. A restart count n that differs from max_restarts means the ledger came from a different budget. A window_ms that differs from soak_ms means it came from a different window. The test is !=, not >, because the kernel rolls back on the exit after the budget, so exactly max_restarts restarting lines must precede a rollback. The script reads only soak_ms and max_restarts from the policy, because the contract has already pinned rollback_target and the script applies the last-confirmed rule directly.

The report branch parses each line with a regular expression into key and value columns and applies three rules about deploy. A soaking deploy whose last_good equals the candidate promoted the release early. A confirmed deploy needs last_good to equal the candidate. A rolled-back deploy needs current to equal last_good. Two more checks compare the report's soak_ms and max_restarts with the policy. The variable good is true only when the deploy is confirmed and no violation exists, so a soaking report is never good. The end of the script prints the violation count, lists each violation indented by two spaces, and exits 1 when the list is not empty.

Reading the verdicts

Running nu probation-report.nu fixtures/storm.apply policy.json answers the opening question for the storm.

probation-report: storm.apply
kind: apply
events: 3
rolled_back_for_budget: v2
restarts_first: v2=2
rollback_targets: v2->v1
confirmed: none
post_soak: 0
violations: 0

v2 was rolled back for budget after two restarts, the target was v1, and nothing in the file contradicts the policy, so the exit code is 0. The chain fixture gives rolled_back_for_budget: v2, v3 with rollback_targets: v2->v1, v3->v1. The survivor and slow-burn fixtures name no rollback and list v2 as confirmed. The slow-burn fixture also counts one restarted_post_soak line, which is an exit after the window closed and therefore not a budget event.

The forged ledger gives the same names and a different verdict.

probation-report: forged.apply
kind: apply
events: 6
rolled_back_for_budget: v2, v3
restarts_first: v2=2, v3=2
rollback_targets: v2->v1, v3->v2
confirmed: none
post_soak: 0
violations: 1
  rollback of v3 went to v2, which was itself rolled back

The script exits 1 here. The restart counts and the window agree with the policy, so the only contradiction is the target v3 to v2. Reports are judged by the same script.

soak_ms: 30000
max_restarts: 2
current: v2
last_good: v2
candidate: v2
candidate.restarts: 1
deploy: soaking

The honest soaking report has last_good: v1. This forged copy changes that line to v2 while leaving deploy: soaking, which claims the candidate is already the rollback target even though its window is still open. If the deployer then rolled back, it would "return" to the release that was failing. The output for the honest report and the forged one shows the difference.

probation-report: soaking.report
kind: report
deploy: soaking
current: v2
last_good: v1
candidate: v2
restarts: 1
good: no
violations: 0

probation-report: forged.report
kind: report
deploy: soaking
current: v2
last_good: v2
candidate: v2
restarts: 1
good: no
violations: 1
  soaking release v2 is already last_good

Both say good: no, because neither deploy is confirmed. Only the forged one exits 1. The honest soaking report exits 0 because it is truthful about being unfinished.

The last probe feeds a valid policy that differs from the one the ledger was written under.

probation-report: storm.apply
kind: apply
events: 3
rolled_back_for_budget: v2
restarts_first: v2=2
rollback_targets: v2->v1
confirmed: none
post_soak: 0
violations: 1
  v2 rolled back after 2 restarts, policy allows 3

The policy here sets max_restarts = 3, which the contract accepts because 3 lies between 0 and 5. The ledger rolled back after 2 restarts, so the two disagree. Either the deployer ran with a different budget or the policy was changed afterwards, and the query reports the drift rather than choosing a side.

Pitfalls

• Symptom: a probe policy with soak_ms = 500 exports without complaint. Cause: the contract is Number, which checks only the type and accepts every number, so nothing tests the range or whether the value is an integer.
• Symptom: the query prints events: 0 and violations: 0 for a ledger that clearly has lines. Cause: the script never opens the input file, so its answer cannot depend on the content.
• Symptom: a hand-written ledger that rolls v2 back to a release named v0, with no confirmed line anywhere, exits 0 with rollback_targets: v2->v0 and violations: 0. Cause: the last-confirmed comparison starts only after a confirmed line appears, because the baseline release has no such line in an apply ledger. Only a target that was itself rolled back is caught before that point. The extra run log captures this output. A hypothetical fix is to hand the script the baseline version as a third argument, so the first rollback can be checked as well.
• Symptom: a wrapper script around the jq program reports success while the output says violations: 1. Cause: jq exits 0 whenever the program ran, so the verdict has to be computed from the output, as the Nushell script does with its own exit code.
• Symptom: a ledger that looks fine exits 1 with policy allows 3. Cause: the ledger and the policy were produced under different budgets. The query is working, and the fix is to find which of the two is stale.

Lab

Two starter files fail on purpose. The starter contract is below.

# probation-policy.ncl: TODO. Make this contract refuse a bad probation policy.
# Today it accepts any number and any string.
# Needed: soak_ms an integer 1000..600000, max_restarts an integer 0..5,
# rollback_target exactly "last_confirmed" (use std.contract.from_validator).
{
  ProbationPolicy = {
    soak_ms | Number,
    max_restarts | Number,
    rollback_target | String,
  },
}

The contract fields are typed Number, Number and String, so every probe policy is accepted: 500, 700000, 9 and "previous" all pass. The comment lists what the finished file needs. The record shape is already correct, so the work is replacing three type names with validators and keeping the exported name ProbationPolicy. The checker prints the contract's own message for each refusal, so messages shaped like those in the worked example are the target.

The starter query is a stub.

  print $"probation-report: ($input | path basename)"
  print "kind: apply"
  print "events: 0"
  print "rolled_back_for_budget: none"
  print "restarts_first: none"
  print "rollback_targets: none"
  print "confirmed: none"
  print "post_soak: 0"
  print "deploy: soaking"
  print "good: no"
  print "violations: 0"

This excerpt is the whole of the starter's output, preceded by a check that the input file exists. The script prints a fixed answer: no rollback, no confirmation, zero post-soak exits, zero violations. It also prints deploy: soaking and good: no, and it exits 0 for every file that exists, which is why it reports kind: apply even for a report file. The task is to replace the stub with the real parsing and judging: read the policy, parse the ledger, count restarts, name the rolled-back releases, flag the violations from the worked example, and judge reports. The exit codes are 0 for agreement, 1 for a contradiction and 2 for a file that cannot be opened.

policy.ncl from the worked example sits beside the two starter files. If it is absent, copy it from the worked example. The task is to finish both files until the checker passes. Run sh check.sh beside the files. It needs Nickel 1.18 and Nushell 0.116, and it needs no network. The code on this page was run under Nushell 0.116, and its behavior under other Nushell versions is unverified.

The checker runs 14 checks.

On the starter, the checker passes 3 checks and fails 11. The passes are policy_accepted, soaking_not_good and missing_file_exit_2. The failures are rejects_short_window, rejects_long_window, rejects_big_budget, rejects_previous_target, storm_named_with_restarts, survivor_not_rolled_back, slowburn_not_counted, chain_both_named, forged_target_named, forged_report_named and policy_drift_named. The starter's first lines show the contract problem directly.

policy accepted: {"max_restarts":2,"rollback_target":"last_confirmed","soak_ms":30000}
RESULT policy_accepted PASS
policy short_window: accepted
RESULT rejects_short_window FAIL

The full list of starter results follows.

RESULT policy_accepted PASS
RESULT rejects_short_window FAIL
RESULT rejects_long_window FAIL
RESULT rejects_big_budget FAIL
RESULT rejects_previous_target FAIL
RESULT storm_named_with_restarts FAIL
RESULT survivor_not_rolled_back FAIL
RESULT slowburn_not_counted FAIL
RESULT chain_both_named FAIL
RESULT forged_target_named FAIL
RESULT soaking_not_good PASS
RESULT forged_report_named FAIL
RESULT policy_drift_named FAIL
RESULT missing_file_exit_2 PASS

Three passes deserve an explanation. policy_accepted passes because the honest policy is in range, and a contract that accepts everything accepts it too. missing_file_exit_2 passes because the starter already tests that the input exists and exits 2 when it does not. soaking_not_good passes by luck. The check expects exit 0, deploy=soaking and good=no for soaking.report, and the stub's fixed answer happens to say deploy: soaking and good: no for every file. A script that decides nothing therefore passes this check, so it proves little until the checks around it pass too. The fixed answer fails forged_report_named in the same run, because that check needs exit 1 and one violation.

The starter output for one fixture shows the fixed answer.

probation-report: storm.apply
kind: apply
events: 0
rolled_back_for_budget: none
restarts_first: none
rollback_targets: none
confirmed: none
post_soak: 0
deploy: soaking
good: no
violations: 0

A finished lab passes all 14 checks and prints these results.

RESULT policy_accepted PASS
RESULT rejects_short_window PASS
RESULT rejects_long_window PASS
RESULT rejects_big_budget PASS
RESULT rejects_previous_target PASS
RESULT storm_named_with_restarts PASS
RESULT survivor_not_rolled_back PASS
RESULT slowburn_not_counted PASS
RESULT chain_both_named PASS
RESULT forged_target_named PASS
RESULT soaking_not_good PASS
RESULT forged_report_named PASS
RESULT policy_drift_named PASS
RESULT missing_file_exit_2 PASS

Other paths

The same query can be written in jq 1.7, with the policy values written into the program instead of read from a contract.

#!/bin/sh
# jq-report.sh: the same "rolled back for budget, how many restarts first" query in jq 1.7,
# for comparison. Policy values are written into the program, not read from a contract.
# On an apply file it checks the rollback target against earlier rollbacks but does not
# compare the ledger window or budget with the policy. jq exits 0 when the program ran.
set -u
if [ ! -f "$1" ]; then
  echo "jq-report.sh: cannot open $1" >&2
  exit 2
fi
jq -s -r '
  . as $all
  | [ .[] | select(.event == "rolled_back") ] as $rb
  | [ range(0; $rb | length) as $i
      | $rb[$i] as $r
      | select(($rb[:$i] | map(.from) | index($r.to)) != null)
      | "rollback of \($r.from) went to \($r.to), which was itself rolled back" ] as $v
  | "kind: apply",
    "events: \($all | length)",
    "rolled_back_for_budget: \(if ($rb | length) == 0 then "none" else ($rb | map(.from) | join(", ")) end)",
    "restarts_first: \(if ($rb | length) == 0 then "none" else ($rb | map(. as $r | "\($r.from)=\([$all[] | select(.event == "restarting" and .version == $r.from and .ts <= $r.ts)] | length)") | join(", ")) end)",
    "violations: \($v | length)",
    ($v[] | "  \(.)")
' "$1"

The shell wrapper handles the missing-file case itself and exits 2. The -s flag slurps every JSON line into one array, so the program sees the whole ledger at once, and -r prints strings without quotes. The program first selects the rolled_back events into $rb. It then builds the violation list by walking the indexes of $rb and keeping a rollback whose target index appears among the from values of the rollbacks before it. That is the same "target was itself rolled back" rule as the Nushell script. The two query lines count restarting events by version up to the rollback timestamp, and the final lines print the violations. The program has no report mode, and it never compares the ledger's window_ms or max with a policy.

On the forged ledger it finds the same lie, and then exits 0.

===== fixtures/forged.apply =====
kind: apply
events: 6
rolled_back_for_budget: v2, v3
restarts_first: v2=2, v3=2
violations: 1
  rollback of v3 went to v2, which was itself rolled back
exit: 0

The exit: 0 line comes from the run log, which records the status of the script after each fixture. jq exits 0 whenever the program ran, and it does not know that violations: 1 should count as a failure, so a wrapper has to read the output and decide the verdict. The Nushell script puts that decision in its own exit code.

The cost of the jq version is a second place to edit when the policy changes, since a changed budget has to be edited into the program and into the policy file. The jq version is the smaller program, and it needs no shell beyond jq itself.

Kubernetes expresses the same two knobs as separate settings. The mapping below is read-only prose and was not run. In the Deployments documentation, .spec.minReadySeconds sets how long a new Pod must be ready without any of its containers crashing before it counts as available, which is the probation window in spirit. The progressDeadlineSeconds setting surfaces a stall as a condition with type: Progressing, status: "False" and reason: ProgressDeadlineExceeded. The "Rolling Back a Deployment" section gives kubectl rollout undo deployment/nginx-deployment, with --to-revision to choose a target. The "Failed Deployment" section states that Kubernetes takes no action on a stalled Deployment other than reporting a status condition, so the undo is a command that a person or a controller above the Deployment has to issue. The Pod Lifecycle page adds that a Pod failing to start repeatedly may show CrashLoopBackOff in the Status field of some kubectl commands. That is a display field, not the Pod phase. A budget, a ledger of decisions and a validated policy are the parts the Kubernetes objects leave to the operator.

Citations

• Kubernetes documentation, "Deployments": sections "Min Ready Seconds", "Progress Deadline Seconds", "Rolling Back a Deployment" and "Failed Deployment".
• Kubernetes documentation, "Pod Lifecycle": the note on CrashLoopBackOff as a status display field.
• Beyer, Murphy, Rensin, Kawahara and Thorne (eds.), The Site Reliability Workbook, O'Reilly, 2018, chapter 16, "Canarying Releases": sections "Requirements of a Canary Process" and "Choosing a Canary Population and Duration" (the "Size and duration" consideration).
• Armstrong, Joe, Making reliable distributed systems in the presence of software errors, KTH, 2003, chapter 6: restart intensity and the {one_for_one,5,1000} supervisor example.