Hedronite · Curriculum · AI · Ops · Thu 2026-11-05

The operator's swap policy A gate contract and a what-is-serving report

Refuse a bad model-swap policy before any model is judged, read the deploy log without the key, and say plainly what a key-less report cannot see.

Lesson Class: Curriculum · O-AI · rung A
Tested with: Nickel 1.18.0 · Nushell 0.116.1 · jq 1.7.1
Lab: lab/ · sh check.sh (11 checks) · no network, no key
Source: Nickel manual: Contracts · Nushell command reference · jq 1.7 Manual · Kubernetes Deployments · OWASP LLM03:2025 · DOP-C02 task 5.3
Refuse the Policy First
A gate of 0 % or 150 %, or a gate with no eval-set hash, is refused by the contract before any model is judged.
Read Without the Key
The report names what is serving, why, and how many swaps were rejected, and flags a line whose event and detail disagree.
Say What You Cannot See
A careful forgery passes the report with exit 0. Only the chain check, which holds the key, names line 4.
The report explains the log. The chain check decides whether to believe it.

O-AI · A · arc: self-healing-edge · langs: nickel, nushell · DOP-5.3, CKA-WL-4

By Evan Eugene Pincham.

Concept

This lesson covers the operator's side of a model swap. It assumes only that a deploy tool exists, so a short recap of the companion lesson, "A model swap that rolls back when it fails its eval, in Rust", comes first.

That companion lesson builds modelgate, a small program that treats a new model version as a deploy. It judges the new version, called the candidate, on an eval set: a fixed list of labeled samples the model has to classify. The eval gate is the check that runs before the candidate serves. It is a threshold on counts, correct / total, so it acts as the readiness probe for a model. A candidate under the threshold is rolled back. The last-good version, the most recent one that passed the gate, keeps serving.

Every decision is appended to a deploy log as one JSON object per line, with the fields ts, event, version and detail. The events are candidate_started, promoted and rolled_back. The log is a sealed log. Each line also carries prev and mac, so the lines form a chain. A MAC (message authentication code) is a short value computed from a line and a secret key. Without the key, an edited line cannot be given a matching MAC. modelgate status holds the key, re-computes the chain and names the first line that does not belong.

Everything in these lessons is a toy. The model is a three-weight linear classifier, the eval set is 20 generated samples, and there is no real language model. The swap, gate, rollback and sealed-log logic is real.

Accuracy is a coarse number. The cited Machine Learning Crash Course page defines it as "the proportion of all classifications that were correct, whether positive or negative", and says other metrics suit imbalanced data. A 20-sample accuracy gate says nothing about a rare class, latency or a prompt-injection regression. That limit is why this lesson keeps asking which eval set a number came from.

The operator's side has three jobs.

1. Refuse a bad policy before any model is judged. A contract in Nickel is a rule attached to a value, and Nickel checks the rule when it evaluates the configuration. The Nickel user manual says record contracts "encode the data schema of a configuration". It also describes user-defined contracts built with std.contract.from_validator, whose function returns 'Ok or 'Error { message = ... }. The swap policy has four fields.

The contract checks the shape of the hash. It does not compare the hash with any file on disk.

2. Read the deploy log without the key. A Nushell script can answer "what is serving, and why" and "how many swaps were rejected" from the log alone. Nushell turns text into structured data. open --raw opens a file as raw text, and from json parses JSON text into structured data. The script also names any line that contradicts itself and checks the day's candidate count against the policy. A contradiction here is a line whose event and detail disagree about the gate. An example is event promoted with a detail that begins gate fail.

3. Admit what the report cannot see. The report has no key, so it never checks a MAC. A careful forger who rewrites both the event and the detail of one line leaves no contradiction behind, and the report passes with exit 0. Only modelgate status catches it. The report therefore prints chain: not checked: run modelgate status on every run. The key-less report is a reading aid and the chain check is the verdict, so run both and read both exit codes.

Two limits of the lab itself:

Worked example

The finished files are below. The lab asks for them to be rebuilt from a starter, so try the lab from the check descriptions before reading these as an answer key.

The contract

swap-policy.ncl defines one validator per rule and a record contract, SwapPolicy, that applies them. The record contract has no trailing .., so it is closed. The manual notes that a trailing .. is what makes a record contract open.

# swap-policy.ncl: the operator's rules for swapping a model, as a contract.
# `nickel export` refuses a policy that breaks a rule, before any model is judged.
let Pct = std.contract.from_validator (fun n =>
  if std.is_number n && std.number.is_integer n && n >= 1 && n <= 100 then 'Ok
  else 'Error { message = "min_accuracy_pct must be a whole number from 1 to 100" })
in
let OnFail = std.contract.from_validator (fun s =>
  if s == "rollback" || s == "hold" then 'Ok
  else 'Error { message = "on_fail must be \"rollback\" or \"hold\"" })
in
let PerDay = std.contract.from_validator (fun n =>
  if std.is_number n && std.number.is_integer n && n >= 1 then 'Ok
  else 'Error { message = "max_candidates_per_day must be a whole number of at least 1" })
in
let Hex64 = std.contract.from_validator (fun s =>
  if std.is_string s && std.string.is_match "^[0-9a-f]{64}$" s then 'Ok
  else 'Error { message = "eval_set_sha256 must be exactly 64 lowercase hex characters" })
in
{
  SwapPolicy = {
    min_accuracy_pct | Pct,
    on_fail | OnFail,
    max_candidates_per_day | PerDay,
    # Which eval set the gate was measured on. A gate number means nothing without it.
    eval_set_sha256 | Hex64,
  },
}

Each number and hash validator checks the type before the rule, so a string given as min_accuracy_pct is refused by the same check as an out-of-range number. The behaviour of std.string.is_match is cited from the lab run on Nickel 1.18.0, not from a documentation page. The lab checker never tries a fractional number, so the is_integer clauses are written but not exercised by it.

policy.ncl is today's policy. It imports the contract and applies it with | SwapPolicy.

# policy.ncl: today's policy for the model service. eval_set_sha256 is the SHA-256 of models/eval.json.
let { SwapPolicy, .. } = import "swap-policy.ncl" in
{
  min_accuracy_pct = 90,
  on_fail = "rollback",
  max_candidates_per_day = 3,
  eval_set_sha256 = "ac723805ee0360093c7e7d6f423fd28cd6bf53a9bec34e13eb905989d83cc9dd",
} | SwapPolicy

nickel export policy.ncl prints the policy as JSON when every rule holds, and stops with a message when one does not. The checker exports the policy and then makes four single-change copies of it. The following lines come from the checker's output, in order.

policy accepted: {"eval_set_sha256":"ac723805ee0360093c7e7d6f423fd28cd6bf53a9bec34e13eb905989d83cc9dd","max_candidates_per_day":3,"min_accuracy_pct":90,"on_fail":"rollback"}
RESULT policy_accepted PASS
policy min_0: refused: min_accuracy_pct must be a whole number from 1 to 100
RESULT rejects_min_accuracy_0 PASS
policy min_150: refused: min_accuracy_pct must be a whole number from 1 to 100
RESULT rejects_min_accuracy_150 PASS
policy hash_63: refused: eval_set_sha256 must be exactly 64 lowercase hex characters
RESULT rejects_63_char_hash PASS
policy no_hash: refused: error: missing definition for `eval_set_sha256`
RESULT rejects_missing_hash PASS

The first line is the exported JSON that the report later reads as policy.json. The two messages that start with a field name are the validator messages from the contract. The last message is Nickel's own, because a field with a contract and no value is a missing definition.

The report

swap-report.nu reads the log line by line.

# swap-report.nu: what is serving, why, and how many swaps were rejected, read from the deploy log.
# Usage: nu swap-report.nu <log> [--policy <policy.json from nickel export>]
# It needs no key, so it never checks a mac. It prints that on every run.
# Exit 0 when nothing looks wrong, 1 when a line contradicts itself or the day's budget is exceeded,
# 2 when the log cannot be opened.

def parse-line [text: string] {
  try { $text | from json } catch { null }
}

# A line contradicts itself when its event and its detail disagree about the gate.
def contradicts [r: record] {
  (($r.event == 'promoted') and ($r.detail | str starts-with 'gate fail')) or (($r.event == 'rolled_back') and ($r.detail | str starts-with 'gate pass'))
}

def main [log: path, --policy: path] {
  if not ($log | path exists) {
    print -e $"swap-report.nu: cannot open ($log)"
    exit 2
  }
  let raw = (open --raw $log | lines | where {|l| ($l | str trim) != '' })
  let rows = ($raw | enumerate | each {|p| { line: ($p.index + 1), rec: (parse-line $p.item) } })
  let unreadable = ($rows | where rec == null)
  let good = ($rows | where rec != null)
  let promoted = ($good | where {|r| $r.rec.event == 'promoted' })
  let rolled = ($good | where {|r| $r.rec.event == 'rolled_back' })
  let started = ($good | where {|r| $r.rec.event == 'candidate_started' })
  let odd = ($good | where {|r| contradicts $r.rec })
  let serving = (if ($promoted | is-empty) { 'none' } else { ($promoted | last).rec.version })
  let why = (if ($promoted | is-empty) { 'no version was ever promoted' } else { ($promoted | last).rec.detail | str replace --regex 'sha256=([0-9a-f]{12})[0-9a-f]*' 'sha256=$1...' })
  let budget = (if $policy == null { null } else { open $policy | get max_candidates_per_day })
  let over = (if $budget == null { false } else { ($started | length) > $budget })

  print $"swap-report: ($log | path basename)"
  print $"lines: ($rows | length)"
  print $"serving: ($serving)"
  print $"why: ($why)"
  print $"candidates: ($started | length)"
  print $"promoted: ($promoted | length)"
  print $"rolled_back: ($rolled | length)"
  print $"unreadable: ($unreadable | length)"
  print $"contradictions: ($odd | length)"
  for r in $odd { print $"  line ($r.line): event ($r.rec.event) but detail says '($r.rec.detail | split row ';' | first)'" }
  if $budget != null { print $"budget: ($started | length) of ($budget) candidates(if $over { ' EXCEEDED' } else { '' })" }
  print 'chain: not checked: run modelgate status'
  if ($odd | is-not-empty) or $over or ($unreadable | is-not-empty) { exit 1 }
}

How it works:

What the report says about a clean log

The checker runs the report on the six-line clean log with the exported policy. The log has three candidates: v1 promoted, v2 rolled back, and v3 promoted over v1.

clean log: exit=0 serving=v3 promoted=2 rolled_back=1 candidates=3
RESULT report_clean_log PASS
clean log why: gate pass: 19/20 correct (min 90%); sha256=4880d855d5fb...; replaced v1
RESULT report_says_why PASS

The why line says that v3 serves because it got 19 of 20 samples right against a 90% gate, and that it replaced v1.

A sloppy edit

Suppose someone changes the event on line 4 from rolled_back to promoted and leaves the detail alone. The report shows this.

swap-report: edited.log
lines: 6
serving: v3
why: gate pass: 19/20 correct (min 90%); sha256=4880d855d5fb...; replaced v1
candidates: 3
promoted: 3
rolled_back: 0
unreadable: 0
contradictions: 1
  line 4: event promoted but detail says 'gate fail: 12/20 correct (min 90%)'
chain: not checked: run modelgate status

The edit turned a rejected swap into an apparent promotion, so rolled_back fell from 1 to 0. The report still names line 4 because its event and its detail disagree. The checker records edited log: exit=1 rolled_back=0 contradictions=1.

A careful forgery

A forger who also rewrites the detail to a gate pass string leaves nothing for the report to compare. The checker's line for that log is below.

forged log: exit=0 promoted=3 rolled_back=0 contradictions=0 chain=not checked: run modelgate status

The exit code is 0 and the counts are promoted 3, rolled_back 0, contradictions 0. Every number the report prints looks like a healthy log. The only warning is the final line, which says the chain was not checked. Running modelgate status on the same file gives the verdict.

$ modelgate status forged.log
lines: 6
verified: 5
broken: 1
unreadable: 0
first_break: line 4 (mac)
serving: v3
exit=1

status holds the key, so it sees that line 4's MAC does not match its content. It exits 1 and names the line. Note that serving: v3 is still printed. The forged line is simply not counted as a promotion, and the version happens to be the same. The exit code is the verdict, so read the exit code first.

The report is cheap and needs no secret. For that reason it can run anywhere, and it cannot tell a true log from a well-made false one. The chain check is the opposite. Used together, the report explains the log in operator terms, and the chain check decides whether to believe it.

Lab

The lab rebuilds the contract and the report from a starter. It is simulated: three committed log files stand in for a running deployer, and no model, key or network is involved.

Starter files

All of these sit in one directory beside check.sh:

Task

  1. In swap-policy.ncl, replace Number with a contract that accepts only whole numbers from 1 to 100. Add the other three rules (on_fail, max_candidates_per_day and a required eval_set_sha256 of exactly 64 lowercase hex characters). Closing the SwapPolicy record (dropping the trailing ..) matches the reference, but no check tests it. Leave policy.ncl as it is. The checker edits it with sed, so reformatting it will break the probes.
  2. In swap-report.nu, take serving from the last promoted line. Print why from that line's detail. Count contradictions and print each one as line N: event E but detail says '...'. Accept --policy and print a budget: line, exiting 1 when the candidate count exceeds the budget. Print chain: not checked: run modelgate status on every run. Exit 2 when the log cannot be opened.

Checker

Run sh check.sh in that directory. It needs Nickel 1.18 and Nushell 0.116, no network and no key. It writes policy.json, probe-*.ncl and *.out files beside the others, and prints one result line per check, giving the check name and then PASS or FAIL. It exits 0 only if all 11 pass. The outputs quoted in this lesson come from that checker run on 2026-11-05 in a clean container with Nickel 1.18 and Nushell 0.116 and networking disabled.

What the starter does

The starter fails 8 of the 11 checks and passes 3. The three that pass are policy_accepted, report_clean_log and missing_log_exit_2.

Failing checkWhat the starter lacks
rejects_min_accuracy_0Any number is accepted for min_accuracy_pct.
rejects_min_accuracy_150Same cause.
rejects_63_char_hashThere is no eval_set_sha256 field to check.
rejects_missing_hashThe field is not required, and the record is open.
report_says_whyThe report prints no why: line.
sloppy_edit_namedThe report finds no contradictions.
forgery_invisible_and_says_soThe report prints no chain: line.
budget_exceeded_flaggedThe report has no budget check.

report_clean_log passes on the starter because its last line happens to be the promoted v3 line. That passing check does not show that the starter reads serving correctly. On a log that ends with a rolled-back candidate, taking the version from the last line would name a candidate that was rejected. That case is not run here.

Expected result

A finished lab prints this.

policy accepted: {"eval_set_sha256":"ac723805ee0360093c7e7d6f423fd28cd6bf53a9bec34e13eb905989d83cc9dd","max_candidates_per_day":3,"min_accuracy_pct":90,"on_fail":"rollback"}
RESULT policy_accepted PASS
policy min_0: refused: min_accuracy_pct must be a whole number from 1 to 100
RESULT rejects_min_accuracy_0 PASS
policy min_150: refused: min_accuracy_pct must be a whole number from 1 to 100
RESULT rejects_min_accuracy_150 PASS
policy hash_63: refused: eval_set_sha256 must be exactly 64 lowercase hex characters
RESULT rejects_63_char_hash PASS
policy no_hash: refused: error: missing definition for `eval_set_sha256`
RESULT rejects_missing_hash PASS
clean log: exit=0 serving=v3 promoted=2 rolled_back=1 candidates=3
RESULT report_clean_log PASS
clean log why: gate pass: 19/20 correct (min 90%); sha256=4880d855d5fb...; replaced v1
RESULT report_says_why PASS
edited log: exit=1 rolled_back=0 contradictions=1
RESULT sloppy_edit_named PASS
forged log: exit=0 promoted=3 rolled_back=0 contradictions=0 chain=not checked: run modelgate status
RESULT forgery_invisible_and_says_so PASS
budget 2: exit=1 budget=3 of 2 candidates EXCEEDED
RESULT budget_exceeded_flagged PASS
missing log: exit=2
RESULT missing_log_exit_2 PASS
--- edited.out
swap-report: edited.log
lines: 6
serving: v3
why: gate pass: 19/20 correct (min 90%); sha256=4880d855d5fb...; replaced v1
candidates: 3
promoted: 3
rolled_back: 0
unreadable: 0
contradictions: 1
  line 4: event promoted but detail says 'gate fail: 12/20 correct (min 90%)'
chain: not checked: run modelgate status

The budget 2 check lowers max_candidates_per_day to 2 in a copy of the policy, exports it and runs the report on the three-candidate clean log. Exit 1 and 3 of 2 candidates EXCEEDED is the expected answer.

Pitfalls

Other paths

Two other ways to state the same operator check follow. jq was run on the same three logs. The Kubernetes mapping is read-only and was not run.

jq 1.7

The same counts can be read with jq. This script is shipped with the lab.

#!/bin/sh
# jq-report.sh: the same "what is serving" answer in jq 1.7, for comparison. No key, no chain check.
jq -s -r '
  (map(select(.event == "promoted")) | last) as $p
  | "serving: \($p.version // "none")",
    "promoted: \(map(select(.event == "promoted")) | length)",
    "rolled_back: \(map(select(.event == "rolled_back")) | length)",
    "contradictions: \(map(select((.event == "promoted" and (.detail | startswith("gate fail"))) or (.event == "rolled_back" and (.detail | startswith("gate pass"))))) | length)",
    "chain: not checked: run modelgate status"' "$1"

Per the jq 1.7 manual, --slurp (-s) reads the entire input stream into one array and runs the filter once, and --raw-output (-r) writes strings without quotes. These runs were captured on the three fixtures.

$ sh jq-report.sh fixtures/deploy.log
serving: v3
promoted: 2
rolled_back: 1
contradictions: 0
chain: not checked: run modelgate status
exit=0
$ sh jq-report.sh fixtures/edited.log
serving: v3
promoted: 3
rolled_back: 0
contradictions: 1
chain: not checked: run modelgate status
exit=0
$ sh jq-report.sh fixtures/forged.log
serving: v3
promoted: 3
rolled_back: 0
contradictions: 0
chain: not checked: run modelgate status
exit=0

The counts match the Nushell report on all three logs, and jq also finds the contradiction in the sloppy edit. The difference is the exit code. The edited log gives exit=0 here, but the Nushell report exits 1 on it. The manual says jq "exits with 2 if there was any usage problem or system error, 3 if there was a jq program compile error, or 0 if the jq program ran." Under that rule, finding a contradiction is just output. A script that wants a verdict has to read the counts and exit non-zero itself. The jq version has no budget check, no why line and no handling of an unreadable line. What -s does with a line that is not valid JSON was not run.

Kubernetes progressDeadlineSeconds (read-only mapping, not run)

A Kubernetes Deployment accepts .spec.progressDeadlineSeconds, a limit on how long a rollout may make no progress. Past it, Kubernetes reports reason: ProgressDeadlineExceeded. The Deployments documentation, section "Failed Deployment", states that "Kubernetes takes no action on a stalled Deployment other than to report a status condition with reason: ProgressDeadlineExceeded. Higher level orchestrators can take advantage of it and act accordingly, for example, rollback the Deployment to its previous version."

This is the closest analogue to a deadline policy, and the mapping is partial. The operator declares a limit in configuration, and something reports when it is crossed. In this lesson the limit is an accuracy threshold and a daily count, not a time. The swap policy declares the limits, and the report reads them. modelgate deploy is the "higher level" actor that rolls back, as the companion lesson describes. The mapping does not cover how the two systems detect failure. Kubernetes measures rollout progress, and the gate measures accuracy on 20 samples. None of this was run against a cluster.

For the readiness half, the probes documentation says that "a pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services." The eval gate plays that role for a model before it serves.

Citations