Hedronite · Curriculum · ML · Ops · Fri 2026-11-06

The operator's train policy A gate contract and a train-log report

Refuse a bad train policy before any Job is launched, read the train log without a key, and name a line whose event and detail disagree.

Lesson Class: Curriculum · O-ML · rung A
Tested with: Nickel 1.18.0 · Nushell 0.116.1 · jq 1.7.1
Lab: lab/ · sh check.sh (9 checks) · no network, no key
Source: Nickel manual: Contracts · Nushell command reference · jq 1.7 Manual · Kubernetes Jobs · DOP-C02 task 3.3

Lab download. Starter is the tree you edit. Worked answer is a separate file; open it after the starter passes. Nickel 1.18 and Nushell 0.116. The reference tree's jq-report.sh also needs jq 1.7. Fixture files named .log are the lab inputs, not operator logs. All lab requirements.

Refuse the Policy First
A backoff_limit of 7, a max_drop of 150, or a holdout with no hash is refused by the contract before any Job is launched.
Read the Train Log
The report answers whether the run resumed, how many drift failures it recorded, and what artifact exists.
Name the Contradiction
A sloppy edit that marks completed while the detail still refuses the artifact exits 1 and names line 7.
The contract refuses a bad policy. The report refuses a silent success.

O-ML · A · arc: self-healing-edge · langs: nickel, nushell · DOP-3.3, CKA-WL-4

By Evan Eugene Pincham.

Concept

A training Job is a run of a training program that is launched, may be killed, may be retried, and ends with a verdict. In Kubernetes, a Job is the object that runs a pod to completion and retries it when it fails. The companion lesson, "A training job that resumes from a checkpoint and fails honest drift, in Rust", built a toy version of that. Its tool is called trainjob, and it is CPU-only with a three-weight linear model. This lesson covers the operator's side of the same Job. The operator decides what a Job is allowed to be, before launch, and reads what the Job reported, after.

Two earlier ideas are needed, and each is defined here so the companion lesson is optional.

The **backoffLimit** is the Job's retry budget. The Kubernetes "Jobs" page says to "set .spec.backoffLimit to specify the number of retries before considering a Job as failed" and that it "is set by default to 6". The same page says failed pods are recreated "with an exponential back-off delay (10s, 20s, 40s ...) capped at ten minutes." The policy in this lesson caps the budget at 6, the Kubernetes default, so a policy file can never ask for more retries than an unconfigured Job would get.

The holdout set is a file of labeled samples the trainer never trained on. The drift check compares the trained model's score on the holdout set with a frozen baseline model's score on the same set. In trainjob the check passes when correct * 100 >= bcorrect * (100 - max_drop), using whole-number counts only. With a baseline of 20 of 20 and max_drop 20, a model scoring 14 of 20 gives 1400 against 1600 and fails. A model scoring 20 of 20 gives 2000 against 1600 and passes. The Google "Rules of Machine Learning" guide names the underlying problem as training-serving skew and says "The best solution is to explicitly monitor it so that system and data changes don't introduce skew unnoticed." The drift check is that monitor, run once at the end of training.

A drop of 30% means nothing unless you know which holdout set produced the counts. That is why the policy below requires holdout_sha256, the SHA-256 hash of the holdout file.

Two tools do the operator's work:

One kind of mistake gets a name. A contradiction is a log line whose event and detail disagree. For example, a line that says completed while its detail still says refusing artifact means someone changed the event and forgot the detail.

The teaching point is that a training Job is a deployable unit with the same honesty rules as a serving swap. It has a policy checked before launch. Its log is read after the run by something that exits non-zero when the run did not succeed. For this lesson, that is how the self-healing loop (kill, resume, retry, refuse) stays honest.

Worked example

The operator side has three files and three log fixtures. The fixtures hold the same events as the companion walkthrough. resumed.log is the run that was killed at step 4, resumed, and completed. drift_failed.log is the run that finished all 10 steps but scored 14 of 20 against a baseline of 20 of 20. edited.log is the drift log with the event on line 7 changed from drift_failed to completed:

{"ts":"2026-11-06T18:27:22Z","event":"completed","detail":"holdout: 14/20 vs baseline 20/20 (max drop 20%); refusing artifact"}

Step 1: the contract. train-policy.ncl defines one validator per field and one record contract that uses them. Each validator checks that the value is a number, that it is an integer, and that it is in range. Each returns a message that names the field.

# train-policy.ncl: the operator's rules for a training Job, as a contract.
# `nickel export` refuses a policy that breaks a rule, before any Job is launched.
let Steps = std.contract.from_validator (fun n =>
  if std.is_number n && std.number.is_integer n && n >= 1 then 'Ok
  else 'Error { message = "steps must be a whole number of at least 1" })
in
let CkptEvery = std.contract.from_validator (fun n =>
  if std.is_number n && std.number.is_integer n && n >= 1 then 'Ok
  else 'Error { message = "ckpt_every must be a whole number of at least 1" })
in
let Backoff = std.contract.from_validator (fun n =>
  if std.is_number n && std.number.is_integer n && n >= 0 && n <= 6 then 'Ok
  else 'Error { message = "backoff_limit must be a whole number from 0 to 6" })
in
let MaxDrop = std.contract.from_validator (fun n =>
  if std.is_number n && std.number.is_integer n && n >= 0 && n <= 100 then 'Ok
  else 'Error { message = "max_drop must be a whole number from 0 to 100" })
in
let Hex64 = std.contract.from_validator (fun s =>
  if std.is_string s && std.string.is_match "^[0-9a-f]{64}$" s then 'Ok
  else 'Error { message = "holdout_sha256 must be exactly 64 lowercase hex characters" })
in
{
  TrainPolicy = {
    steps | Steps,
    ckpt_every | CkptEvery,
    backoff_limit | Backoff,
    max_drop | MaxDrop,
    # Which holdout set the drift check was measured on. A drop % means nothing without it.
    holdout_sha256 | Hex64,
  },
}

Four things are worth noticing. The record is closed (no trailing ..), so a misspelled field is an error instead of being silently accepted. Every field is listed without a default, so every field is required, including holdout_sha256. The hash pattern is anchored with ^ and $ and lists only 0-9a-f, so the string must be exactly 64 lowercase hex characters. The behaviour of std.string.is_match here was observed in Nickel 1.18.0, the version the lab was run with.

Step 2: today's policy. policy.ncl imports the contract and applies it to a record with | TrainPolicy. The hash is the SHA-256 of the companion lesson's holdout.jsonl.

# policy.ncl: today's policy for the training Job. holdout_sha256 is the SHA-256 of holdout.jsonl.
let { TrainPolicy, .. } = import "train-policy.ncl" in
{
  steps = 10,
  ckpt_every = 2,
  backoff_limit = 2,
  max_drop = 20,
  holdout_sha256 = "af3101ce9a6f6e252137327e700181156b42f0cdd805777f579325ae0f6e858e",
} | TrainPolicy

nickel export policy.ncl prints this record as JSON. A copy of the policy with one field changed is refused with the validator's message. These lines come from the checker, which makes four such probes:

policy accepted: {"backoff_limit":2,"ckpt_every":2,"holdout_sha256":"af3101ce9a6f6e252137327e700181156b42f0cdd805777f579325ae0f6e858e","max_drop":20,"steps":10}
policy backoff_7: refused: backoff_limit must be a whole number from 0 to 6
policy maxdrop_150: refused: max_drop must be a whole number from 0 to 100
policy hash_63: refused: holdout_sha256 must be exactly 64 lowercase hex characters
policy no_hash: refused: error: missing definition for `holdout_sha256`

The hash_63 probe deletes the first character of the hash. The no_hash probe deletes the field. Different errors come from different layers. The first three come from a validator. The last comes from Nickel itself, because a required field has no definition.

The contract checks the shape of the hash, not its truth. Nothing in Nickel confirms that the hash matches the holdout file on disk. Computing the hash of the file with a SHA-256 tool and comparing it by hand is not run in this lesson.

Step 3: the report. train-report.nu reads the log line by line. A line that is not valid JSON becomes null and is counted as unreadable instead of crashing the script. The contradicts helper holds the sloppy-edit rule. A line contradicts itself when the event is completed but the detail contains refusing artifact. It also does when the event is drift_failed and the detail contains vs baseline but not refusing.

# train-report.nu: did it resume, how many drift failures, what artifact exists, from the train log.
# Usage: nu train-report.nu <log> [--policy <policy.json from nickel export>]
# It needs no key. Exit 0 when the last terminal event is completed and nothing looks wrong,
# 1 when last_event is drift_failed or job_failed, or a line contradicts itself,
# 2 when the log cannot be opened.

def parse-line [text: string] {
  try { $text | from json } catch { null }
}

# A line contradicts itself when its event and its detail disagree about the drift outcome.
def contradicts [r: record] {
  (($r.event == 'completed') and ($r.detail | str contains 'refusing artifact')) or (($r.event == 'drift_failed') and ($r.detail | str contains 'vs baseline') and (not ($r.detail | str contains 'refusing')))
}

def main [log: path, --policy: path] {
  if not ($log | path exists) {
    print -e $"train-report.nu: cannot open ($log)"
    exit 2
  }
  let raw = (open --raw $log | lines | where {|l| ($l | str trim) != '' })
  let rows = ($raw | enumerate | each {|p| { line: ($p.index + 1), rec: (parse-line $p.item) } })
  let unreadable = ($rows | where rec == null)
  let good = ($rows | where rec != null)
  let resumed = ($good | where {|r| $r.rec.event == 'resumed' })
  let drift = ($good | where {|r| $r.rec.event == 'drift_failed' })
  let art = ($good | where {|r| $r.rec.event == 'artifact_written' })
  let odd = ($good | where {|r| contradicts $r.rec })
  let last = (if ($good | is-empty) { null } else { ($good | last).rec })
  let last_event = (if $last == null { 'none' } else { $last.event })
  let artifact = (if ($art | is-empty) { 'none' } else {
    let d = ($art | last).rec.detail
    let path = (try { $d | parse -r 'path=(?<p>\S+)' | get 0.p } catch { 'unknown' })
    let sha = (try { $d | parse -r 'sha256=(?<s>[0-9a-f]{64})' | get 0.s } catch { '' })
    if ($sha | str length) == 64 { $"($path) sha256=($sha | str substring 0..12)..." } else { $path }
  })
  let attempts = (try {
    let starts = ($good | where {|r| $r.rec.event == 'job_started' or $r.rec.event == 'resumed' })
    if ($starts | is-empty) { 0 } else {
      let d = ($starts | first).rec.detail
      try { $d | parse -r 'attempt=(?<a>\d+)' | get 0.a | into int } catch { ($starts | length) }
    }
  } catch { 0 })
  # policy is accepted for future use (max_drop echo); unused for verdict today
  let _pol = (if $policy == null { null } else { open $policy })

  print $"train-report: ($log | path basename)"
  print $"lines: ($rows | length)"
  print $"resumed: ($resumed | length)"
  print $"drift_failed: ($drift | length)"
  print $"artifact: ($artifact)"
  print $"last_event: ($last_event)"
  print $"attempts: ($attempts)"
  print $"unreadable: ($unreadable | length)"
  print $"contradictions: ($odd | length)"
  for r in $odd { print $"  line ($r.line): event ($r.rec.event) but detail says '($r.rec.detail | split row ';' | first)'" }
  let bad_terminal = ($last_event == 'drift_failed') or ($last_event == 'job_failed')
  if ($odd | is-not-empty) or $bad_terminal or ($unreadable | is-not-empty) { exit 1 }
  if $last_event != 'completed' { exit 1 }
}

The exit codes are the contract between the report and whatever calls it. A missing log exits 2 before anything is printed. A log whose last event is drift_failed or job_failed, or that has a contradiction or an unreadable line, exits 1. A log that does not end in completed also exits 1. Everything else falls off the end of main and exits 0. The --policy flag is read but not yet used for the verdict, as the comment in the script says.

Step 4: run it on the three logs. On the resumed log, the report finds the one resumed event and the artifact with a short prefix of its hash:

train-report: resumed.log
lines: 10
resumed: 1
drift_failed: 0
artifact: artifacts/model-v1.json sha256=8d6c14555bf2f...
last_event: completed
attempts: 1
unreadable: 0
contradictions: 0

On the drift log, there is no artifact and no resume, and the exit code is 1:

train-report: drift_failed.log
lines: 7
resumed: 0
drift_failed: 1
artifact: none
last_event: drift_failed
attempts: 1
unreadable: 0
contradictions: 0

On the edited log, the last event says completed, so a reader who looks only at the last line would be satisfied. The report prints the disagreement and also exits 1:

train-report: edited.log
lines: 7
resumed: 0
drift_failed: 0
artifact: none
last_event: completed
attempts: 1
unreadable: 0
contradictions: 1
  line 7: event completed but detail says 'holdout: 14/20 vs baseline 20/20 (max drop 20%)'

The checker records the three exit codes on its own lines:

clean log: exit=0 resumed=1 drift_failed=0 last_event=completed artifact=artifacts/model-v1.json sha256=8d6c14555bf2f...
drift log: exit=1 last_event=drift_failed artifact=none drift_failed=1
edited log: exit=1 contradictions=1 last_event=completed
missing log: exit=2

Four pitfalls are worth knowing, each as a symptom and its cause.

Lab

Starter files. The lab directory has these files: check.sh, policy.ncl (the same file shown above), train-policy.ncl, train-report.nu, and a fixtures directory holding resumed.log, drift_failed.log, and edited.log. The two files to edit start like this.

train-policy.ncl is missing three rules. backoff_limit and max_drop are typed only as Number, there is no holdout_sha256, and the record is open:

# train-policy.ncl (starter): the operator's rules for a training Job, as a contract.
# TODO: backoff_limit must be a whole number from 0 to 6.
# TODO: max_drop must be a whole number from 0 to 100.
# TODO: there is no holdout_sha256. Require it, as exactly 64 lowercase hex characters.
let Steps = std.contract.from_validator (fun n =>
  if std.is_number n && std.number.is_integer n && n >= 1 then 'Ok
  else 'Error { message = "steps must be a whole number of at least 1" })
in
{
  TrainPolicy = {
    steps | Steps,
    ckpt_every | Number,
    backoff_limit | Number,
    max_drop | Number,
    ..
  },
}

train-report.nu hard-codes resumed: 0, artifact: none and attempts: 1, has no contradiction check, and exits 0 even on drift_failed:

# train-report.nu (starter): read the train log.
# Usage: nu train-report.nu <log> [--policy <policy.json from nickel export>]
# TODO: count resumed events (do not ignore them).
# TODO: report the artifact from artifact_written, not "none".
# TODO: name lines whose event and detail disagree; exit 1 on drift_failed / job_failed / contradictions.
def main [log: path, --policy: path] {
  if not ($log | path exists) { exit 2 }
  let recs = (open --raw $log | lines | where {|l| ($l | str trim) != '' } | each {|l| $l | from json })
  print $"train-report: ($log | path basename)"
  print $"lines: ($recs | length)"
  print $"resumed: 0"
  print $"drift_failed: ($recs | where event == 'drift_failed' | length)"
  print $"artifact: none"
  print $"last_event: ($recs | last | get event)"
  print $"attempts: 1"
}

Task.

  1. In train-policy.ncl, make backoff_limit accept only whole numbers from 0 to 6, and max_drop only whole numbers from 0 to 100. Add a required holdout_sha256 that must be exactly 64 lowercase hex characters. Each rule needs a message that names the field. Closing the record (dropping the trailing ..) matches the reference, but no check tests it. Leave policy.ncl as it is; the checker edits it with sed, so reformatting it will break the probes.
  2. In train-report.nu, count resumed events. Read the artifact_written line and print its path= value, followed by a short prefix of its sha256= value and .... Print unreadable and contradictions lines. For each contradicting line, print line N: event E but detail says '...' with the part of the detail before the first ;. Exit 1 when the last event is drift_failed or job_failed, when any contradiction exists, or when any line is not valid JSON. Keep exit 2 for a missing log.
  3. Do not edit check.sh, policy.ncl, or the fixtures.

Check. Run this beside the files. It needs Nickel 1.18 and Nushell 0.116, no network, and no key. The fixtures were produced by running the companion trainjob walkthrough (killed at step 4 then resumed; force-weights drift failure) and committing the JSONL logs:

$ sh check.sh

The starter fails 7 of the 9 checks. It passes policy_accepted and missing_log_exit_2. It fails rejects_backoff_limit_7, rejects_max_drop_150, rejects_63_char_hash, rejects_missing_hash, report_clean_resumed, report_drift_failed_named, and sloppy_edit_named. This is the starter output:

policy accepted: {"backoff_limit":2,"ckpt_every":2,"holdout_sha256":"af3101ce9a6f6e252137327e700181156b42f0cdd805777f579325ae0f6e858e","max_drop":20,"steps":10}
RESULT policy_accepted PASS
policy backoff_7: accepted
RESULT rejects_backoff_limit_7 FAIL
policy maxdrop_150: accepted
RESULT rejects_max_drop_150 FAIL
policy hash_63: accepted
RESULT rejects_63_char_hash FAIL
policy no_hash: accepted
RESULT rejects_missing_hash FAIL
clean log: exit=0 resumed=0 drift_failed=0 last_event=completed artifact=none
RESULT report_clean_resumed FAIL
drift log: exit=0 last_event=drift_failed artifact=none drift_failed=1
RESULT report_drift_failed_named FAIL
edited log: exit=0 contradictions= last_event=completed
RESULT sloppy_edit_named FAIL
missing log: exit=2
RESULT missing_log_exit_2 PASS

Expected on a finished lab. Every check ends with PASS. The nine names, in order, are policy_accepted, rejects_backoff_limit_7, rejects_max_drop_150, rejects_63_char_hash, rejects_missing_hash, report_clean_resumed, report_drift_failed_named, sloppy_edit_named, and missing_log_exit_2. The checker also prints the three reports after those lines. This is the full reference output:

policy accepted: {"backoff_limit":2,"ckpt_every":2,"holdout_sha256":"af3101ce9a6f6e252137327e700181156b42f0cdd805777f579325ae0f6e858e","max_drop":20,"steps":10}
RESULT policy_accepted PASS
policy backoff_7: refused: backoff_limit must be a whole number from 0 to 6
RESULT rejects_backoff_limit_7 PASS
policy maxdrop_150: refused: max_drop must be a whole number from 0 to 100
RESULT rejects_max_drop_150 PASS
policy hash_63: refused: holdout_sha256 must be exactly 64 lowercase hex characters
RESULT rejects_63_char_hash PASS
policy no_hash: refused: error: missing definition for `holdout_sha256`
RESULT rejects_missing_hash PASS
clean log: exit=0 resumed=1 drift_failed=0 last_event=completed artifact=artifacts/model-v1.json sha256=8d6c14555bf2f...
RESULT report_clean_resumed PASS
drift log: exit=1 last_event=drift_failed artifact=none drift_failed=1
RESULT report_drift_failed_named PASS
edited log: exit=1 contradictions=1 last_event=completed
RESULT sloppy_edit_named PASS
missing log: exit=2
RESULT missing_log_exit_2 PASS
--- clean.out
train-report: resumed.log
lines: 10
resumed: 1
drift_failed: 0
artifact: artifacts/model-v1.json sha256=8d6c14555bf2f...
last_event: completed
attempts: 1
unreadable: 0
contradictions: 0
--- drift.out
train-report: drift_failed.log
lines: 7
resumed: 0
drift_failed: 1
artifact: none
last_event: drift_failed
attempts: 1
unreadable: 0
contradictions: 0
--- edited.out
train-report: edited.log
lines: 7
resumed: 0
drift_failed: 0
artifact: none
last_event: completed
attempts: 1
unreadable: 0
contradictions: 1
  line 7: event completed but detail says 'holdout: 14/20 vs baseline 20/20 (max drop 20%)'

Other paths

jq 1.7. The same counts can be taken without Nushell. jq-report.sh runs jq with -s and -r. The jq manual describes --slurp/-s as reading "the entire input stream into a large array" and running the filter once, which is what lets one filter count events across all lines. --raw-output/-r prints strings without quotes. The script computes the resumed count, the drift failure count, the artifact path, the last event, and the same contradiction rule. It was run with jq 1.7.1 on the three fixtures in order (resumed, drift, edited):

resumed: 1
drift_failed: 0
artifact: artifacts/model-v1.json
last_event: completed
contradictions: 0
resumed: 0
drift_failed: 1
artifact: none
last_event: drift_failed
contradictions: 0
resumed: 0
drift_failed: 0
artifact: none
last_event: completed
contradictions: 1

The counts match the Nushell report, and the sloppy edit is caught on the third log. The difference is the exit code. The jq manual says that "Normally jq exits with 2 if there was any usage problem or system error, 3 if there was a jq program compile error, or 0 if the jq program ran." The program ran on the drift log, so jq exits 0 there. A script that wants the same verdict as the Nushell report must read the output and choose its own exit code. Writing that wrapper is not run here. The jq version also does not count unreadable lines. With -s, jq stops at the first line that is not valid JSON, so a damaged log fails the whole program instead of being counted.

Not run. A Kubernetes Job manifest built from policy.ncl, with backoffLimit: 2 and a PVC mount, is not run, because no cluster is involved. A signed log, where a forged but consistent edit would also be detected, is also not run. The report is key-less.

Citations