A training-run page that never turns green on drift_failed or job_failed A static export in Bun and TypeScript

Four logs in, four pages out: resumed-and-completed goes green; drift_failed, job_failed, and a contradicted completed never do.

Lab download. Starter is the tree you edit. Worked answer is a separate file; open it after the starter passes. Bun 1.3 or newer. package.json has no dependencies to install. All lab requirements.

Honesty line. The training job behind this lesson is a toy: a three-weight linear classifier on 40 made-up samples, CPU only, no GPU, no framework download. What is real in this lesson is the web page's logic: parsing a log, cross-checking it against a second file, and refusing to show green unless every check holds. What is simulated is the job itself and the "durable storage" it writes to (a plain directory standing in for a Kubernetes PersistentVolumeClaim). Nothing here runs on a real cluster, and the lab never ran the trainer: the page is built from four small log fixtures written by the lab's check script.

Frontend rule, stated plainly. This is a content page, produced by a Bun static export. It has no accounts, no data store, no server and no client-side JavaScript. Bun is used only as a build tool and test runner, so there is no Bun backend. TypeScript clients generated from Rust schemas do not apply today: there is no API, only two text files on disk.

1. Concept

A training run finishes in more than one way. It can complete and write an artifact, fail a holdout drift check and refuse the artifact, or burn through its retry budget and emit job_failed. This section defines a static page that reads the run's log and status report together, cross-checks every number, and turns green only when the last terminal event is completed with no contradictions. Red is the default; green is earned.

The problem: a status page that says "done" when the job did not succeed

Suppose a small program trains a model. It saves progress as it goes, may be killed and restarted, is allowed only a few retries, and at the end either writes a model file or refuses to. Somebody builds a dashboard page for it. What is the single most damaging thing that page can do? Show a green "done" for a run that failed.

That is not far-fetched. A bad training run still finishes its steps. A job that exhausted its retries still ends. If the page treats "the process ended" as "the process succeeded", it lies. So this lesson builds a page whose rule is the reverse: green is the exception, and it has to be earned by several independent facts agreeing.

What the training job writes (so you don't need to have met it)

The job is a command-line program called trainjob. You don't need it to follow this lesson, only its two outputs.

Output 1: a log. One JSON object per line, each with a timestamp ts, an event and a free-text detail. The events that matter here:

Three of these are terminal: completed, drift_failed, job_failed. They say how a run ended. The rest are progress notes.

Output 2: a status report. The command trainjob status LOG prints five lines summarising a log:

lines: 10
last_event: completed
attempts: 1
artifact: artifacts/model-v1.json
drift: pass

Three ideas the page borrows

Durable checkpoints. Resuming after a kill only works if progress was saved somewhere that outlives the process. The PyTorch tutorial on saving and loading models says that for a general checkpoint, "you must save more than just the model's state_dict" and that "It is important to also save the optimizer's state_dict, as this contains buffers and parameters that are updated as the model trains." The toy job saves the same idea in miniature: step number, weights, bias and loss, in a JSON file. In Kubernetes terms the "somewhere" is a volume. The Persistent Volumes documentation defines a PersistentVolume as "a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes", and a PersistentVolumeClaim as "a request for storage by a user." In the lab, that storage is just a directory.

A retry budget. Kubernetes Jobs have a limit on how many times failed pods are retried. The Jobs documentation says to "set .spec.backoffLimit to specify the number of retries before considering a Job as failed. The .spec.backoffLimit is set by default to 6". The toy job uses a small number, so a failing run reaches job_failed quickly. A job that has run out of retries has not "completed" by any reading, and the page must say so.

A check that can fail. After training, the job compares the new model against a frozen baseline on held-out data, using whole-number counts (for example 14 correct out of 20, against 20 out of 20). If the new model falls too far behind, the job writes drift_failed and no model file. Exact floating-point equality is never used. This is a holdout comparison, not the training-serving skew that Google's Rules of Machine Learning describes, but the guidance points the same way: skew should be monitored "explicitly" so that changes "don't introduce skew unnoticed." The page's job is the last step of making a failure noticeable: showing it.

The design decision: two files, cross-checked

The page reads two files per run: the log, and the text trainjob status printed for that log.

It then computes its own numbers from the log and compares them to what the status report claims:

• resumes = how many resumed events there are;
• last terminal event = the last line whose event is completed, drift_failed or job_failed (or, if there is none, simply the last event);
• artifact path and short hash = taken from the last artifact_written line (the short hash is the first 12 hex characters of the recorded SHA-256);
• attempts = the attempt= value on job_started/resumed lines, or the attempts= value on job_failed;
• holdout counts = parsed from the completed/drift_failed detail when present.

A disagreement is a build error. The build throws. It does not render a warning. The reason: a page that warns still ships, and someone will glance at it and see the colour. A build that fails publishes nothing new. The five checks that throw:

1. status lines differs from the log's line count;
2. status last_event differs from the log's last terminal event;
3. status attempts differs from the attempt count parsed from the log;
4. status artifact differs from the log's last artifact_written path (none matches "no such line");
5. status drift differs from what the terminal event implies (completed → pass; drift_failed or job_failed → fail).

A contradiction is listed and blocks green. Some problems are not disagreements between two files but a log that disagrees with itself. Two are detected: an event completed whose detail says "refusing artifact", and an event drift_failed whose detail claims success without refusing. Such lines are shown on the page and the banner stays red.

Green is conjunctive. The banner is green only when the last terminal event is completed and there are no contradictions (and the build got this far, which means the status report agreed). Everything else is red, including terminal events the page doesn't recognise.

2. Worked example

The fixtures come in pairs: <name>.log and <name>.status.

Case 1: killed, resumed, completed (should be green)

{"ts":"2026-11-06T18:27:22Z","event":"job_started","detail":"steps=10 attempt=1"}
{"ts":"2026-11-06T18:27:22Z","event":"checkpointed","detail":"step=2 loss_milli=310"}
{"ts":"2026-11-06T18:27:22Z","event":"checkpointed","detail":"step=4 loss_milli=241"}
{"ts":"2026-11-06T18:27:22Z","event":"killed","detail":"at_step=4"}
{"ts":"2026-11-06T18:27:22Z","event":"resumed","detail":"from_step=4"}
{"ts":"2026-11-06T18:27:22Z","event":"checkpointed","detail":"step=6 loss_milli=228"}
{"ts":"2026-11-06T18:27:22Z","event":"checkpointed","detail":"step=8 loss_milli=226"}
{"ts":"2026-11-06T18:27:22Z","event":"checkpointed","detail":"step=10 loss_milli=225"}
{"ts":"2026-11-06T18:27:22Z","event":"artifact_written","detail":"sha256=8d6c14555bf2fabeb363b86bc8c185c13cd0abe77dadc1b10e4e6f29b4f164d7 path=artifacts/model-v1.json"}
{"ts":"2026-11-06T18:27:22Z","event":"completed","detail":"holdout: 20/20 vs baseline 20/20"}

with status lines: 10 / last_event: completed / attempts: 1 / artifact: artifacts/model-v1.json / drift: pass.

Tracing the checks:

• 10 log lines, status says 10. ✔
• Attempts: job_started says attempt=1; the resumed line carries no attempt=, so the value stays 1 (a resume is the same attempt continuing). Status says 1. ✔
• Resumes: one resumed line, so 1.
• Artifact: the path is artifacts/model-v1.json, the recorded SHA-256 starts 8d6c14555bf2, so the short hash is 8d6c14555bf2. Status path matches. ✔
• Last terminal: completed. Status says completed; it expects drift pass and status says pass. ✔
• Contradictions: none. Holdout: 20/20 vs baseline 20/20.

Verdict: green. The page carries class="banner ok", data-banner="ok" and data-verdict="completed".

Case 2: the drift check failed (must be red)

The log has the same checkpoint progress and ends with one line:

{"ts":"2026-11-06T18:27:22Z","event":"drift_failed","detail":"holdout: 14/20 vs baseline 20/20 (max drop 20%); refusing artifact"}

Status: lines: 7 / last_event: drift_failed / attempts: 1 / artifact: none / drift: fail. The files agree with each other, there is no artifact, and the terminal event is not completed, so the banner is class="banner bad" with text "Not green: drift check failed." Note what this case shows: agreement between two files is not the same as success. The page can be fully consistent and still red.

Case 3: retries exhausted (must be red)

{"ts":"2026-11-06T18:27:22Z","event":"job_started","detail":"steps=4 attempt=1"}
{"ts":"2026-11-06T18:27:22Z","event":"drift_failed","detail":"holdout: 14/20 vs baseline 20/20 (max drop 20%); refusing artifact"}
{"ts":"2026-11-06T18:27:22Z","event":"job_started","detail":"steps=4 attempt=2"}
{"ts":"2026-11-06T18:27:22Z","event":"drift_failed","detail":"holdout: 14/20 vs baseline 20/20 (max drop 20%); refusing artifact"}
{"ts":"2026-11-06T18:27:22Z","event":"job_failed","detail":"attempts=2 backoffLimit=1"}

Status: lines: 5 / last_event: job_failed / attempts: 2 / artifact: none / drift: fail. This log was hand-written to show the retry-budget pattern, not captured from a trainer run. Two things to notice: the last terminal event wins (there are two drift_failed lines before it), and attempts are read from the last attempt=/attempts= value, giving 2. Banner: "Not green: job failed (retry budget exhausted)."

Case 4: the edited log (the teaching case)

Take Case 2's log and change one thing, line 7: the event name drift_failed becomes completed. The detail is left alone, so it still says "refusing artifact". Edit the status to match: last_event: completed, drift: pass, with artifact: none.

{"ts":"2026-11-06T18:27:22Z","event":"completed","detail":"holdout: 14/20 vs baseline 20/20 (max drop 20%); refusing artifact"}

Every cross-check passes: seven lines, terminal completed, drift pass, no artifact on either side. A page with only the five build errors would go green here. The contradiction check is what catches it: an event completed whose own detail says it refused the artifact. The page lists "Line 7: event completed but completed but detail refuses artifact", the banner is bad (with data-verdict="completed", and the reason text "log contradicts itself"), and no green appears. (The wording of that list item repeats "completed but"; it's a quirk of how the sentence is assembled, left as written in the reference.)

How the code does it

Everything lives in two files. train.ts is the logic: parseStatus, parseLog, and buildTrain, which returns an object with ok. The key lines:

const driftWanted = expectedDrift(lastTerminal);
if (status.drift !== driftWanted && !(driftWanted === "unknown" && status.drift === "unknown"))
  throw new Error(`terminal ${lastTerminal} expects drift ${driftWanted} but status says ${status.drift}`);

const ok = lastTerminal === "completed" && contradictions.length === 0;

That second line is the entire definition of green. Reaching it already implies the status agreed, since any disagreement threw earlier.

build.ts turns a Train into HTML. The banner function starts with the only branch that can be green:

function banner(t: Train): string {
  if (t.ok)
    return `<p class="banner ok" data-banner="ok" data-verdict="completed">Training completed: last event completed, status agrees, no contradictions.</p>`;
  const why =
    t.lastTerminal === "drift_failed" ? "drift check failed"
    : t.lastTerminal === "job_failed" ? "job failed (retry budget exhausted)"
    : t.contradictions.length ? "log contradicts itself"
    : `last event ${esc(t.lastTerminal)}`;
  return `<p class="banner bad" data-banner="bad" data-verdict="${esc(t.lastTerminal)}">Not green: ${why}.</p>`;
}

Everything not ok falls through to bad. The default is red; green has one door.

Two details worth copying. First, the data-* attributes (data-banner, data-verdict, data-resumes, data-last-event, data-artifact, data-hash, data-contradiction-line) make the page testable with plain grep, with no browser. Second, every string that came from a log is passed through esc() before it goes into HTML, because log details are free text and < or " in them must not become markup.

For file work the page uses what Bun documents: Bun.file(path) returns a lazily-loaded file object whose .text() reads the contents as a string, and Bun.write(destination, data) writes. Directory operations (mkdir, readdir) come from node:fs, which Bun implements.

What the page does not prove

State this to anyone who relies on the page:

• It checks that two files agree with each other and with the trainer's rules. It does not prove either file is authentic.
• A careful rewrite that changes both the event and the detail, so no contradiction is visible, would still build, and could look green if the terminal event is completed.
• A stale status report from a different log that happens to match lines, last_event, attempts, artifact and drift would also pass.

Two further gaps, found by reading the reference code. These are not in the lab's checks and were not run, so treat them as exercises to confirm:

• Green does not require an artifact. Take Case 4 and delete ; refusing artifact from the detail. The status already says artifact: none. Reading the code, nothing then objects: completed with no artifact_written line goes green.
• A failed build can leave an old page behind. build.ts creates dist/ but never empties it, and one throwing fixture stops the whole loop. A previous run's green HTML could still sit in dist/ next to the failure. The lab's check script deletes dist first; a real pipeline must do the same.

A page like this is a consistency check and a guard against honest mistakes. Signing the log (for example, hash-chaining its lines) addresses authenticity, which is a separate job.

3. Lab

Needs: Bun 1.x. No network, no trainer binary, no other dependencies.

Layout: a reference/ folder (the solution), a starter/ folder (your starting point), and a POSIX shell script check.sh. Copy one of the two folders, put check.sh beside it, and run:

sh check.sh

The script writes the four fixture pairs into fixtures/ itself, deletes dist/, runs bun run build, and prints one RESULT <name> PASS|FAIL line per check. Seven checks:

The starter shows only the last event and never throws. It is supposed to pass static_build and fail the other six. Your task is to turn it into the reference: write parseStatus and parseLog, compute the numbers, add the five throwing checks and the contradiction check, then extend build.ts so the page carries the banner and the data-* attributes the checks look for.

Captured result (a clean container run on 2026-11-06 at 16:30 -0500, image oven/bun:1.3-debian, started with --pull=never --network=none):

RESULT static_build PASS
RESULT resumed_completed_green PASS
RESULT drift_failed_never_green PASS
RESULT job_failed_never_green PASS
RESULT contradiction_listed PASS
RESULT no_green_on_bad_terminal PASS
RESULT disagreeing_numbers_fail_build PASS

for the reference, and for the starter static_build PASS followed by six FAIL lines. The gate verdict was GATE PASS: reference 7 of 7, starter failing 6 of 7. The build summary for the reference:

drift_failed.log -> dist/drift_failed.html ok=false last=drift_failed resumes=0 artifact=none hash=none contra=0
edited.log -> dist/edited.html ok=false last=completed resumes=0 artifact=none hash=none contra=1
job_failed.log -> dist/job_failed.html ok=false last=job_failed resumes=0 artifact=none hash=none contra=0
resumed.log -> dist/resumed.html ok=true last=completed resumes=1 artifact=artifacts/model-v1.json hash=8d6c14555bf2 contra=0

and the four doctored-status probes each threw, for example log has 10 lines but status says 9 and terminal completed expects drift pass but status says fail. Only the resumed page is ok=true.

Not checked by the gate: HTML escaping (read esc() and its call sites yourself), a rewrite that changes both event and detail, a stale status report with matching fields, and the two gaps listed at the end of section 2.

Exercises

1. Make the first gap fail: change the rule so green also requires an artifact_written line. Add a fixture and a RESULT line that goes red without it.
2. Make build.ts delete dist/ before writing, and add a check that a throwing fixture leaves no old HTML behind.
3. Add a rule: if the status attempts exceeds the backoffLimit= value shown in a job_failed line, list it as a contradiction.
4. Try breaking the escape: put <b> in a log detail. Confirm the page shows it as text.

4. Other paths

The same honesty rule can be stated in another language, and the same page can be built without a server in more than one way. Two paths are sketched here: a Gleam/Lustre static document that makes a green banner hard to represent incorrectly, and a short note on why a client-side page is the wrong shape for this lesson.

Gleam with Lustre (hypothetical sketch, not run)

Gleam is statically typed with custom types, so "green only from completed with no contradictions" can be made unrepresentable to get wrong: the banner function pattern-matches on the verdict and the contradictions list together. Lustre can render HTML to a string without its client runtime, using element.to_string or element.to_document_string. Its element.text turns a Gleam string into a text node, and element.none renders nothing. Per the Lustre documentation, element.unsafe_raw_html is not escaped and is an XSS risk if given untrusted input, so log text must go through text, never raw HTML.

// Sketch only. Not compiled or run. Element names other than text / to_document_string
// are from memory of Lustre's html and attribute modules and need checking.
import lustre/attribute
import lustre/element.{type Element}
import lustre/element/html

pub type Terminal {
  Completed
  DriftFailed
  JobFailed
  Other(String)
}

pub fn banner(terminal: Terminal, contradictions: List(String)) -> Element(a) {
  case terminal, contradictions {
    // The only green case.
    Completed, [] ->
      html.p([attribute.class("banner ok"), attribute.attribute("data-banner", "ok")], [
        element.text("Training completed: last event completed, status agrees, no contradictions."),
      ])
    DriftFailed, _ -> bad("drift check failed")
    JobFailed, _ -> bad("job failed (retry budget exhausted)")
    Completed, _ -> bad("log contradicts itself")
    Other(e), _ -> bad("last event " <> e)
  }
}

fn bad(why: String) -> Element(a) {
  html.p([attribute.class("banner bad"), attribute.attribute("data-banner", "bad")], [
    element.text("Not green: " <> why <> "."),
  ])
}

Compared with the TypeScript, the compiler can warn if a case is missing, so adding a new terminal event forces you to decide its colour. TypeScript gets the same benefit only if you use a union type and an exhaustive switch, which the reference does not (it compares strings). Note also that Lustre's text escapes for you, whereas the TypeScript must call esc() by hand every time.

Why not a client-side page?

You could ship JavaScript that fetches the files and decides on the colour in the browser. For a page that reads two small files at build time, this adds a server, a runtime and a way for the page to be wrong after it was checked. A static export decides once, at build time, where a failure stops the build.

Check yourself

5. Citations

• Kubernetes documentation, "Jobs", section on Pod backoff failure policy (.spec.backoffLimit; default 6; exponential back-off delay capped at ten minutes). Fetched 2026-11-06.
• Kubernetes documentation, "Persistent Volumes", opening definitions of PersistentVolume and PersistentVolumeClaim. Fetched 2026-11-06.
• PyTorch Tutorials, "Saving and Loading Models", section "Saving & Loading a General Checkpoint for Inference and/or Resuming Training". Fetched 2026-11-06. The lab uses no PyTorch; the toy job's checkpoint is the same idea written as JSON.
• Google for Developers, "Rules of Machine Learning", discussion of training-serving skew (rule 32 on reusing code between training and serving pipelines is nearby). Fetched 2026-11-06.
• Bun documentation, "File I/O" (Bun.file, .text(), Bun.write; mkdir/readdir via node:fs). Fetched 2026-11-06.
• Lustre documentation, module lustre/element, version 5.7.1 (to_document_string, to_string, text, none, unsafe_raw_html). Fetched 2026-11-06.
• sha2 crate documentation, version 0.11 (SHA-256 used by the trainer to fingerprint the artifact; the page only reads the recorded value and shows its first 12 hex characters, and never recomputes it).
• AWS Certified DevOps Engineer - Professional (DOP-C02) Exam Guide, version 1.6, Domain 3, Task Statement 3.3, "Implement automated recovery processes to meet RTO and RPO requirements". The page is only a reporting layer for recovery: it shows whether a resume happened and whether the run really finished.
• Certified Kubernetes Administrator (CKA) Curriculum, version 1.37, domain "Storage", item "Manage persistent volumes and persistent volume claims". Relevance is limited: the artifact path the page shows lives on the volume; the page does not manage volumes.