Checkpoint, resume, and fail honest drift A training job that refuses silent success

Kill at step 4, resume to model-v1, zeros fail 14/20 vs baseline 20/20, backoffLimit stops the loop.

Lab download. Starter is the tree you edit. Worked answer is a separate file; open it after the starter passes. Rust 1.85 or newer, tar, and python3 for walkthrough.sh. One online cargo vendor vendor. The zip has no vendored crates. All lab requirements.

B-ML · A · arc: self-healing-edge · langs: rust, python · DOP-3.3, CKA-WL-4, CKA-STOR-1

By Evan Eugene Pincham.

Concept

This lesson builds the decision logic of trainjob, a small Rust command-line program that trains a model, survives being killed, and refuses to report success when the result is worse than a reference. A training run is something you deploy, and it owes you the same truthful reporting as any other deployment: an exit code and a log that say what happened, including when the answer is bad news. A model swap in a serving system should happen only after an evaluation passes; a training job deserves the same rule.

Be clear about what is real and what is not. The model is a toy: three weights and a bias. The data is generated from a fixed seed (40 training samples and 20 holdout samples). Everything runs on the CPU with no download, and Kubernetes is not running. What is real: the checkpoint and resume state machine, the retry budget, the versioned artifact with a SHA-256 hash, the honest drift failure (exit code 1, no silent success), and the JSON-lines decision log. What is simulated: the model, the durable storage (a directory on disk), the Job controller (a script that relaunches the program), and Kubernetes itself, which appears only as a read-only mapping in prose near the end.

Each term below gets a definition, the reason it exists, and one symptom you can check when it is missing or wrong.

The toy model. The model is a linear classifier: it multiplies three input numbers by three weights, adds a bias, and answers 1 if the total is above zero and 0 otherwise. Training uses plain stochastic gradient descent (SGD), which means that for each example it nudges every weight a small amount in the direction that reduces the error on that example. One "step" in this lesson is one pass over all 40 training samples.

Training job. A training job is a program that runs to completion, adjusts a model's parameters against example data, and leaves behind a file describing the result; unlike a web server, it is meant to exit. Without honest reporting, a job that died halfway and a job that finished look identical to whoever launched it. Check: after any run, echo $? and the last line of the log must tell the same story.

Exit code. An exit code is the small integer a process hands back to whatever started it, and it is the only thing a launcher can branch on without parsing text. In trainjob, 0 means completed, 1 means an honest failure (drift_failed or job_failed), 2 means a usage or input error, and 75 means "killed mid-run, safe to relaunch". Without distinct codes, a launcher cannot tell "try again" from "give up". Check: run trainjob status on a log file that does not exist and confirm you get 2, not 1.

Checkpoint. A checkpoint is a snapshot of everything needed to continue training later; here that is the step number, the three weights, the bias, and the latest loss, stored as JSON. The PyTorch tutorial "Saving and Loading Models" says it plainly: "When saving a general checkpoint, to be used for either inference or resuming training, you must save more than just the model's state_dict. It is important to also save the optimizer's state_dict, as this contains buffers and parameters that are updated as the model trains." This SGD has no momentum or other optimizer buffers, so weights plus step is enough; an optimizer with buffers would need them saved too. Without checkpoints, a kill at step 9 of 10 costs all nine steps. With one every K steps, you lose at most K steps of work; the amount of work you accept losing is what recovery planning calls the recovery point objective (RPO). Check: after a kill, the loss logged by the next checkpoint should continue downward from the last pre-kill value, not jump back to the first-step value.

Atomic write. An atomic write is a file update that a reader sees either completely or not at all. The program writes the bytes to a temporary file in the same directory and then renames it over the real name, and a rename replaces the old file in one move. Without it, a kill during the write leaves a half-written checkpoint.json that the next run would try to load. Check: after a kill, checkpoint.json always parses as JSON; a stray checkpoint.json.tmp.<pid> beside it is harmless extra file. This protects against a process being killed. It does not protect against power loss, which would also need the data flushed to disk (fsync) before the rename, and the Rust code here does not do that.

Resume. To resume is to start a new process that loads the latest checkpoint and continues from the saved step instead of step 0. Without it, a checkpoint file is decoration, and a job that gets killed more often than it takes to run can never finish. Check: the second run's log begins with resumed, not job_started, and its checkpoint steps continue from where the first run stopped.

PVC as durable storage. In Kubernetes, a PersistentVolumeClaim (PVC) is how a workload asks for storage that outlives any single container. The Kubernetes documentation, "Persistent Volumes", defines the two halves: "A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes." and "A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources." In the lab, --pvc P is simply a directory on disk, and everything that must survive a kill lives there: the checkpoint, the attempts counter, and the artifacts. Without durable storage, the checkpoint dies with the process that wrote it. Check: after a kill and relaunch, P/checkpoint.json is still there; if it is missing, it was written somewhere outside the volume.

**backoffLimit (retry budget).** A retry budget is the number of times a failed job may be started again before it is declared failed for good; Kubernetes calls it backoffLimit. The Kubernetes documentation, "Jobs", says: "set .spec.backoffLimit to specify the number of retries before considering a Job as failed. The .spec.backoffLimit is set by default to 6". Without a budget, a job that fails every time is relaunched forever and burns compute while looking busy. In the lab the default is 2, so a limit of B allows B + 1 starts. Check: count the job_started lines in the log; they must never exceed the limit plus one, and the next start must produce job_failed.

Job controller (a relaunch loop). A Job controller is the part of an orchestrator that watches a job, notices a failed attempt, and starts a new one until the retry budget runs out. From the same Jobs page: "Failed Pods associated with the Job are recreated by the Job controller with an exponential back-off delay (10s, 20s, 40s ...) capped at ten minutes." Without it, a killed training run stays dead until a person notices. In the lab the controller is the checker script, which runs the program, looks at the exit code, and runs it again. It has no delay. Check: the relaunch decision should depend on the exit code and the log, never on the controller guessing.

Versioned artifact. A versioned artifact is the output of a run stored under a name that identifies which attempt produced it, here artifacts/model-v{attempt}.json. Without versions, a later run overwrites the earlier file and nobody can say which attempt made the model now in use. Check: the number in the artifact name equals the attempt= value in that run's job_started line.

SHA-256 file hash. A SHA-256 hash is a 64-character hexadecimal fingerprint computed from a file's exact bytes; change one byte and the fingerprint changes completely. The program records it in the log next to the artifact name, so anyone holding the file can verify it is the one the log describes. Without it, model-v1.json could be truncated or hand-edited and nothing would show it. Check: run sha256sum (or shasum -a 256 on macOS) on the file and compare to the value in the artifact_written line.

Holdout set. A holdout set is data the model never trained on, kept aside to measure how it behaves on unseen examples. Scoring on the training data only tells you the model memorized it. Here the generator draws 20 holdout samples after the 40 training samples from the same seeded stream, so none of them appear in training. Check: train.jsonl and holdout.jsonl share no lines.

Baseline model. A baseline model is a frozen reference that the new model is compared against. Without one, "accuracy 70%" has no meaning; with one, "worse than the reference by more than we allow" is a decision. The lab's baseline.json has fixed weights [1.0, 0.5, -1.0] and bias 0. One honest caveat: those weights are exactly the rule the generator uses to label the data, so the baseline is perfect on this synthetic holdout by construction, which real data would never allow. Check: scoring the baseline on the holdout gives the same count on every run.

Drift check. A drift check compares a trained model's score on the holdout set with the baseline's score and fails the job when the gap is too large. It borrows from the training-serving skew idea in Google's "Rules of Machine Learning": "Training-serving skew is a difference between performance during training and performance during serving." That guide's advice is "The best solution is to explicitly monitor it so that system and data changes don't introduce skew unnoticed." The lab's check is a simpler cousin, run at the end of training with no serving traffic, but it has the property that matters: it can say no. Without a check that can fail, a bad train is silently called done. The rule is integer arithmetic on counts, with no float threshold: pass when correct * 100 >= bcorrect * (100 - max_drop). Check: feed it a result that is exactly on the boundary and confirm the answer is the same on every machine.

Worked example

The walkthrough below is a captured run of the reference program inside the lab container with networking disabled. It shows a kill at step 4, a resume that finishes, a versioned artifact whose SHA-256 is recorded, and a second run that fails the drift check on purpose.

Generating the data

trainjob gen DIR writes train.jsonl, holdout.jsonl and baseline.json. JSON-lines (JSONL) means one JSON object per line. The numbers come from a linear congruential generator (LCG), a tiny formula that turns the previous state into the next one, started at state 7:

    let mut state: u64 = 7;
    let mut next = move || {
        state = state
            .wrapping_mul(6364136223846793005)
            .wrapping_add(1442695040888963407);
        ((state >> 40) as f64 / (1u64 << 24) as f64) * 2.0 - 1.0
    };
    let label = |x: [f64; 3]| u8::from(x[0] + 0.5 * x[1] - x[2] > 0.0);

Same seed, same bytes, every time. The checker confirms this by generating twice and comparing the files.

The three decisions

The program's judgment lives in three small functions. This is the reference version:

/// Resume only when a checkpoint file is present on the PVC.
fn should_resume(ckpt_exists: bool) -> bool {
    ckpt_exists
}

/// Drift passes when holdout correct-count stays within max_drop percent of the baseline.
/// Integer arithmetic: correct * 100 >= bcorrect * (100 - max_drop).
fn drift_passes(correct: usize, total: usize, bcorrect: usize, _btotal: usize, max_drop: usize) -> bool {
    total > 0 && correct * 100 >= bcorrect * (100 - max_drop)
}

/// Still inside the retry budget when prior attempts have not yet exceeded the limit.
fn within_backoff(attempts: usize, limit: usize) -> bool {
    attempts <= limit
}

Take the drift rule with the numbers from the walkthrough below. With max_drop 20 and a baseline of 20 correct out of 20, a pass needs correct * 100 >= 20 * 80, which is correct * 100 >= 1600, so 16 or more correct. A model scoring 14 gives 1400, which is less than 1600, so it fails. No division happens anywhere.

Start: resume or begin

run checks the PVC directory for a checkpoint before it does anything else:

    let ckpt_path = format!("{pvc}/checkpoint.json");
    let ckpt_exists = std::path::Path::new(&ckpt_path).exists();
    let loaded: Option<Checkpoint> = if should_resume(ckpt_exists) {
        fs::read(&ckpt_path)
            .ok()
            .and_then(|b| serde_json::from_slice::<Checkpoint>(&b).ok())
            .filter(|c| c.step < steps) // a finished-but-failed attempt is not a resume
    } else {
        None
    };

    let prior = read_attempts(&pvc);
    let (attempt, mut step, mut weights, mut bias) = if let Some(c) = loaded {
        // Same Job attempt: do not bump attempts; durable PVC is what makes resume possible.
        let attempt = prior.max(1);
        let _ = append_event(&log, "resumed", &format!("from_step={}", c.step));
        (attempt, c.step, c.weights, c.bias)
    } else {
        if !within_backoff(prior, backoff_limit) {
            let _ = append_event(
                &log,
                "job_failed",
                &format!("attempts={prior} backoffLimit={backoff_limit}"),
            );
            return ExitCode::from(1);
        }
        let attempt = prior + 1;
        // ... write the attempts file, append `job_started`, begin at step 0

Two details matter here. The .filter(|c| c.step < steps) means a checkpoint only counts as a resume point if training is not already finished. If a run finished all its steps and then failed the drift check, its checkpoint sits at the final step. Resuming from it would skip training, re-score the same weights, fail again, and never bump the attempts counter, so the retry budget would never run out. Treating that checkpoint as "not a resume" sends the next start down the fresh-attempt path, which does count.

The second detail is the attempts file. P/attempts is a plain integer. A resume keeps the same attempt number, so a kill at step 4 followed by a relaunch produces model-v1. A fresh start bumps it, and within_backoff is checked first: if the attempts already recorded exceed the limit, the program appends job_failed and exits 1 without training.

Train, checkpoint, die

Each step is one SGD pass over the training samples:

fn train_one_epoch(weights: &mut [f64; 3], bias: &mut f64, data: &[Sample], lr: f64) -> f64 {
    let mut loss_sum = 0.0;
    for s in data {
        let target = if s.y == 1 { 1.0 } else { -1.0 };
        let score: f64 = weights.iter().zip(&s.x).map(|(w, v)| w * v).sum::<f64>() + *bias;
        let err = score - target;
        loss_sum += 0.5 * err * err;
        for i in 0..3 {
            weights[i] -= lr * err * s.x[i];
        }
        *bias -= lr * err;
    }
    loss_sum / data.len().max(1) as f64
}

The loss is squared error against targets of +1 and -1, a simple choice that is enough for a toy. The loop around it writes the checkpoint through the atomic helper and then honors --die-at:

fn atomic_write(path: &str, bytes: &[u8]) -> std::io::Result<()> {
    let tmp = format!("{path}.tmp.{}", std::process::id());
    if let Some(parent) = std::path::Path::new(path).parent() {
        fs::create_dir_all(parent)?;
    }
    fs::write(&tmp, bytes)?;
    fs::rename(&tmp, path)?;
    Ok(())
}

        if step % ckpt_every == 0 || step == steps {
            let ckpt = Checkpoint {
                step,
                weights,
                bias,
                loss_milli,
            };
            // ... serialize, atomic_write to checkpoint.json, append `checkpointed`
        }
        if let Some(d) = die_at {
            if step == d {
                let _ = append_event(&log, "killed", &format!("at_step={d}"));
                return ExitCode::from(75);
            }
        }

The temporary file name carries the process id and lives next to the real file, so the rename never crosses a filesystem boundary. The kill check comes after the checkpoint write, so --die-at 4 with the default checkpoint interval of 2 leaves a checkpoint at step 4. Note that this kill is polite: the program exits itself. A real kill can land anywhere, which is why the write is atomic. If you kill at a step that is not a checkpoint step, such as --die-at 3, the newest checkpoint is still step 2, and a resume repeats step 3. That repeated step is the recovery cost described under "Checkpoint".

Drift gate and artifact

After the last step, the program scores the trained weights and the baseline on the same holdout. For testing, a hook replaces the weights with zeros so a finished training run fails the check without any change to SGD:

    if let Some(fw) = force_weights {
        weights = fw;
        bias = 0.0;
    }

    let (correct, total) = evaluate(&weights, bias, &holdout);
    let (bcorrect, btotal) = evaluate(&baseline.weights, baseline.bias, &holdout);

    if !drift_passes(correct, total, bcorrect, btotal, max_drop) {
        let _ = append_event(
            &log,
            "drift_failed",
            &format!(
                "holdout: {correct}/{total} vs baseline {bcorrect}/{btotal} (max drop {max_drop}%); refusing artifact"
            ),
        );
        return ExitCode::from(1);
    }

On failure the function returns before any artifact is written. On success it writes the artifact, hashes the same bytes it just wrote, and logs the hash:

    let art_path = format!("{art_dir}/model-v{attempt}.json");
    // ... art_bytes is the serialized artifact
    if let Err(e) = atomic_write(&art_path, &art_bytes) {
        eprintln!("trainjob: run: artifact: {e}");
        return ExitCode::from(2);
    }
    let hex = to_hex(&Sha256::digest(&art_bytes));
    let _ = append_event(&log, "artifact_written", &format!("sha256={hex} path=artifacts/model-v{attempt}.json"));

The hash comes from the sha2 crate's Sha256 type, documented on docs.rs.

What a kill and a resume look like

This is the captured walkthrough: generate data, kill at step 4, relaunch without --die-at, then read the status.

$ trainjob gen data
wrote train.jsonl (40), holdout.jsonl (20), baseline.json to data
$ trainjob run --die-at 4 --steps 10
exit=75
checkpoint step: 4
$ trainjob run (resume, no die-at)
exit=0
$ trainjob status wt.log
lines: 10
last_event: completed
attempts: 1
artifact: artifacts/model-v1.json
drift: pass
exit=0
--- wt.log (event | detail)
job_started | steps=10 attempt=1
checkpointed | step=2 loss_milli=310
checkpointed | step=4 loss_milli=241
killed | at_step=4
resumed | from_step=4
checkpointed | step=6 loss_milli=228
checkpointed | step=8 loss_milli=226
checkpointed | step=10 loss_milli=225
artifact_written | sha256=8d6c14555bf2... path=artifacts/model-v1.json
completed | holdout: 20/20 vs baseline 20/20

Read it as a state machine: job_started, two checkpoints, killed, then resumed from step 4 (not 0), three more checkpoints, the artifact, completed. The loss keeps falling across the kill (241, then 228), which is the checkpoint doing its job. The attempt stays 1 and the artifact is model-v1. The walkthrough shortens the hash for display; the real log line carries all 64 characters.

Now the failure path, with the forced-zero weights on a fresh storage directory:

$ trainjob run --force-weights 0,0,0 (fresh pvc)
exit=1
$ trainjob status bad.log
lines: 7
last_event: drift_failed
attempts: 1
artifact: none
drift: fail
exit=1
--- bad.log (event | detail)
job_started | steps=10 attempt=1
checkpointed | step=2 loss_milli=310
checkpointed | step=4 loss_milli=241
checkpointed | step=6 loss_milli=228
checkpointed | step=8 loss_milli=226
checkpointed | step=10 loss_milli=225
drift_failed | holdout: 14/20 vs baseline 20/20 (max drop 20%); refusing artifact

Training ran to the end, and the program still said no: exit 1, artifact: none, no completed. The raw log line the checker greps is a single JSON object:

drift detail: {"ts":"2026-11-06T15:29:56Z","event":"drift_failed","detail":"holdout: 14/20 vs baseline 20/20 (max drop 20%); refusing artifact"}

The retry budget

With --backoff-limit 1, the checker starts the failing job three times. The first two starts train and fail the drift check. The third start reads attempts (now 2), finds that 2 exceeds 1, appends job_failed, and exits without training. The gate recorded this:

backoff: exits=1,1,1 drift_failed=2 job_failed=1 attempts_file=2

A limit of 1 allowed two real attempts, which matches the "B + 1 starts" arithmetic from the Concept section.

The starter, and why it fails

The starter copy of the program differs from the reference in exactly these three functions:

fn should_resume(_ckpt_exists: bool) -> bool {
    false // TODO: resume when a checkpoint exists on the PVC
}

fn drift_passes(_correct: usize, _total: usize, _bcorrect: usize, _btotal: usize, _max_drop: usize) -> bool {
    true // TODO: pass only when holdout stays within max_drop of baseline
}

fn within_backoff(_attempts: usize, _limit: usize) -> bool {
    true // TODO: fail the job when attempts have exceeded backoffLimit
}

Each stub fails in the dangerous direction: it never resumes, never rejects a model, and never gives up. The program still builds and runs, and a clean uninterrupted run looks perfect, which is why the stubs are easy to miss. The checker's gate run printed the following for the reference and then for the starter:

RESULT build PASS
RESULT gen_is_repeatable PASS
RESULT resume_after_kill PASS
RESULT artifact_sha_matches PASS
RESULT drift_fails_honestly PASS
RESULT backoff_limit_honoured PASS
RESULT clean_run_completes PASS
RESULT status_and_usage_exit_2 PASS
RESULT checkpoint_is_json PASS
RESULT build PASS
RESULT gen_is_repeatable PASS
RESULT resume_after_kill FAIL
RESULT artifact_sha_matches FAIL
RESULT drift_fails_honestly FAIL
RESULT backoff_limit_honoured FAIL
RESULT clean_run_completes PASS
RESULT status_and_usage_exit_2 PASS
RESULT checkpoint_is_json PASS

The first nine lines are the reference; the last nine are the starter. The starter passes build, gen_is_repeatable, clean_run_completes, status_and_usage_exit_2 and checkpoint_is_json, and fails resume_after_kill, artifact_sha_matches, drift_fails_honestly and backoff_limit_honoured.

Five failures as diagnostics

Each of these is a symptom you can observe, the cause behind it, and the check that confirms it.

1. **Symptom: after a kill at step 4, the next run starts from step 0 and writes model-v2.** Cause: should_resume ignored the checkpoint, so the relaunch took the fresh-attempt path and bumped the counter. The starter's gate output shows it:

   resume run: exit=0 status_exit=0 last=completed artifact=artifacts/model-v2.json
   resume events: resumed=0 completed=1 artifact_written=1

Check: resumed=0 after a kill means nothing resumed.

1. **Symptom: a holdout of 14/20 against a baseline of 20/20 still produces completed and an artifact.** Cause: drift_passes always returns true. The starter's forced-bad run exits 0 where the reference exits 1:

   drift run: exit=0 status_exit=0 last=completed drift=pass

The reference's line for the same run is drift run: exit=1 status_exit=1 last=drift_failed drift=fail. Check: trainjob status must print drift: fail and exit 1 for the bad run.

1. **Symptom: with --backoff-limit 1, the job never emits job_failed after repeated drift failures.** Cause: within_backoff always returns true. In the starter, the bad first run is accepted as completed, and the later starts are refused because the log already has a completed event:

   backoff: exits=0,2,2 drift_failed=0 job_failed=0 attempts_file=1

Because pitfall 2 hides the drift failures, this check cannot pass until drift_passes works too. Check: count job_failed lines after a limit-plus-one number of starts.

1. **Symptom: the SHA-256 in the log does not match sha256sum of the artifact file.** Cause: the hash was computed on different bytes than the file that was written, or the file was edited after the write. The reference hashes the same art_bytes buffer it wrote, and the gate compared the two:

   artifact sha: file=8d6c14555bf2fabeb363b86bc8c185c13cd0abe77dadc1b10e4e6f29b4f164d7 log=8d6c14555bf2fabeb363b86bc8c185c13cd0abe77dadc1b10e4e6f29b4f164d7

Check: if the two strings differ, do not trust the file. In the starter's run the file is simply missing, because the artifact was written as model-v2.json and the check looks for model-v1.json.

1. Symptom: a comparison of accuracies passes on one machine and fails on another. Cause: the check compares floating-point ratios instead of integer counts. When the two sides are mathematically equal (16 of 20 against a limit of 16), rounding in a float expression can land on either side, and a refactor that reorders the arithmetic can flip the answer. Counts and the multiply-by-100 form are exact. Check: run the boundary case, 16 of 20 here, and confirm it gives the same verdict every time.

Lab

You will fix the three stubs in the starter and run the checker. The lab needs no cloud account, costs nothing, uses the CPU only, and downloads nothing. The checker runs with networking disabled; the Rust crates it needs (serde, serde_json, sha2 and their dependencies) are packed in vendor.tgz next to the code. The lab container provides Rust 1.98.

1. Start from lab/starter/. Inside the lab container, run sh check.sh. The script extracts the vendored crates, builds offline, generates the data, and prints one RESULT line per check. Confirm the starter prints four FAIL lines: resume_after_kill, artifact_sha_matches, drift_fails_honestly, backoff_limit_honoured.
2. In src/main.rs, make should_resume return true when a checkpoint file exists on the storage directory. Rerun the checker. resume_after_kill needs exactly one killed, a checkpoint at step 4, at least one resumed, one completed, and artifacts/model-v1.json. artifact_sha_matches reads that same file.
3. Make drift_passes implement the integer rule from the Concept section. Do not convert anything to a float. Rerun; drift_fails_honestly should flip.
4. Make within_backoff return true while the recorded attempts have not exceeded the limit. Rerun; backoff_limit_honoured needs at least two drift_failed and one job_failed, so it depends on step 3.
5. The finished lab prints nine PASS lines, the same as the reference block shown above.

Optional: before running anything, predict what --die-at 3 with the default checkpoint interval does to the resumed line, then run it and compare. Also try --steps 10 --max-drop 0 and explain the result in terms of counts.

Check yourself

1. After --die-at 4 and a plain relaunch, why is the artifact model-v1 and not model-v2? A resume keeps the attempt number because the checkpoint's step (4) is below the target (10), so the attempts file is not bumped.
2. A model scores 15/20 on the holdout against a baseline of 20/20 with max_drop 20. Does it pass? No: 15 100 = 1500 is below 20 * 80 = 1600, so the job appends drift_failed and exits 1.*
3. With --backoff-limit 1, how many starts actually train before job_failed? *Two; the third start sees attempts=2, which exceeds 1.*
4. Why compare integer counts and not accuracy percentages? Counts are exact, so a result on the boundary gets the same verdict on every machine and build.
5. The log's SHA-256 differs from sha256sum of the artifact. What does that tell you? The bytes hashed are not the bytes on disk, so the file was edited after the write or the hash was taken from a different buffer, and the artifact should not be trusted.

Other paths

A Python sketch, not run. The same JSON shapes work in Python with the standard library only. This sketch has not been executed. It adds an fsync that the Rust code omits, which closes the power-loss gap described under "Atomic write".

import json, os

def save_checkpoint(pvc, step, weights, bias, loss_milli):
    path = os.path.join(pvc, "checkpoint.json")
    tmp = f"{path}.tmp.{os.getpid()}"
    with open(tmp, "w") as f:
        json.dump({"step": step, "weights": weights,
                   "bias": bias, "loss_milli": loss_milli}, f)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)  # atomic replace on the same filesystem

def load_checkpoint(pvc, steps):
    try:
        with open(os.path.join(pvc, "checkpoint.json")) as f:
            ckpt = json.load(f)
    except (FileNotFoundError, json.JSONDecodeError):
        return None
    return ckpt if ckpt["step"] < steps else None  # finished runs are not resumes

def drift_passes(correct, total, bcorrect, max_drop):
    return total > 0 and correct * 100 >= bcorrect * (100 - max_drop)

Kubernetes Job and PVC, read-only mapping. Nothing in this lesson ran on a cluster, and no manifest was written or applied. If you moved the idea to Kubernetes, the pieces would map like this:

The Jobs page's example manifest uses backoffLimit: 4 and restartPolicy: Never. The pieces that carry over unchanged are the ideas: state that survives the process, a bounded number of retries, and a result that names itself and can be verified.

Citations

• Kubernetes documentation, "Jobs": .spec.backoffLimit (default 6) and the Job controller's exponential back-off for failed Pods. Fetched 2026-11-06.
• Kubernetes documentation, "Persistent Volumes": the definitions of PersistentVolume and PersistentVolumeClaim. Fetched 2026-11-06.
• PyTorch Tutorials, "Saving and Loading Models", section "Saving & Loading a General Checkpoint for Inference and/or Resuming Training". The lab's checkpoint carries the same idea (step, weights, bias, loss) without PyTorch. Fetched 2026-11-06.
• Google for Developers, "Rules of Machine Learning": the definition of training-serving skew and the advice to monitor it explicitly (Rule #32 on reusing code between training and serving pipelines is the related practice). Fetched 2026-11-06.
• Google for Developers, Machine Learning Crash Course, "Monitoring ML pipelines": further reading on watching a pipeline for skew. Fetched 2026-11-06.
• sha2 crate 0.11 / Rust Sha256 (https://docs.rs/sha2/latest/sha2/ ): artifact fingerprint.
• AWS Certified DevOps Engineer - Professional (DOP-C02) Exam Guide v1.6, Domain 3, Task Statement 3.3, "Implement automated recovery processes to meet RTO and RPO requirements". The checkpoint and resume path is a small local version of automated recovery. The lab runs no AWS service.
• Certified Kubernetes Administrator (CKA) Curriculum v1.37, Workloads and Scheduling: understanding the primitives used to build self-healing application deployments; the Job retry budget is one such primitive. The lab runs no Kubernetes service.
• Certified Kubernetes Administrator (CKA) Curriculum v1.37, Storage: "Manage persistent volumes and persistent volume claims"; the lab practices the idea with a directory standing in for the claim.