Hedronite · Curriculum · AI · Rust · Thu 2026-11-05

Treat a model swap like a deploy A model that must pass its eval before it serves

Three model versions, one rejected at 12/20, one forged log line refused, one swapped file refused.

Lesson Class: Curriculum · B-AI · rung A
Tested with: Rust 1.98 · sha2 0.11 · hmac 0.13 · serde_json 1
Lab: lab/ · sh check.sh (10 checks) · CPU only. Crates are not in the download
Source: Kubernetes Deployments and probes · SageMaker deployment guardrails · OpenAI chat completions reference · Google ML Crash Course · OWASP LLM03:2025 · Rust std TcpListener · Go crypto/sha256

Lab download. Starter is the tree you edit. Worked answer is a separate file; open it after the starter passes. Rust 1.85 or newer (the lesson was tested with Rust 1.98; cargo vendor was checked on Rust 1.99) and tar. One online command, cargo vendor vendor, from inside starter/ or reference/. The zip has no vendor directory, so it is not offline-ready until that command finishes. The checker then builds with cargo build --offline. DEPLOY_LOG_KEY in the checker is the demo string lab-only-key-0123456789abcdef0123456789. All lab requirements.

Gate on Counts
Promote only when correct * 100 >= total * threshold, integers, no float compare.
Serve the Last-Good
What serves is the last verified promoted line, never the newest line.
Hash the Identity
The version is a label; the SHA-256 recorded at promotion is the identity.
A model swap with no gate is a file copy; a model swap with a gate and no record is a story.

B-AI · A · arc: self-healing-edge · langs: rust, go · DOP-5.3, CKA-WL-1, CKS-SC-3

By Evan Eugene Pincham.

This page walks through a small Rust program, modelgate, that treats replacing a machine-learning model as a deploy. A new model version is scored on a fixed set of questions before it serves. If it scores too low, it is rolled back, the previous version keeps serving, and the decision is written to a log that cannot be quietly edited afterward.

The scale is deliberately small. The "model" is a linear classifier with three weights and a bias, generated from a fixed seed. The eval set is 20 generated samples. Everything runs on a CPU with no download. The "chat completion" is the word positive or negative, chosen for three numbers sent as the message text.

What is real is the state machine, the sealed decision log, the file-hash check, the HTTP surface and the probes. What is simulated is the model, the eval set and the traffic. A swap is all-at-once, with no canary.

Concept

A model version is one named, frozen set of numbers that turns inputs into answers. Here v1, v2 and v3 are JSON files, each holding a version string, three weights and a bias. Replacing v1 with v2 changes every answer the service gives without changing a line of code, so the swap needs the same care as a code release.

A deploy is the act of replacing what is serving with something new, together with a decision about whether the new thing may take over. Without that decision, a model swap is a file copy, and the first sign of a bad swap is a user complaint. The check: pick a past moment and ask which version was serving and who approved it. If no record answers, the swap was a copy.

An eval set is a fixed list of inputs with known correct answers, used only to judge candidates. In this lab eval.json holds 20 generated samples, each shaped {"x":[3 numbers],"label":0|1}. It must be fixed. If the set changes between runs, a pass on Monday and a fail on Tuesday say nothing about the model.

An eval gate is a check that runs a candidate over the eval set and allows promotion only when the score reaches a threshold. Without it, whatever was most recently copied into place serves. The pitfall is a gate that is wired up but not enforced. The symptom is a log line that contradicts itself: a candidate that scored 12/20 appears as promoted, and /readyz answers ready v2. The cause is a gate function that returns true, or whose result is never read. The check is to read the detail of the promoted line and compare its counts with the threshold printed next to them.

Accuracy is "the proportion of all classifications that were correct, whether positive or negative" (Google, Machine Learning Crash Course, "Classification: Accuracy, recall, precision, and related metrics"). Here it is correct / total. The same page adds a caveat: "when the dataset is imbalanced, or where one kind of mistake (FN or FP) is more costly than the other... it's better to optimize for one of the other metrics instead."

That caveat is why a 20-sample accuracy gate is a teaching device and not a production gate. With 20 samples, one sample is worth 5 percentage points. The gate says nothing about a rare class, the cost of one kind of mistake, latency, or a regression in how the model handles hostile input.

The gate compares counts, not floats, so there is no division. A candidate passes when correct * 100 >= total * min_pct. A float comparison against 0.9 is where machines and libraries can disagree. The symptom of that mistake is the same model passing on one machine and failing on another. The lab therefore asserts thresholds and state transitions and never an exact float.

A liveness probe is a periodic check that decides whether a process is stuck. When it fails, Kubernetes' kubelet "kills the container and restarts it". A readiness probe is a check that decides whether a process should receive requests right now. The Kubernetes documentation ("Configure Liveness, Readiness and Startup Probes") describes the case: "you don't want to kill the application, but you don't want to send it requests either," and "A pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services." For an HTTP probe, "Any code greater than or equal to 200 and less than 400 indicates success. Any other code indicates failure."

A model that loads fine but answers badly is a readiness problem. Restarting it cannot help, because the same bytes produce the same wrong answers. The symptom of confusing the two is /healthz returning 200 while answers are wrong, and a restart changing nothing. The eval gate is the readiness probe for a model: the candidate does not serve until it has answered a fixed set of questions well enough. The analogy is loose in one way. The gate runs once, offline, at deploy time, and is not a recurring check on live traffic.

A rollback is returning to the previous working version after a new one is rejected or fails. The last-good version is the most recent one that passed the gate, and so the one to return to. While a candidate is being judged, last-good keeps serving.

The pitfall is deriving "what serves" from the wrong place. If the service restarts after a rollback and serves the rejected version, serving was taken from the newest log line instead of the last verified promoted line. The check is to look at the log's final line. After a rollback it names the rejected candidate, so any code that reads the last line gets the wrong answer.

An artifact is the file a build step produces and a deploy installs. Here it is a model file such as v1.json. A SHA-256 file hash is a 32-byte fingerprint of a file's exact bytes, written as 64 hexadecimal characters, such that changing any byte changes the fingerprint. The version string v1 is a label, and the hash is the identity.

The pitfall: /readyz says ready v1, but answers differ from yesterday's. The cause is that the file behind the name v1 changed. The check is to recompute the hash and compare it with the one recorded when the version passed the gate.

OWASP's "LLM03:2025 Supply Chain" says "Currently there are no strong provenance assurances in published models". Among its mitigations is to "use third-party model integrity checks with signing and file hashes to compensate for the lack of strong model provenance." The lab records a hash it computed itself at promotion. That proves "same bytes as the ones that passed the gate". It does not prove "from a trustworthy source".

A hash chain is a log in which every line carries a code computed over its own content and the previous line's code, so editing one line is detectable. In modelgate the code is an HMAC-SHA-256, a hash that needs a secret key to produce, so someone without the key cannot edit a line and recompute its code. The lab's key is a throwaway constant set by the checker. A real key comes from a secret store and is rotated; RFC 2104, section 3 ("Keys"), discusses keys being "periodically refreshed".

The pitfall: someone rewrote a rolled_back line as promoted and the service starts anyway. The cause is a chain that was never verified at start-up. The check is modelgate status: read its exit code first, then broken and first_break. One limit applies. A chain shows edits and removed middle lines, but it cannot show that lines were cut off the end.

An OpenAI-compatible chat completions endpoint is an HTTP route, POST /v1/chat/completions, that accepts a JSON body with messages and model and answers with a JSON object. The answer has object: "chat.completion" and choices, and each choice has an index, a message (role: "assistant" and content) and a finish_reason. OpenAI's docs say finish_reason is stop "if the model hit a natural stop point or a provided stop sequence". The response's model field is described as "The model used for the chat completion." That is where a client can see which version answered (OpenAI API reference, "Create chat completion"). The point is that existing clients can talk to it unchanged.

The stub implements only a few of the fields on that page. The page lists many more, including usage, logprobs and tool calls. The symptom of a client expecting more is an HTTP 400 with an error object when the message text is not exactly three numbers.

An exit code is the small integer a process hands back to its parent when it ends. By convention 0 means success and anything else means something went wrong, and a shell reads it as $?. Scripts and orchestrators branch on it, because they do not parse prose. modelgate uses four values: 0 for success, 1 for a decision that went the wrong way (a rollback, an untrusted log), 2 for a missing or short key, a missing file or a usage error, and 3 for a refusal to deploy or serve. The pitfall is reading printed text before the exit code. The example appears under "An edited log" below.

Worked example

The example below follows one run of the walkthrough script from a clean checkout: generate three model versions, deploy a good one, try a bad one, deploy a third, then edit the log and watch the service refuse it. Each code block comes from the reference program, and each console block is captured output.

The model and the score

fn predict(m: &Model, x: &[f64; 3]) -> u8 {
    let score: f64 = m.weights.iter().zip(x).map(|(w, v)| w * v).sum::<f64>() + m.bias;
    u8::from(score > 0.0)
}

/// Run the model over the fixed eval set and count the correct answers.
fn evaluate(m: &Model, eval: &[Sample]) -> (usize, usize) {
    let correct = eval.iter().filter(|s| predict(m, &s.x) == s.label).count();
    (correct, eval.len())
}

predict multiplies three inputs by three weights, adds the bias, and answers 1 when the score is above zero. evaluate returns two integers, (correct, total). Everything the gate decides from here is made of those two integers.

The generator labels each eval sample with the rule x[0] + 0.5 * x[1] - x[2] > 0.0. v1 uses exactly those weights, so it scored 20/20 in the observed run. That is what makes it a toy. v2 flips the sign of the first weight and scored 12/20. v3 keeps v1's weights and adds a bias of 0.05. It scored 19/20 in the observed run, which still passes a 90% gate because 19 × 100 = 1900 is at least 20 × 90 = 1800.

The three decisions

The program is complete except for three small functions. These are the reference versions:

/// The eval gate: pass only when at least `min_pct` percent of the eval set is answered correctly.
/// Integer arithmetic on counts, so there is no float comparison to get wrong.
fn gate_passes(correct: usize, total: usize, min_pct: usize) -> bool { total > 0 && correct * 100 >= total * min_pct }

/// A log may be trusted only if every line is verified.
fn chain_trusted(v: &Verdict) -> bool { v.broken == 0 && v.unreadable == 0 }

/// The model file on disk must be the bytes that passed the gate: its SHA-256 is recorded in the `promoted` line.
fn artifact_matches(bytes: &[u8], promoted_detail: &str) -> bool { promoted_detail.contains(&format!("sha256={}", to_hex(&Sha256::digest(bytes)))) }

correct * 100 >= total * min_pct is the same test as correct / total >= min_pct / 100 for a positive total, without a division. The total > 0 guard matters. With an empty eval set, 0 * 100 >= 0 * 90 is true, and the gate would pass anything.

The boundary is inclusive. 18/20 is exactly 90% and passes at a 90 threshold, and 17/20 does not.

chain_trusted is true only when no line is broken and no line is unreadable. artifact_matches recomputes the SHA-256 of the file's bytes and looks for sha256=<those 64 characters> in the promoted line's detail. That detail is only consulted after the chain has verified, so it cannot have been edited.

The starter has the identical program with these three bodies replaced:

fn gate_passes(_correct: usize, _total: usize, _min_pct: usize) -> bool { true } // TODO: promote only when the eval gate passes
fn chain_trusted(_v: &Verdict) -> bool { true } // TODO: believe the log only when every line verified
fn artifact_matches(_bytes: &[u8], _promoted_detail: &str) -> bool { true } // TODO: compare the file hash with the one recorded at promotion

A swap that is rejected

The walkthrough script generates the fixtures, deploys v1, then tries v2. It abbreviates the command lines. The real flags are --dir, --log and --candidate. It also shortens the hash for display, while the logged line holds all 64 hex characters.

$ modelgate gen models
wrote v1 v2 v3 and eval.json (20 samples) to models
$ modelgate deploy --candidate v1
promoted v1
exit=0
$ modelgate deploy --candidate v2
rolled_back v2
exit=1
$ modelgate status deploy.log
lines: 4
verified: 4
broken: 0
unreadable: 0
first_break: none
serving: v1
exit=0
--- deploy.log (event version | detail)
candidate_started v1 | pid 541 min_accuracy 90%
promoted v1 | gate pass: 20/20 correct (min 90%); sha256=589469a4dcdc...; replaced none
candidate_started v2 | pid 542 min_accuracy 90%
rolled_back v2 | gate fail: 12/20 correct (min 90%); v1 still serving: true

deploy exits 1 when it rolls back, so a script can stop a pipeline on a rejected model. The last line of the log names v2, yet status says serving: v1. This is the second pitfall. The code that produces serving is in verify:

            Ok(s) => {
                if !mac_ok(key, &s) {
                    v.broken += 1;
                    v.first_break.get_or_insert((n, "mac"));
                } else if s.prev != expected_prev {
                    v.broken += 1;
                    v.first_break.get_or_insert((n, "link"));
                } else {
                    v.verified += 1;
                    if s.event == "promoted" {
                        v.serving = Some((s.version.clone(), s.detail.clone()));
                    }
                }
                expected_prev = s.mac.clone();
                v.head = Some(s.mac);
            }

serving is assigned only inside the verified branch, and only for a promoted event. A rolled_back line, a candidate_started line and a line that fails verification all leave it alone. The lab avoids the newest-line mistake by construction. This lesson does not run a broken variant to show it.

Each line is a Sealed record with ts, event, version, detail, prev and mac. The mac covers the other five fields in a fixed order, and prev is the previous line's mac. A line is "broken" when its mac does not match its contents or its prev does not match the line before. A line is "unreadable" when it cannot be parsed at all, for example blank, truncated, not JSON, or the wrong shape. first_break names the first line that does not belong.

The order of steps inside deploy

deploy first verifies the existing chain. If it is not trusted, deploy exits 3 and writes nothing. Then it appends candidate_started, loads the candidate and the eval set, evaluates, and takes one of two branches:

    let (correct, total) = evaluate(&model, &eval);
    if gate_passes(correct, total, min_pct) {
        let sha = to_hex(&Sha256::digest(&bytes));
        let prev = last_good.unwrap_or_else(|| "none".into());
        let detail = format!("gate pass: {correct}/{total} correct (min {min_pct}%); sha256={sha}; replaced {prev}");
        let detail = format!("gate fail: {correct}/{total} correct (min {min_pct}%); {still}");

The hash is computed from the same bytes that were just evaluated, so "what passed" and "what is hashed" are one buffer. The candidate_started line is written before the evaluation, so a crash mid-evaluation leaves an attempt with no decision. A candidate that cannot be loaded is also rolled back, with cannot load: ... in the detail. If no promoted version exists yet, the failure detail says nothing to serve: no last-good version.

deploy only appends to the log. It never touches a running server. A server loads its model once at start, so a promotion takes effect when a server is next started from the log. The lab has no hot reload.

An edited log

After v3 is promoted, the script edits line 4 from rolled_back to promoted, which is the forgery that would make a rejected model look approved.

$ modelgate deploy --candidate v3
promoted v3
exit=0
$ modelgate status deploy.log
lines: 6
verified: 6
broken: 0
unreadable: 0
first_break: none
serving: v3
exit=0
--- edit line 4: rolled_back -> promoted
$ modelgate status edited.log
lines: 6
verified: 5
broken: 1
unreadable: 0
first_break: line 4 (mac)
serving: v3
exit=1
$ modelgate serve (edited log)
modelgate: log chain is broken at line 4 (mac); refusing to serve
exit=3

Only one line is counted broken. The edit changed the text but not the mac field, so line 5's prev still matches what verify carries forward. Line 4's mac no longer covers its edited contents, which is why the break is reported as (mac).

Read the exit code first. status still prints serving: v3, because the forged line is not counted as a promotion, but the exit code is 1. This is where "text before exit code" goes wrong. Someone scanning only the serving: line would see a healthy answer from a log that failed verification. serve and deploy both refuse on a broken chain with exit 3.

Serving

serve runs its checks in order. It requires a trusted chain, a verified promoted line, a loadable model file and a matching hash. Only then does it bind a socket:

    if !chain_trusted(&v) {
        let (n, kind) = v.first_break.unwrap_or((0, "?"));
        eprintln!("modelgate: log chain is broken at line {n} ({kind}); refusing to serve");
        return ExitCode::from(3);
    }

After that, the request loop:

    for conn in listener.incoming() {
        let Ok(mut stream) = conn else { continue };
        let _ = stream.set_read_timeout(Some(Duration::from_secs(2)));
        if let Some((method, path, body)) = read_request(&mut stream) {
            match (method.as_str(), path.as_str()) {
                ("GET", "/healthz") => respond(&mut stream, 200, "ok"),
                ("GET", "/readyz") => respond(&mut stream, 200, &format!("ready {ver}")),

TcpListener::incoming is, in the Rust standard library documentation's words, "equivalent to calling TcpListener::accept in a loop", and accept "will block the calling thread until a new TCP connection is established". So the server handles one request at a time. A slow client is bounded by the two-second read timeout. That is fine for a lab and wrong for production.

The same blocking loop never ends on its own. --max-requests N makes it stop after N connections, so a test can start the server, make exactly N requests and see it exit. The checker passes 4 because it makes four: a readiness probe, a health probe, one completion and one bad request.

/readyz always answers 200 with the version, which is honest here because serve never reaches the loop unless every check passed. A server that fails a check exits 3 and never answers anything.

The completion route reads only the last message's content, splits it into numbers, and requires exactly three. It ignores the request's model field and reports the version it is actually serving. The response is built with:

                        let out = json!({"id": "chatcmpl-lab", "object": "chat.completion", "created": created, "model": ver,
                            "choices": [{"index": 0, "message": {"role": "assistant", "content": label}, "finish_reason": "stop"}]});

Serving the good log, the walkthrough shows the probes and one completion. The message text was -0.4 0.2 0.6, whose score under v3 is -0.85, so the answer is negative:

--- serve the good log
listening 127.0.0.1:18096 serving v3
status: 200
body: ok
status: 200
body: ready v3
status: 200
body: {"choices":[{"finish_reason":"stop","index":0,"message":{"content":"negative","role":"assistant"}}],"id":"chatcmpl-lab","model":"v3","object":"chat.completion"}

What the starter does

Because the starter's three decisions all return true, it behaves like a deploy tool with no judgment. This is the starter's run under the same checker:

deploy v2: exit=0 promoted v2 | log lines=4 verified=4 serving=v2
last line detail: gate pass: 12/20 correct (min 90%); sha256=844df8030549c1e7496510fb49366678f63bb16245b9a39a04435495cafe0f98; replaced v1
RESULT bad_model_rolled_back FAIL
readyz: status: 200 body: ready v2 
completion: status: 200 body: {"choices":[{"finish_reason":"stop","index":0,"message":{"content":"negative","role":"assistant"}}],"created":1793891157,"id":"chatcmpl-lab","model":"v2","object":"chat.completion"} 
bad request: status: 400
RESULT serving_stays_on_last_good FAIL
deploy v3: exit=0 promoted v3 | log lines=6 verified=6 serving=v3
RESULT later_good_model_promoted FAIL
edited log: exit=0 first_break=none serving=v3
serve on edited log: exit=0 answered_a_probe=1
RESULT edited_log_refused FAIL
serve with swapped v1.json: exit=0 answered_a_probe=1
RESULT swapped_artifact_refused FAIL
deploy onto edited log: exit=0 
RESULT deploy_refuses_broken_chain FAIL

Each line maps to one of the pitfalls in Concept:

later_good_model_promoted fails on the starter only as a consequence of the first failure. It counts the rollback, and the starter never rolled anything back. The other four checks pass on the starter: build, gen_is_repeatable, good_model_promoted and key_and_file_exit_2. A first model promoted on a good score, a repeatable generator and a rejected short key are all things the broken program also does, so those four checks cannot tell a good program from a bad one.

Lab

The three exam objectives this lesson touches ask for the following. DOP-5.3 is AWS Certified DevOps Engineer - Professional (DOP-C02) exam guide v1.6, Task Statement 5.3, "Troubleshoot system and application failures", whose skills include "Analyzing failed deployments". CKA-WL-1 is the Certified Kubernetes Administrator curriculum v1.37, Workloads and Scheduling: "Understand application deployments and how to perform rolling update and rollbacks". CKS-SC-3 is the Certified Kubernetes Security Specialist curriculum v1.37, Supply Chain Security: "Secure your supply chain (permitted registries, sign and validate artifacts, etc.)". The lab runs no AWS or Kubernetes service, signs nothing and uses no registry. It exercises the reasoning, in the form of reading a failed swap from a log, a rollback decision, and validating an artifact by hash.

Setup. The lab runs on a CPU with no account, no cost, no model download and no network. The checker needs cargo and tar. The sha2 and hmac crates and their dependencies are vendored in vendor.tgz, which check.sh extracts the first time it runs. The reference checker run used the lab container with Rust 1.98 and networking disabled. Copy check.sh into the starter tree, next to Cargo.toml and src/main.rs (vendor.tgz is already there), and run:

sh check.sh

Step 1: run the checker on the starter. Expect 4 PASS (build, gen_is_repeatable, good_model_promoted, key_and_file_exit_2) and 6 FAIL (bad_model_rolled_back, serving_stays_on_last_good, later_good_model_promoted, edited_log_refused, swapped_artifact_refused, deploy_refuses_broken_chain).

Step 2: edit src/main.rs. Change only the three decision functions. Write them from the descriptions rather than copying the reference above:

  1. gate_passes must pass only when the share of correct answers reaches min_pct, using integer arithmetic on the counts, and must not pass an empty eval set.
  2. chain_trusted must be true only when every line verified.
  3. artifact_matches must compare the SHA-256 of the file's bytes with the hash recorded in the promoted line's detail.

Step 3: run sh check.sh again. All ten checks should pass. This is the reference run from the lab container, with networking disabled:

RESULT build PASS
RESULT gen_is_repeatable PASS
deploy v1: exit=0 promoted v1 | log lines=2 verified=2 serving=v1
RESULT good_model_promoted PASS
deploy v2: exit=1 rolled_back v2 | log lines=4 verified=4 serving=v1
last line detail: gate fail: 12/20 correct (min 90%); v1 still serving: true
RESULT bad_model_rolled_back PASS
readyz: status: 200 body: ready v1 
completion: status: 200 body: {"choices":[{"finish_reason":"stop","index":0,"message":{"content":"positive","role":"assistant"}}],"created":1793891153,"id":"chatcmpl-lab","model":"v1","object":"chat.completion"} 
bad request: status: 400
RESULT serving_stays_on_last_good PASS
deploy v3: exit=0 promoted v3 | log lines=6 verified=6 serving=v3
RESULT later_good_model_promoted PASS
edited log: exit=1 first_break=line 4 (mac) serving=v3
serve on edited log: exit=3 answered_a_probe=0
RESULT edited_log_refused PASS
serve with swapped v1.json: exit=3 answered_a_probe=0
RESULT swapped_artifact_refused PASS
deploy onto edited log: exit=3 modelgate: log chain is broken at line 4 (mac); refusing to deploy
RESULT deploy_refuses_broken_chain PASS
key empty: exit=2; key short: exit=2; no such log: exit=2
RESULT key_and_file_exit_2 PASS

The checker's key is a throwaway constant exported by check.sh. Do not copy that pattern into anything real.

Try it. In a fresh log, deploy v3 with --min-accuracy 96. From the 19/20 count, 1900 is less than 1920, so you should see rolled_back v3, exit 1, and a detail ending in nothing to serve: no last-good version. This is derived from the code and the observed score, not a captured run. Then try --min-accuracy 95. 1900 is at least 1900, so it should promote.

Check yourself

Question 1
Why does the gate compare correct * 100 >= total * min_pct instead of correct as f64 / total as f64 >= 0.9?
Click to reveal
Counts are exact integers, and a float comparison against a threshold is where machines and libraries can disagree.
Question 2
After v2 is rolled back and the process restarts, what does serving report, and from which line?
Click to reveal
It reports v1, from the last verified promoted line, not from the newest line.
Question 3
status on an edited log prints serving: v3. Why is that not a clean bill of health?
Click to reveal
The exit code is 1 and broken is 1, so the chain failed verification, and the exit code is the verdict.
Question 4
What does the recorded SHA-256 prove, and what does it not?
Click to reveal
It proves the file is the same bytes that passed the gate, and it does not prove those bytes came from a trustworthy source.
Question 5
Why can /readyz always answer 200?
Click to reveal
serve exits 3 before listening unless the chain, the model file and the hash all checked out.

Other paths

None of the three paths below was run for this lesson. The Go code is a sketch that was not compiled, and the Kubernetes and SageMaker sections are read-only mappings from documentation.

Go: the artifact hash

Go's crypto/sha256 package "implements the SHA224 and SHA256 hash algorithms as defined in FIPS 180-4". Sum256(data []byte) returns the checksum of data as a [Size]byte, and Size is 32. The check from artifact_matches would look like this:

// Sketch, not compiled or run.
sum := sha256.Sum256(data)                        // [32]byte
got := fmt.Sprintf("%x", sum)                     // 64 hex characters
ok := strings.Contains(promotedDetail, "sha256="+got)

The comparison target is the hash recorded at promotion, taken from a log line whose chain has already verified.

Kubernetes: Deployment rollout and probes

A Kubernetes Deployment rollout maps onto the same questions. The Deployments documentation says "Sometimes, you may want to rollback a Deployment; for example, when the Deployment is not stable, such as crash looping", and gives kubectl rollout undo deployment/nginx-deployment for it. It also says "The Deployment controller stops the bad rollout automatically, and stops scaling up the new ReplicaSet", which "depends on the rollingUpdate parameters (maxUnavailable specifically)", with a default of 25%. The readiness probe is what tells the controller a new pod is not fit to take traffic.

.spec.progressDeadlineSeconds handles a rollout that stalls. In the page's words, "Kubernetes takes no action on a stalled Deployment other than to report a status condition with reason: ProgressDeadlineExceeded. Higher level orchestrators can take advantage of it and act accordingly, for example, rollback the Deployment to its previous version." That gap is the role modelgate deploy fills here: it decides, acts and records. A real rollout also replaces pods gradually, while this lab swaps all at once.

Amazon SageMaker: deployment guardrails

The SageMaker AI Developer Guide page "Deployment guardrails for updating models in production" describes "built-in safeguards such as auto-rollbacks that help you catch issues early and automatically take corrective action". It covers blue/green deployments with traffic shifting modes (all at once, canary, linear) and rolling deployments, and a "baking period" watched by CloudWatch alarms. The lab does none of the traffic shifting and has no alarms. Its gate runs once, before any traffic, and its only watcher is the person reading status.

Citations