Hedronite · Curriculum · Sec · Rust · Wed 2026-11-04

Show where the log was edited A deploy log that does not trust its own file

Six lines in, one edited, one deleted, one forged, and a head stored elsewhere for the damage the chain cannot show.

Lesson Class: Curriculum · B-Sec · rung A
Tested with: Rust 1.98 · sha2 0.11 · hmac 0.13 · serde_json 1
Lab: lab/ · sh check.sh (12 checks) · no network, crates vendored
Source: RFC 2104 · RFC 8785 · NIST SP 800-53 Rev. 5 AU-9 · hmac and sha2 crate docs · CloudTrail log file integrity · OWASP LLM01
Seal with a Key
Each line carries the previous line's tag, and the tag needs the key to compute.
Count Every Line
Lines in equals verified plus broken plus unreadable, asserted in the code.
Name What the Chain Cannot See
A cut tail needs a head stored elsewhere; a key holder's rewrite needs the same.
A log that the writer can rewrite is a claim; a log the writer cannot rewrite unseen is evidence.

B-Sec · A · arc: self-healing-edge · langs: rust, go · DOP-6.3, DOP-6.2

By Evan Eugene Pincham.

Concept

A log written by the program it describes is a claim, not evidence. Suppose a small deploy program appends one JSON line per event to a file. Each line has the fields ts (a Unix timestamp), event (candidate_started, promoted or rolled_back), version and detail. A report that counts those lines will believe whatever the file says. Take the six-line log used below, where line 2 is a promotion and line 4 is a rollback of v3. If someone changes line 4 from rolled_back to promoted, the line is still valid JSON with the right fields. The report now shows two promotions where there was one, and nothing in it hints that the history was edited. Line counts and parse checks cannot tell an honest rollback from one rewritten as a promotion.

This lesson builds a verifier in Rust that makes edits visible. You need the following terms, each defined where it first appears.

JSON Lines is a text format in which every line of a file is one complete JSON value, so a program can append an event with a single write and a reader can parse each line on its own. It matters because one damaged line costs you one line, not the whole file. The pitfall is a reader that silently skips lines it cannot parse. You can check for it by counting: if a six-line file reports lines=4, the reader dropped two lines before counting them.

SHA-256 is a hash function: it turns any input into a fixed 32-byte digest, and changing any bit of the input changes the digest unpredictably. Without a hash, "this line is unchanged" would mean comparing the whole line to a stored copy.

A hash chain gives each line a field, here prev, holding the tag of the line before it, and the first line points at 64 zeros. Deleting, inserting or reordering a line leaves the next line's prev pointing at something that is no longer its neighbor, so the link breaks at the hole. The pitfall is a chain with no secret in it. Anyone who edits a line can recompute every tag after it, and the file looks perfect.

A MAC (message authentication code) is a short tag computed from a message and a secret key, such that producing a correct tag for new content requires the key. HMAC is the standard way to build one from a hash. RFC 2104 section 2 defines it as H(K XOR opad, H(K XOR ipad, text)), and here H is SHA-256. With HMAC, a line cannot be given a correct tag without the key, so someone who edits the file with a text editor can no longer recompute the tags that follow the edit.

A key is the secret byte string that the writer and the verifier both hold. RFC 2104 section 3 says a key shorter than the hash output is strongly discouraged, that keys need to be chosen at random, and that they should be periodically refreshed. Section 6 says the mechanism depends on good secrecy protection of keys. The symptom of the wrong key is distinctive: every line reports a mac break, starting at line 1, even though nobody touched the file (the wrong key row in the worked example shows it). Before you conclude a log was tampered with, check a file you know is good with the same key.

Canonical bytes are the exact byte string that gets fed to the MAC for a line. The writer and the verifier must produce identical bytes from identical content, or good lines will fail. Plain concatenation is ambiguous. As an illustration, the made-up pair event="promoted", version="v2" and event="promote", version="dv2" both concatenate to promotedv2, so a MAC over concatenated text cannot tell them apart. The lab avoids this by serializing one Rust struct, Body, with serde_json::to_vec. The fields come out in the struct's declaration order with no whitespace, and the lab's repeatable-seal check confirms that two runs give the same bytes. That works because one Rust program both writes and checks. If a second writer in another language ever appends lines, you need an agreed byte rule or a general scheme. RFC 8785, JSON Canonicalization Scheme (JCS), is that scheme. Its section 1 says hashing and signing need data in an invariant format so that operations are reliably repeatable, and section 3.2.3 requires object properties to be sorted. The lab does not implement JCS.

Constant-time comparison means checking two tags in a time that does not depend on how many leading bytes match. A naive == is free to stop at the first differing byte, which is the timing difference a constant-time check exists to remove; the lab does not measure timing, so this is the reason for the choice and not something the checker demonstrates. In the hmac crate, verify_slice does this check and returns an error if the tag is not valid or not the right length. The pitfall is comparing hex strings with == where a secret-derived tag meets a value someone else supplied. You can find it by searching your code for the place where the stored tag is compared with the recomputed one. The reference code's other comparison, prev against the previous line's tag, uses plain !=, because both values are stored in the file in the clear, so there is no secret for a timing difference to leak.

Tamper-evident means that a change can be detected afterward. Tamper-proof means a change cannot be made at all. A file that someone with write access can open in an editor is never tamper-proof. The best it can be is tamper-evident, and what it is evident against depends on what the attacker lacks.

The head is the mac of the last line. Because every line's tag covers the previous tag, the head commits to the whole history. This matters because a chain only binds a line to the lines before it. Cut lines off the end and what remains is a perfectly valid shorter chain, so the head has to be recorded somewhere the log's writer cannot reach. NIST SP 800-53 Rev. 5 control AU-9 asks you to protect audit information from unauthorized access, modification and deletion. Its enhancement AU-9(2), "Store on Separate Physical Systems or Components", says separate storage helps ensure that a compromise of the audited system does not also compromise the audit records.

An exit code is the small integer a process returns to the shell when it ends. By convention 0 means success, and scripts and CI jobs usually look at nothing else. The verifier uses 0 only when every line is verified (and the head matches, when one is given), 1 for any problem, and 2 when it cannot run at all because the file cannot be opened or the key is missing or under 32 bytes.

The verifier puts every line in exactly one of three buckets. Verified means the MAC matches and prev matches the line before. Broken means the line is readable but wrong: kind mac when the tag does not match the content, kind link when the tag is right but prev is wrong. Unreadable means the line is not a sealed record at all, with the kinds truncated, not json, blank, schema and not utf-8. The totals invariant says lines = verified + broken + unreadable, and the code asserts it so a line cannot be lost between buckets. After a problem the verifier resyncs: it continues from what the line claims, so one deleted line is one link break and not a cascade of failures down the file.

Four symptoms to know

  1. You edit one line and verify still exits 0. The verifier checks links but never recomputes the MAC. A link check only sees that line N points at line N-1, and an edit inside line N changes neither.
  2. The report says lines=4 for a six-line file. Lines that failed to parse were skipped before counting. The row numbers in the problem list also stop matching the file, because they count only the lines that survived.
  3. You delete the last two lines and the exit code is still 0. A chain binds a line to the ones before it, not after. Check the head against a copy stored elsewhere.
  4. verify succeeds after someone with the key rewrote history. The file proves nothing about who held the key. Only the externally stored head shows the rewrite.

Worked example

The sealer reads the plain log, which is what the deployer wrote. One line looks like this:

{"ts":1793632142,"event":"promoted","version":"v2","detail":"replaced v1 on port 18091"}

It writes the same line with two more fields, prev and mac. The bytes the MAC covers come from this struct:

/// The bytes the MAC covers. Field order is fixed by this struct, and serde_json
/// writes no whitespace, so one Rust program always produces the same bytes.
#[derive(Serialize)]
struct Body<'a> {
    ts: u64,
    event: &'a str,
    version: &'a str,
    detail: &'a str,
    prev: &'a str,
}

prev is inside the covered bytes, so the link is authenticated too. Nobody can repoint a line at a different predecessor without breaking its MAC. The core of seal builds the line with an empty mac, tags the body, stores the tag and carries it forward:

let mut s = Sealed { ts: p.ts, event: p.event, version: p.version, detail: p.detail, prev: prev.clone(), mac: String::new() };
let mut m = keyed(key);
m.update(&body_bytes(&s));
s.mac = to_hex(&m.finalize().into_bytes());
println!("{}", serde_json::to_string(&s).expect("sealed line serializes"));
prev = s.mac;

mac_ok recomputes the tag from the stored content and compares it with the stored one:

fn mac_ok(key: &[u8], s: &Sealed) -> bool {
    let Some(tag) = from_hex(&s.mac) else { return false };
    let mut m = keyed(key);
    m.update(&body_bytes(s));
    m.verify_slice(&tag).is_ok()
}

verify reads raw bytes split on newlines, counts each line before looking at it, and sorts it into a bucket:

for (i, raw) in input.split(b'\n').enumerate() {
    let raw = raw?;
    let n = i + 1;
    v.lines += 1;
    match read_line(&raw) {
        Err((kind, why)) => {
            // The line is lost, so the next line's `prev` will point at a mac we never saw.
            v.unreadable += 1;
            v.problems.push((n, kind, why));
        }
        Ok(s) => {
            if !mac_ok(key, &s) {
                v.broken += 1;
                v.problems.push((n, "mac", "mac does not match the line's content".to_string()));
            } else if s.prev != expected_prev {
                v.broken += 1;
                v.problems.push((n, "link", "prev does not match the line before it".to_string()));
            } else {
                v.verified += 1;
            }
            // Resync on what this line claims, so one removed line is one link break, not a cascade.
            expected_prev = s.mac.clone();
            v.head = Some(s.mac);
        }
    }
}
assert_eq!(v.lines, v.verified + v.broken + v.unreadable, "every line is counted exactly once");

The MAC is checked before the link, so the two kinds mean different things. mac says the stored tag does not match the content: the line was changed, or written without the key. link says the tag matches but prev is not the tag of the line before it: the line is intact, but what precedes it is not what it was sealed after, which is what a deletion or a reorder produces.

Here is what the reference verifier reports for the six-line log and for each kind of damage. Each run used the throwaway key in the checker script, in a container with no network.

clean sealed log: exit=0 lines=6 verified=6 broken=0 unreadable=0 first_break=none
edited line 4: exit=1 lines=6 verified=5 broken=1 unreadable=0 first_break=line 4 (mac)
deleted line 3: exit=1 lines=5 verified=4 broken=1 unreadable=0 first_break=line 3 (link)
forged line 7: exit=1 lines=7 verified=6 broken=1 unreadable=0 first_break=line 7 (mac)
tail cut, no head: exit=0 lines=4 verified=4
tail cut, with head: exit=1 tail=MISMATCH (expected head 7f56f0dce49cc6f511ff7894fb32d2c23d884ea70d761ba73b69f80abff352dd)
damaged log: exit=1 lines=6 verified=2 broken=2 unreadable=2 first_break=line 2 (truncated)
wrong key: exit=1 lines=6 verified=0 broken=6 unreadable=0 first_break=line 1 (mac)

Reading these:

chainlog verify: damaged.log
lines: 6
verified: 2
broken: 2
unreadable: 2
head: 7f56f0dce49cc6f511ff7894fb32d2c23d884ea70d761ba73b69f80abff352dd
first_break: line 2 (truncated)
problems:
  2  truncated  EOF while parsing a value at line 1 column 60
  3  link  prev does not match the line before it
  5  not json  expected ident at line 1 column 2
  6  link  prev does not match the line before it

The starter's flaw

The lab starts from a verifier that checks links only. Its loop is:

for line in input.lines().filter_map(|l| l.ok()) {
    let Ok(s) = serde_json::from_str::<Sealed>(&line) else { continue };
    v.lines += 1;
    if s.prev != expected_prev {

It never calls mac_ok. It throws away lines it cannot read, and continue runs before v.lines += 1. It also reuses v.lines as the row number in the problem list, which is why those rows count surviving lines and not file lines. In main it ignores --head and loads no key for verify. Here is what it reports for the same files:

edited line 4: exit=0 lines=6 verified=6 broken=0 unreadable=0 first_break=none
forged line 7: exit=0 lines=7 verified=7 broken=0 unreadable=0 first_break=none
tail cut, no head: exit=0 lines=4 verified=4
tail cut, with head: exit=0 tail=
damaged log: exit=1 lines=4 verified=2 broken=2 unreadable=0 first_break=line 2 (link)
wrong key: exit=0 lines=6 verified=6 broken=0 unreadable=0 first_break=none
no key: exit=0; short key: exit=0

The starter calls the edited file fully verified and reports four lines for a six-line file. Look at the damaged-file totals: 2 + 2 + 0 = 4, which equals the lines it printed. The invariant holds, but lines is wrong, so the invariant is necessary and not sufficient. It only checks that the buckets add up to the count, and says nothing about whether the count matches the file. Its report for the damaged file is:

chainlog verify: damaged.log
lines: 4
verified: 2
broken: 2
unreadable: 0
head: 7f56f0dce49cc6f511ff7894fb32d2c23d884ea70d761ba73b69f80abff352dd
first_break: line 2 (link)
problems:
  2  link  prev does not match the line before it
  4  link  prev does not match the line before it

The rows say lines 2 and 4, but the file's damage is at lines 2, 3, 5 and 6. Those numbers count the surviving lines, not the lines of the file.

What a key holder can still do

Someone who holds the key can edit line 4 and seal the whole file again. A separate script sealed four lines, rewrote line 4 and sealed the file a second time. The head recorded before the rewrite was kept elsewhere:

head recorded elsewhere: 8d4fcdcfad140115e80c97b15da3e1c18d89e59e04c58ea4b7d65d0a57968086
--- key holder rewrites line 4 and re-seals, no head given
exit=0
lines: 4
verified: 4
broken: 0
unreadable: 0
head: aef8111edba92b55e8e182ccde85f3c36ef8ceb7f9372f22444d6be305291237
first_break: none
problems:
--- same file, checked against the head recorded earlier
exit=1
lines: 4
verified: 4
broken: 0
unreadable: 0
head: aef8111edba92b55e8e182ccde85f3c36ef8ceb7f9372f22444d6be305291237
first_break: none
tail: MISMATCH (expected head 8d4fcdcfad140115e80c97b15da3e1c18d89e59e04c58ea4b7d65d0a57968086)

The file on its own verifies cleanly. Only the head stored elsewhere shows that history changed. An HMAC chain is also symmetric, so anyone who can verify can also forge. NIST AU-9(3), "Cryptographic Protection", names "signed hash functions using asymmetric cryptography" for this reason: a public key can verify while only the secret key signs. That is the next step beyond this lesson.

Lab

The exam objectives this touches are AWS Certified DevOps Engineer - Professional (DOP-C02) Domain 6: task statement 6.3, "Implement security monitoring and auditing solutions" (skill: "Configuring service and application logging (for example, CloudTrail, CloudWatch Logs)"), and task statement 6.2, "Apply automation for security controls and data protection" (knowledge: "Data management (for example, data classification, encryption, key management, access controls)"), where the lab touches only the key-management part. The lab runs no AWS service, needs no cloud account and costs nothing. It is a simulation on local files.

You start from lab/starter/, a verifier that checks links only. You edit src/main.rs and run the checker, check.sh, with sh inside the lab container (Rust 1.98, no network). The checker extracts the vendored crates sha2 and hmac and their dependencies from vendor.tgz, builds offline, builds its own six-line fixture, damages it in six ways and prints one RESULT line per check. The key in the checker is a throwaway constant, lab-only-key-0123456789abcdef0123456789abcdef. A real key comes from a secret store and is rotated. RFC 2104 section 3 asks for random keys that are periodically refreshed.

Run the checker first. The starter passes six checks and fails six:

RESULT build PASS
RESULT seal_and_verify_clean PASS
RESULT seal_is_repeatable PASS
RESULT edit_caught FAIL
RESULT deletion_caught PASS
RESULT forged_append_caught FAIL
RESULT tail_cut_caught_by_head FAIL
RESULT damaged_totals FAIL
RESULT totals_add_up PASS
RESULT wrong_key_breaks_every_line FAIL
RESULT key_required FAIL
RESULT missing_file_exit_2 PASS

Then change the starter until all twelve pass. The helpers you need, read_line and mac_ok, are already in the file and currently unused. The work is in verify and main:

  1. Make verify take the key and return std::io::Result<Verdict>. Read raw bytes with input.split(b'\n') instead of lines().filter_map(...).
  2. Increment v.lines before you look at the line, so nothing can skip the count.
  3. Call read_line. An Err goes in the unreadable bucket with its kind. An Ok is checked with mac_ok first (kind mac), then against the expected prev (kind link), and otherwise counts as verified.
  4. After every readable line, resync: the next expected prev is this line's stored mac, and the head is that mac too.
  5. Assert the totals invariant at the end.
  6. In main, load the key for verify before opening the file, the way seal does, so a missing or short key exits 2 and prints no report. Use the --head value the starter currently discards: if the final head differs, print tail: MISMATCH (expected head ...) and exit 1.

Run the checker again. When the build passes and no RESULT line says FAIL, you are done. If edit_caught still fails, you are not calling mac_ok. If damaged_totals fails, look at where v.lines is incremented. If key_required fails, check the order of key loading and file opening.

Check yourself

Question 1
Why does the starter report verified=6 for the edited file? It checks only that each prev points at the previous line's stored mac, and an in-place edit changes neither.
Click to reveal
Question 2
The starter's damaged-file totals add up, so why is the report wrong? The invariant only compares the buckets to lines, and lines was undercounted because unreadable lines were skipped first.
Click to reveal
Question 3
Which damage can the chain alone never show, and what shows it? A cut tail, which leaves a valid shorter chain; a head stored somewhere the writer cannot reach, passed as --head.
Click to reveal
Question 4
Why does a key holder's rewrite verify cleanly? Every tag in the rewritten file is correct under the key, so only a head recorded before the rewrite disagrees.
Click to reveal
Question 5
Why use verify_slice instead of comparing the tags yourself? It compares in constant time and rejects tags of the wrong length, so timing does not reveal how many leading bytes were right.
Click to reveal

Other paths

Go. The same tag in Go uses crypto/hmac: hmac.New(sha256.New, key) returns a MAC, you Write the bytes into it, and Sum(nil) gives the tag. For comparing tags, the package documentation says receivers should be careful to use Equal to avoid timing side-channels, and describes it as comparing two MACs "without leaking timing information". This is a sketch, not run in this lab:

// Sketch only. Not compiled or run in this lab.
func tag(key, body []byte) []byte {
	m := hmac.New(sha256.New, key)
	m.Write(body)
	return m.Sum(nil)
}

func macOK(key, body []byte, hexTag string) bool {
	want, err := hex.DecodeString(hexTag)
	if err != nil {
		return false
	}
	return hmac.Equal(tag(key, body), want)
}

What changes is the byte layout. Rust gave you Body and serde_json::to_vec. The sketch above says nothing about the bytes it is given, so you define the canonical bytes yourself, and you should test that two runs produce identical bytes, as the lab's repeatable-seal check does. If a Go writer and a Rust verifier share one log, they need an agreed byte rule, and RFC 8785 is the general answer.

AWS CloudTrail log file integrity validation (read-only mapping, no AWS service used here). The AWS CloudTrail User Guide describes the same idea at service scale: SHA-256 for hashing and SHA-256 with RSA for digital signing. CloudTrail writes digest files hourly. Each is signed and contains the digital signature of the previous digest file if one exists, and digest files are put into a folder separate from the log files. Mapped onto this lesson: the digest chain is the hash chain, the signature replaces the shared-key MAC, and the separate folder plays the role of the externally stored head.

An AI assistant reading the log (hypothetical, nothing here was run). Suppose an assistant were asked "what happened to v3?" It would read detail text written by other processes, which makes unverified lines external content. A safer arrangement is to hand it verified lines only, plus the counts of what was withheld, so it can say "two lines were left out". OWASP's LLM01:2025 describes indirect prompt injection as occurring when an LLM accepts input from external sources, such as websites or files, and lists "Segregate and identify external content" as a mitigation. Any action the assistant proposes, such as a restart or a rollback, should wait on a human or a deterministic check, which matches the mitigation "Require human approval for high-risk actions".

Citations