Hedronite · Curriculum · Data · Rust · Tue 2026-11-03

Count the lines you cannot read A deploy report that never loses a log line

Twelve lines in, six readable, six named as damaged, and a report whose totals add up.

Lesson Class: Curriculum · B-Data · rung A
Tested with: Rust 1.98 · serde 1 · serde_json 1
Lab: lab/ · sh check.sh (8 checks) · no network
Source: Designing Data-Intensive Applications ch. 4 · serde and serde_json docs · JSON Lines · CloudWatch Logs filter patterns
Name the Damage
A truncated line, a stray word, a blank line and a bad byte are each reported with a reason.
Count Every Line
Lines in equals readable plus unreadable, asserted in the code.
Say It in the Exit Code
0 clean, 1 some lines unreadable, 2 file not readable.
A report that skips what it cannot read is a report about something else.

B-Data · A · arc: self-healing-edge · langs: rust, go · DOP-4.1, DOP-4.2, CKA-troubleshooting(output-streams)

By Evan Eugene Pincham.

Concept

You have a log file written by a small deploy program that appends one JSON line per event. Each line has four fields: ts (a Unix timestamp as an unsigned integer), event (normally one of candidate_started, promoted, rolled_back), version (a string), and detail (a string). You want a report that answers an operator's question: how many releases rolled back, and why?

A report is a claim about the whole file. If the file holds twelve lines and the report is built from six of them, the report is wrong, and it is wrong in the worst way: it still looks finished. Most of this lesson is about making that failure impossible to miss.

JSON Lines is a text format in which every line is one complete JSON value, the file is UTF-8 (the standard way to store text as bytes, in which a byte sequence can be invalid), and lines end with \n; the format's own description says that a blank line is not a valid line, and that the format suits data "that may be processed one record at a time". The practical effect is that a line is the smallest unit you can read, skip or count on its own, so a file that grows by appending can be read one line at a time. Without the rule you cannot say where one record ends and the next begins. The pitfall to check for is a blank line or two in the middle of a log. Symptom: the report's total is smaller than wc -l of the same file. The cause is a reader that skipped lines it could not parse and never said so.

A schema is the agreed list of fields a record must have and the type of each one; here ts is an unsigned integer and the other three are strings. To deserialize is to turn text into a typed value in memory, and in Rust the serde crate (a crate is a Rust library) does it from a struct definition: you derive Deserialize on a Record struct and call serde_json::from_str::<Record>(text). The serde_json documentation says from_str fails if the structure does not match the target type, if a required field is missing, or if a number is too big for the primitive type. Without a schema, every reader invents its own idea of what a good line is, and two tools will disagree about the same file. The pitfall here is treating every parse failure as the same thing. A line with "ts":"yesterday" is valid JSON of the wrong shape, and a line with half a brace is not JSON at all. These point to different problems, so the report should name them differently.

serde_json gives you the distinction. Error::classify returns one of four categories: Syntax (input that is not syntactically valid JSON), Data (input that is semantically incorrect), Eof (an unexpected end of the input data) and Io. The reader in this lesson maps them to words an operator can use (Category is serde_json::error::Category, and Io becomes io):

A truncated line is a line whose text stops before its JSON value is complete, such as a record cut off in the middle of a string. Naming it separately matters because a truncated line and a stray error message mean different things when you are investigating a log. The symptom you can check is a line that starts like a record, {"ts":..., and ends mid-word, with a parser message that begins EOF while parsing.

One detail about the error positions: Error::line and Error::column are one-based, but they are relative to the text you handed to from_str. Since we parse one line at a time, the line is always 1. The file's line number has to come from your own counter.

Forward compatibility means older code can read data written by newer code. Martin Kleppmann (Designing Data-Intensive Applications, ch. 4, introduction) says it "can be trickier, because it requires older code to ignore additions made by a newer version of the code". A writer and a reader are rarely upgraded at the same moment, so the day the writer adds a field or a new event name, the old reader either keeps working or reports a wall of damage. In serde, the default helps you: for self-describing formats like JSON, unknown fields are ignored. The attribute #[serde(deny_unknown_fields)] is the strict opposite, erroring on any unknown field, and using it trades forward compatibility away. For event names, the reader here keeps event as a plain String and matches it with a _ arm that counts anything unrecognized as an unknown event. It does not use #[serde(other)], because the serde documentation describes that attribute for internally or adjacently tagged enums, which is not what this record is. The pitfall, as a symptom: after the writer is upgraded, the old reader reports nearly every line unreadable. The cause is a strict schema (deny_unknown_fields) or a closed enum of event names that rejects anything new.

An exit code is the small integer a process hands back when it ends; by convention 0 means success and anything else means something needs attention. Scripts and CI jobs read it without parsing your text, so it is the cheapest honest channel you have. This reader uses 0 when every line is readable, 1 when at least one line is not, and 2 when the file cannot be opened. The pitfall: you run the report on a damaged file, run echo $?, and see 0. A report that prints a number but exits 0 on damage tells automation that everything is fine.

The last term is a trap in the standard library rather than in the format. BufRead::lines yields io::Result<String> items, so a line that is not valid UTF-8 arrives as an Err (verified by running it: InvalidData, "stream did not contain valid UTF-8"). If your code writes .filter_map(|l| l.ok()), that line is dropped, and the next .filter_map(|l| serde_json::from_str(&l).ok()) drops every line that does not parse. A loop that applies ? to each item behaves differently: it ends at the first Err. The symptom of either version is a count that is short after one odd line, or a report that stops early. BufRead::split(b'\n') yields io::Result<Vec<u8>> with the delimiter removed, so you receive raw bytes for every line, decide yourself whether they are UTF-8, and count the ones that are not.

The fix, in one rule: put every physical line in exactly one bucket, readable or unreadable. Print both bucket sizes next to the total. Assert in code that lines == readable + unreadable. Name each unreadable line by its number and kind. Carry the result in the exit code.

The security angle. A line you cannot parse is evidence, not noise. A log that was truncated, partially overwritten or tampered with will produce unreadable lines, and if the reader drops them quietly, the gap leaves no trace. An audit log is only as trustworthy as its gaps are visible, so the count of unreadable lines and their positions belong in the report.

The operations angle. The operator's question is "how many releases rolled back, and why?" The reader answers it by grouping rollback events by the part of detail before the first ; and counting each group. The groups live in a BTreeMap, which iterates in sorted key order, so the report comes out in the same order every run and two reports can be compared with diff.

Worked example

Start with the reader that lies. It is the starting point of the lab below, and this is its core:

let records: Vec<Record> = BufReader::new(file)
    .lines()
    .filter_map(|l| l.ok())
    .filter_map(|l| serde_json::from_str(&l).ok())
    .collect();

On a clean six-line log it works. The test file for this lesson, damaged.log, has twelve physical lines. Six are readable. Two of those six are from a hypothetical newer writer: line 9 carries an extra key host, and line 10 has a new event name, paused. The other six lines are damaged, each in a different way:

LineWhat it isKind
3a record cut off mid-writetruncated
4a stray panic messagenot json
5an empty lineblank
6valid JSON, detail key missingschema
7ts is the string "yesterday"schema
8bytes that are not UTF-8not utf-8

The starter's report for that file:

deploy-report: damaged.log
lines: 6
promoted: 1
rolled_back: 2
rollback reasons:
  1  candidate exited with exit status: 78 before ready
  1  not ready after 1500 ms

It says lines: 6 for a twelve-line file and it exits 0. Nothing in that output hints at the six missing lines.

The reference reader replaces the whole pipeline with a function that turns one raw line into either a record or a named reason:

fn read_line(bytes: &[u8]) -> Result<Record, (&'static str, String)> {
    let text = std::str::from_utf8(bytes).map_err(|e| ("not utf-8", e.to_string()))?;
    if text.trim().is_empty() {
        return Err(("blank", "empty line".to_string()));
    }
    serde_json::from_str::<Record>(text).map_err(|e| {
        let kind = match e.classify() {
            Category::Eof => "truncated",
            Category::Syntax => "not json",
            Category::Data => "schema",
            Category::Io => "io",
        };
        (kind, e.to_string())
    })
}

UTF-8 is checked first, on bytes, so that case gets its own kind instead of vanishing. A blank line is checked before parsing so it is reported as blank. Only then does the parser run, and its category picks the kind.

The record struct is the schema. It has no deny_unknown_fields, which is what keeps line 9 readable:

#[derive(Debug, Deserialize)]
#[allow(dead_code)]
struct Record {
    ts: u64,
    event: String,
    version: String,
    detail: String,
}

The loop reads raw bytes with split, counts every line, and puts it in one bucket or the other:

for (i, raw) in input.split(b'\n').enumerate() {
    let raw = raw?;
    r.lines += 1;
    match read_line(&raw) {
        Ok(rec) => {
            r.readable += 1;
            match rec.event.as_str() {
                "candidate_started" => r.started += 1,
                "promoted" => r.promoted += 1,
                "rolled_back" => {
                    r.rolled_back += 1;
                    let reason = rec.detail.split(';').next().unwrap_or("").trim();
                    *r.reasons.entry(reason.to_string()).or_insert(0) += 1;
                }
                _ => r.unknown += 1,
            }
        }
        Err((kind, why)) => r.unreadable.push((i + 1, kind, why)),
    }
}
assert_eq!(
    r.lines,
    r.readable + r.unreadable.len(),
    "every line is counted exactly once"
);

The _ arm is what turns paused into unknown_events: 1 instead of an error. The i + 1 converts the zero-based index to the one-based line number a human expects. The assert_eq! is the invariant from the Concept section, written as code. The last step in main is the exit code: 0 if r.unreadable is empty, 1 otherwise, and 2 if the file cannot be opened or read.

The same twelve-line file through this reader:

deploy-report: damaged.log
lines: 12
readable: 6
unreadable: 6
candidate_started: 2
promoted: 1
rolled_back: 2
unknown_events: 1
rollback reasons:
  1  candidate exited with exit status: 78 before ready
  1  not ready after 1500 ms
unreadable lines:
  line 3: truncated: EOF while parsing a string at line 1 column 66
  line 4: not json: expected ident at line 1 column 2
  line 5: blank: empty line
  line 6: schema: missing field `detail` at line 1 column 54
  line 7: schema: invalid type: string "yesterday", expected u64 at line 1 column 17
  line 8: not utf-8: invalid utf-8 sequence of 1 bytes from index 0

Check the arithmetic against the file: the six readable lines are 1, 2, 9, 10, 11 and 12. Two are candidate_started, one is promoted, two are rolled_back, and one is the unknown paused, which adds up to six. Six readable plus six unreadable is twelve. Each damaged line appears with its file line number and the parser's own message. The line 1 column 66 inside the messages is the position within that single line, as noted above. The exit code is 1.

Lab

The exam objectives behind this lab are these. DOP-4.1 and DOP-4.2 come from the AWS Certified DevOps Engineer - Professional exam guide (DOP-C02, version 1.6), Domain 4. DOP-4.1 asks you to "Configure the collection, aggregation, and storage of logs and metrics". DOP-4.2 asks you to "Audit, monitor, and analyze logs and metrics to detect issues". The Troubleshooting item in the Certified Kubernetes Administrator (CKA) curriculum v1.37 from the Cloud Native Computing Foundation asks you to "Manage and evaluate container output streams". That item is a bullet in a curriculum list, not a task-level blueprint. A report that accounts for every line of a log is one small piece of each.

This is a simulated lab. It needs no cloud account and costs nothing.

You start from lab/starter/, which holds the lying reader shown above. You edit src/main.rs, and check.sh builds your reader, builds two log files, runs your reader on each and prints one RESULT line per check. It runs without network access. The lab container image has Rust 1.98 with the crates prefetched. On your own machine, run cargo fetch first so cargo build --offline can find serde and serde_json.

sh check.sh

Run it once before you change anything. The starter passes build, clean_log_counts and rollback_reasons, and fails the other five. The comment block at the top of the starter lists the behavior to add. As steps, in this order:

  1. Replace .lines() with .split(b'\n') so each line arrives as bytes.
  2. Write a function that turns one line's bytes into a Record or a (kind, message) pair, checking UTF-8 first, then blank, then calling from_str and mapping classify() to a kind.
  3. Keep a counter for total lines, one for readable lines, and a vector of (line number, kind, message) for unreadable ones. Number lines from 1.
  4. Match event as a String with a _ arm that counts unknown events. Leave deny_unknown_fields out.
  5. Print lines:, readable:, unreadable:, unknown_events: and an unreadable lines: section whose entries look like line N: <kind>: <why>.
  6. Add the assertion that lines equals readable plus the number of unreadable lines.
  7. Return ExitCode from main: 0 when nothing is unreadable, 1 otherwise, and 2 if File::open fails. The starter calls unwrap() there, which panics with exit code 101.

When your reader is right, the output is these eight lines, in this order:

RESULT build PASS
RESULT clean_log_counts PASS
RESULT rollback_reasons PASS
RESULT damaged_totals PASS
RESULT totals_add_up PASS
RESULT damaged_lines_named PASS
RESULT newer_writer_stays_readable PASS
RESULT missing_file_exit_2 PASS

When it is done, try one experiment. Add #[serde(deny_unknown_fields)] to Record. Before you run check.sh, predict which line of the damaged file changes bucket and which checks should fail. Then run it and compare. Remove the attribute afterwards.

Here is what the reference reader printed with the attribute added, on the same twelve-line file. Only the lines that changed are shown; the full run is in the lab folder as deny-unknown-fields.run.log:

damaged log: exit=1 lines=12 readable=5 unreadable=7
RESULT damaged_totals FAIL
RESULT totals_add_up PASS
  line 9: schema: unknown field `host`, expected one of `ts`, `event`, `version`, `detail` at line 1 column 133
RESULT newer_writer_stays_readable FAIL

Line 9, the record from the newer writer, moved from readable to unreadable, and rolled_back dropped from 2 to 1. The report is still honest: totals_add_up passes, because the line is counted and named. What failed is the policy. A strict reader reports a harmless addition as damage, which is the third pitfall above.

Check yourself

Question 1
A report says lines: 6 and wc -l says 12. What is the likely cause?
Click to reveal
A reader dropped the lines it could not parse without counting them.
Question 2
Why does the reference use split(b'\n') instead of lines()?
Click to reveal
lines() yields an Err for a non-UTF-8 line, and a discarded Result loses that line; split gives raw bytes, so the line can be counted and named.
Question 3
Why is the line with an extra host key counted as readable?
Click to reveal
serde ignores unknown fields on JSON by default, which is what lets an older reader accept a newer writer's additions.
Question 4
What do the three exit codes mean?
Click to reveal
0: every line readable. 1: at least one line unreadable. 2: the file could not be opened (or read).
Question 5
What does Error::line return in this program, and where does the file's line number come from?
Click to reveal
1 every time, because each line is parsed separately; the file line number comes from the loop counter.

Other paths

Go. The same program in Go changes shape in three ways. There is no derive: you declare a struct and attach struct tags such as ` json:"ts" to map keys to fields. Errors are values you check at each call rather than variants you match on. And the strictness switch is a method on the decoder, Decoder.DisallowUnknownFields, which "causes the Decoder to return an error when the destination is a struct and the input contains object keys which do not match any non-ignored, exported fields in the destination." That is the same trade as deny_unknown_fields` in serde, and the same warning applies.

The trap in Go is in bufio.Scanner. Its default maximum token size is MaxScanTokenSize, 64 KiB, unless you supply a larger buffer with Scanner.Buffer. A longer line ends the scan with the error bufio.ErrTooLong (the package lists it among the errors returned by Scanner), and the documentation says scanning "stops unrecoverably" at EOF, the first I/O error, or a token too large to fit. You must check Scanner.Err() after the loop. If you do not, the report ends at the long line and still looks complete, which is the lesson's failure again. This is a sketch and was not run in this lab:

// Sketch only. Not run in this lab.
type Record struct {
	TS      uint64 `json:"ts"`
	Event   string `json:"event"`
	Version string `json:"version"`
	Detail  string `json:"detail"`
}

sc := bufio.NewScanner(f)
sc.Buffer(make([]byte, 0, 64*1024), 1<<20) // allow lines above the 64 KiB default
n := 0
for sc.Scan() {
	n++
	line := sc.Bytes()
	if len(bytes.TrimSpace(line)) == 0 {
		// unreadable: blank, line n
		continue
	}
	dec := json.NewDecoder(bytes.NewReader(line))
	dec.DisallowUnknownFields() // strict; omit it to stay forward compatible
	var rec Record
	if err := dec.Decode(&rec); err != nil {
		// unreadable: line n, err.Error()
		continue
	}
	// readable: count rec.Event, group rollback reasons
}
if err := sc.Err(); err != nil {
	// ErrTooLong lands here: the scan stopped, and the lines after
	// this point were never counted. Say so in the report and exit non-zero.
}

CloudWatch Logs. If these lines were shipped to Amazon CloudWatch Logs (AWS's service for storing and searching logs), you could ask similar questions with filter patterns, which the exam objectives above cover. The User Guide's filter-pattern page says that "Filter patterns only return the log events that contain the terms you define", and that JSON patterns look like { $.eventType = "UpdateTrail" }. Read that sentence as a limit as well as a capability: a filter shows you events that match. If you ask for events with $.event = "rolled_back", a truncated line is not in the result, and nothing in the result tells you it was skipped. The same page documents { $.SomeOtherObject NOT EXISTS } as the way to return JSON events that do not contain a field. Hypothetically, { $.detail NOT EXISTS } would find events missing detail, the equivalent of line 6 in the fixture. This mapping rests on the quoted capability only. It was not run here, and it does not replace counting every physical line in the file.

Citations