Hedronite · Curriculum · Data · Ops · Tue 2026-11-03

The log schema is a contract Check the file, then ask it questions

Write down what a log line must look like, make a tool refuse anything else, and ask the file how many releases rolled back, and why, without losing a line.

Lesson Class: Curriculum · O-Data · rung A
Tested with: Nickel 1.18.0 · Nushell 0.116.1
Lab: lab/ · sh check.sh (11 checks) · no network
Source: Nickel manual: Contracts · Nushell command reference · JSON Lines · DDIA ch. 4 · CloudWatch Logs filter syntax
Write the Schema Down
One contract says what a log line must contain, and a newer writer's extra keys still pass.
Bucket Every Line
Split the raw bytes, give each line exactly one outcome, and print the bucket sizes beside the total.
Keep Two Copies Honest
The contract and the query hold the same rule twice, so a check makes them agree.
A report that cannot say how many lines it skipped has not said how many it read.

O-Data · A · arc: self-healing-edge · langs: nickel, nushell · DOP-4.2, CKA-troubleshooting(output-streams)

By Evan Eugene Pincham.

Concept

You have been handed a log file from a service deployer and asked one question: how many releases were rolled back, and why? The deployer is a readiness-gated deployer: it starts a candidate version of a service, waits for the candidate to report that it is ready, promotes it if it does, and rolls it back otherwise. Every step it takes is one line in a deploy log. The file you were handed has twelve lines. If a tool answers your question from only six of them and says nothing about the other six, you have an answer that looks finished and is wrong.

The log is JSON Lines: a text file where each line is one JSON value. The JSON Lines site states three requirements: the file is UTF-8, each line is a valid JSON value ("a blank line is not"), and lines end with \n. UTF-8 is the standard way to turn text into bytes. Each line here is a JSON object with four keys: ts (a Unix timestamp, a non-negative integer), and event, version, detail (strings). The events this deployer writes are candidate_started, promoted and rolled_back. A rollback's reason is the part of detail before the first ;.

The report is a claim about the whole file. Suppose a tool reads the file, skips any line it cannot parse, and prints counts. The counts look complete but describe less than the file holds. A truncated line, a stray crash message or a line with bad bytes is information, and for a security reader it is evidence. An audit log is only as trustworthy as its gaps are visible. So the report must put every physical line (the bytes between two newline bytes) into exactly one bucket, print the bucket sizes, and check that lines = readable + unreadable. It must also make the exit code carry the result. An exit code is the small number a program returns to the shell, where 0 means success. This report exits 0 when every line is readable, 1 when any line is not, and 2 when the file cannot be opened.

The schema is the thing an operator can check. Operators rarely own the program that writes a log, but they can write down what a valid line looks like and test lines against it. This lesson writes that rule twice:

  1. As a Nickel contract. Nickel is a configuration language. A contract is a check attached to a value with |. If the value breaks it, evaluation stops with an error message. The Nickel manual says: "Record contracts are fundamental in a configuration language like Nickel. They encode the data schema of a configuration."
  2. As a Nushell query. Nushell is a shell whose pipelines pass structured data (tables, records, lists) instead of plain text. The query reads the raw bytes of the log and produces the same numbers as a Rust version of this report that a companion lesson writes, as one line of JSON.

Open or closed. A record is a set of named fields. A closed record contract rejects any field it does not list. The manual shows the error Contract error: contract broken by a value: extra field bar`. To allow extra fields you append , .. after the last field, which makes the contract *open*. This contract is open. The reason is **forward compatibility**. Kleppmann (*Designing Data-Intensive Applications*, chapter 4) defines *backward compatibility* as newer code reading data written by older code, and *forward compatibility* as older code reading data written by newer code. He notes: "Forward compatibility can be trickier, because it requires older code to ignore additions made by a newer version of the code." A newer deployer might add a host key. An older reader should count that line as readable, not as corrupt. The Rust serde documentation describes the same default. Without #[serde(deny_unknown_fields)]`, "unknown fields are ignored for self-describing formats like JSON", and that attribute is how you opt in to the strict behaviour.

For the same reason, event stays a plain string. If the contract listed the three known event names as the only allowed values, the first new event name would turn a harmless addition into a contract failure. The known names live in a separate list, known_events. The report counts a readable line with an event outside that list as an unknown event. It is still readable.

What the contract can and cannot judge. A contract judges a value that already exists. The bytes of a line must first be valid UTF-8, then non-blank, then valid JSON, and only then is there a record to check. So the contract covers one bucket, schema. The other three damage kinds (not utf-8, blank, not json) are caught earlier, in the query. The five buckets are not utf-8, blank, not json, schema, and readable.

Scope. The query and the contract are two separate files. Making the deployer validate its own output with the contract is a later step. The test file below is built to hold six kinds of damage. A real log will not be so tidy.

Worked example

The worked example is the twelve-line file from the opening of this lesson, called damaged.log, together with a six-line clean.log in which every line is intact. First the line format is stated as a contract, then the query is written, then the query runs on both files. It serves two readers: an operator who wants to know how many releases rolled back and why, and a security reader for whom the six damaged lines are evidence, so each must be counted and named.

The contract

# log-record.ncl: the deploy log's line format as a contract.
# The record is open (the trailing ..), so a newer writer's extra keys still pass.
let Uint = std.contract.from_validator (fun n =>
  if std.is_number n && std.number.is_integer n && n >= 0 then 'Ok
  else 'Error { message = "ts must be a non-negative integer" })
in
{
  LogRecord = {
    ts | Uint,
    event | String,
    version | String,
    detail | String,
    ..
  },
  known_events = ["candidate_started", "promoted", "rolled_back"],
}

A validator is a function that takes a value and returns 'Ok or 'Error { message = ... }. The leading apostrophe marks a Nickel enum tag. std.contract.from_validator turns a validator into a contract. Uint accepts a number that is an integer and at least 0. LogRecord lists the four fields. ts | Uint means "the value of ts must satisfy Uint". The trailing .. makes the record open. The file's value is a record holding two things, the contract and the list of known events.

To see the contract work, here is a fixture of three records:

let { LogRecord, .. } = import "../log-record.ncl" in
[
  { ts = 1793632142, event = "promoted", version = "v2", detail = "replaced v1 on port 18091" },
  { ts = 1793632144, event = "rolled_back", version = "v3", detail = "not ready after 1500 ms", host = "edge-1" },
  { ts = 1793632146, event = "paused", version = "v6", detail = "operator hold" },
] | Array LogRecord

The first line imports the file above and pulls out LogRecord. The second record carries the extra key host. The third has a brand-new event name, paused. | Array LogRecord applies the contract to every element. Running nickel export fixtures/sample.ncl evaluates the file and prints it as JSON, and it only succeeds if every record passes. The lab checker does exactly this and prints:

contract accepted 3 records, including extra key host and new event paused
RESULT contract_accepts_newer_writer PASS

The checker then builds two probe files, each a record that should be refused. The refusals are:

contract refused { ts = 1793632144, }: error: missing definition for `detail`
RESULT contract_rejects_missing_detail PASS
contract refused { ts = "yesterday", detail = "x", }: ts must be a non-negative integer
RESULT contract_rejects_ts_string PASS

The contract is checked by nickel export on files like these. It is not run by the query.

The query

# report.nu: read a deploy log (JSON Lines) and account for EVERY line.
# Usage: nu report.nu <log>   prints one JSON object. Exit 0 all readable, 1 some unreadable, 2 cannot open.

# True when a parsed value has the four required keys with the right types.
# Extra keys are allowed (a newer writer may add them).
def valid-record [r: any] {
  if not (($r | describe) | str starts-with 'record') { return false }
  let cols = ($r | columns)
  if not (['ts' 'event' 'version' 'detail'] | all {|k| $k in $cols }) { return false }
  if ($r.ts | describe) != 'int' { return false }
  if $r.ts < 0 { return false }
  [$r.event $r.version $r.detail] | all {|v| ($v | describe) == 'string' }
}

# Put one raw line (bytes) in exactly one bucket.
def classify [raw: binary] {
  # `decode utf-8` never fails: it swaps bad bytes for U+FFFD and honours a byte-order mark.
  # So decode, encode again, and call the line bad if the bytes did not survive the trip.
  let text = ($raw | decode utf-8)
  if ($text | encode utf-8) != $raw { return { ok: false, kind: 'not utf-8' } }
  if ($text | str trim | is-empty) { return { ok: false, kind: 'blank' } }
  # Plain `from json` is lenient (it accepts bare words and trailing commas); --strict follows the JSON spec.
  let parsed = (try { { value: ($text | from json --strict) } } catch { null })
  if $parsed == null { return { ok: false, kind: 'not json' } }
  if not (valid-record $parsed.value) { return { ok: false, kind: 'schema' } }
  { ok: true, rec: $parsed.value }
}

def main [log: path] {
  if not ($log | path exists) {
    print -e $"report.nu: cannot open ($log)"
    exit 2
  }
  # Split the raw bytes ourselves: `lines` would replace bad bytes and hide the damage.
  let parts = (open --raw $log | into binary | bytes split 0x[0a])
  let parts = (if (($parts | is-not-empty) and (($parts | last | bytes length) == 0)) { $parts | drop } else { $parts })
  let known = (^nickel export --field known_events ($env.FILE_PWD | path join 'log-record.ncl') | from json)
  let rows = ($parts | enumerate | each {|p| (classify $p.item) | insert line ($p.index + 1) })
  let good = ($rows | where ok == true | get rec)
  let bad = ($rows | where ok == false)
  let rolled = ($good | where event == 'rolled_back')
  let report = {
    lines: ($rows | length)
    readable: ($good | length)
    unreadable: ($bad | length)
    events: {
      candidate_started: ($good | where event == 'candidate_started' | length)
      promoted: ($good | where event == 'promoted' | length)
      rolled_back: ($rolled | length)
      unknown_events: ($good | where {|r| $r.event not-in $known } | length)
    }
    reasons: ($rolled | each {|r| { reason: ($r.detail | split row '; ' | first) } } | group-by reason | transpose reason rows | each {|g| { reason: $g.reason, count: ($g.rows | length) } } | sort-by reason)
    unreadable_lines: ($bad | each {|b| { line: $b.line, kind: $b.kind } })
  }
  print ($report | to json --raw)
  if $report.unreadable > 0 { exit 1 }
}

The file has three definitions:

Four Nushell behaviours that hide damage

These are behaviours observed in Nushell 0.116.1 and Nickel 1.18.0. The --strict flags are documented in the Nushell command reference. The rest comes from running the lab's probe script. The script is probe.nu in the lab folder. To run it yourself, start from the lab folder, where it can find reference/fixtures/damaged.log, and run nu probe.nu:

let panic = "thread 'main' panicked at src/deployer.rs:41:9:"
print $"default from json on the panic line: (($panic | from json) | describe)"
print $"strict from json on the panic line:  (try { $panic | from json --strict | describe } catch { 'error' })"
let n = (open --raw reference/fixtures/damaged.log | lines | length)
let s = (try { open --raw reference/fixtures/damaged.log | lines --strict | length | into string } catch {|e| $e.msg })
print $"lines on damaged.log: ($n) lines; lines --strict: ($s)"
let bad = (open --raw reference/fixtures/damaged.log | into binary | bytes split 0x[0a] | get 7)
let back = ($bad | decode utf-8 | encode utf-8)
print $"line 8 is ($bad | bytes length) bytes; decode utf-8 never errors; re-encoded bytes match the original: ($back == $bad)"

Its output:

0.116.1
nickel-lang-cli nickel 1.18.0 (rev e6515c4)
default from json on the panic line: string
strict from json on the panic line:  error
lines on damaged.log: 12 lines; lines --strict: Non-UTF8 string
line 8 is 11 bytes; decode utf-8 never errors; re-encoded bytes match the original: false
  1. Symptom: you split with lines, get 12 lines for the 12-line file, and see no complaint, yet line 8 contains bytes that are not text. Switching to lines --strict gives a single Non-UTF8 string error for the whole stream and does not say which line is at fault. Cause: plain lines replaces bad bytes silently, and the strict form fails the stream on the first bad byte instead of naming the line. The lab splits the raw bytes itself with bytes split 0x[0a].
  2. Symptom: you decode line 8 with decode utf-8, no error appears, and the text looks fine. Cause: decode utf-8 never errors. The probe shows that decoding and then re-encoding line 8 gives different bytes from the original, so the lab compares the two and calls the line not utf-8 on a mismatch.
  3. Symptom: the stray panic message parses as JSON and has a type, string. Cause: plain from json is lenient. from json --strict follows the JSON specification, and the probe shows it raising an error on the same text.
  4. Symptom: a report on the 12-line file says it has fewer lines, no unreadable ones, and exits 0. Cause: a pipeline of the shape try { ... } catch { null } followed by dropping the nulls. The failures become nothing, and nothing cannot be counted. The lab's classify also uses try ... catch { null }, but it tests the result and puts the line in the not json bucket instead of dropping it.

Running the query on the damaged file

The damaged file has twelve physical lines. Lines 1, 2 and 9 to 12 are readable. One of the readable lines has an extra key host, and another has a new event name paused. Lines 3 to 8 are damaged in six different ways: line 3 is cut off mid-write, line 4 is a stray panic message, line 5 is blank, line 6 has no detail, line 7 has ts as the string "yesterday", and line 8 holds bytes that are not UTF-8.

The query's output on that file, as the lab checker prints it (observed in the lab container, networking off). It is one long line:

damaged.log -> exit=1 {"lines":12,"readable":6,"unreadable":6,"events":{"candidate_started":2,"promoted":1,"rolled_back":2,"unknown_events":1},"reasons":[{"reason":"candidate exited with exit status: 78 before ready","count":1},{"reason":"not ready after 1500 ms","count":1}],"unreadable_lines":[{"line":3,"kind":"not json"},{"line":4,"kind":"not json"},{"line":5,"kind":"blank"},{"line":6,"kind":"schema"},{"line":7,"kind":"schema"},{"line":8,"kind":"not utf-8"}]}

Read it against the operator's question. Twelve lines are accounted for (6 + 6). Two rollbacks have two different reasons. One readable line carries an event name the reader does not know, and it is counted, not rejected. Every damaged line is named with its line number and kind. On the clean log (six lines the real gate produced) the same query gives lines=6, three candidate_started, one promoted, two rolled_back, no unreadable lines, and exit 0.

The two copies of one rule

In words, the rule is: a record has ts as a non-negative integer, event, version and detail as strings, and any other keys are allowed. The Nickel contract states it once and valid-record states it again. Nothing in the language links them. The only thing that stops them drifting apart is the lab checker, which runs seven records through both and compares the verdicts. The contract's verdict comes from nickel export on a one-record file. The query's verdict comes from calling valid-record directly.

agree plain: contract=true query=true
agree extra_key: contract=true query=true
agree new_event: contract=true query=true
agree no_detail: contract=false query=false
agree ts_string: contract=false query=false
agree ts_negative: contract=false query=false
agree ts_fraction: contract=false query=false
RESULT query_agrees_with_contract PASS

Seven records is a sample, not a proof, and the sample has a gap. The fractional probe uses 1.5, which both copies refuse, but a ts written as 1.0 is the same number as 1 to Nickel, so the contract accepts it, but Nushell's strict JSON parser reads 1.0 as a float, so valid-record refuses it. The two copies disagree on that record. This is the output of ts-float.sh, run in the lab container:

$ sh ts-float.sh
contract says: true
query says: false

The checker does not test this record, and the lab does not fix it. Which verdict is right depends on the writer: a deployer that writes 1.0 for a timestamp is writing it unusually, but the log would still be readable. Stretch (c) below asks you to decide and make the two copies agree. Hypothetical: if someone added host | String as a required field to the contract and forgot valid-record, the plain record would be refused by the contract and accepted by the query, and the checker's agree line for plain would show it.

What this query cannot do

Lab

You get a starter tree and a checker, check.sh, which runs eleven checks and prints one result line for each, made of the word RESULT, the check's name, and PASS or FAIL. The tree holds the same fixtures as the reference: fixtures/clean.log, fixtures/damaged.log and fixtures/sample.ncl. You need nickel and nu on your path. Run the checker with sh check.sh from the lab folder, and close the worked example before you start editing.

Step 1: see the starter fail. The starter contract gives every field a contract that always passes and makes detail optional, so it accepts any record. The starter query reads the file with lines, parses each line leniently inside try ... catch { null }, keeps whatever is a record, and always exits 0 for a file that exists. The checker shows it:

contract accepted { ts = 1793632144, }
RESULT contract_rejects_missing_detail FAIL
contract accepted { ts = "yesterday", detail = "x", }
RESULT contract_rejects_ts_string FAIL
damaged.log -> exit=0 {"lines":8,"readable":8,"unreadable":0,"events":{"candidate_started":2,"promoted":2,"rolled_back":3,"unknown_events":1},"reasons":[{"reason":"candidate exited with exit status: 78 before ready","count":1},{"reason":"not ready after 1500 ms","count":1},{"reason":"unknown","count":1}],"unreadable_lines":[]}
RESULT query_damaged_totals FAIL

The file has twelve lines. The starter reports eight, says none are unreadable, and exits 0. It also counts the two schema-damaged lines as real events: promoted is 2 instead of 1, rolled_back is 3 instead of 2, and a third rollback reason, unknown, appears. Dropping lines made the report say less, and keeping bad ones made it say wrong things.

Step 2: three tasks.

  1. In log-record.ncl, give ts a contract that accepts only non-negative integers, and make detail required with a String contract. event and version get String too. Keep the record open. Check with nickel export fixtures/sample.ncl, then with the checker's probes.
  2. In report.nu, split the raw bytes with bytes split, classify each line into exactly one bucket, count each bucket, and exit 1 when any line is unreadable. Use the round trip (decode, encode, compare) to catch non-UTF-8 lines.
  3. Make valid-record check the four keys and their types, and make the parse use from json --strict.

Step 3: pass all eleven checks. The goal is that sh check.sh prints PASS for every check. When query_agrees_with_contract fails, look at which of the seven agree lines disagrees. The two copies of the rule differ on exactly that record.

When all eleven checks pass, the result lines read:

RESULT contract_accepts_newer_writer PASS
RESULT contract_rejects_missing_detail PASS
RESULT contract_rejects_ts_string PASS
RESULT query_clean_log PASS
RESULT query_damaged_totals PASS
RESULT totals_add_up PASS
RESULT damaged_lines_named PASS
RESULT rollback_reasons_match PASS
RESULT newer_writer_stays_readable PASS
RESULT query_agrees_with_contract PASS
RESULT missing_file_exit_2 PASS

contract_accepts_newer_writer already passes on the starter, because the starter contract accepts everything. It is there to stop you from fixing the other checks by making the contract too strict.

Step 4: stretch.

Other paths

Rust with serde_json. The same task can be written in Rust. Rust's BufRead::lines yields io::Result<String>, and a non-UTF-8 line arrives as Err(InvalidData, "stream did not contain valid UTF-8") (observed by running a Rust version of this report). A filter_map(|l| l.ok()) therefore drops it, which is the same flaw as the Nushell starter. BufRead::split returns raw bytes without the delimiter, which is the Rust analogue of bytes split. For parsing, serde_json::from_str fails when the structure does not match the target type or a required field is missing. Error::classify separates Eof (truncated) from Syntax (stray text) and Data ("input data that is semantically incorrect", the schema bucket here). That is the distinction the Nushell query lacks. The reference unknown-event handling uses a _ arm on a String, not #[serde(other)], because the serde documentation describes other only for internally or adjacently tagged enums.

Go (sketch, not run). A Go reader would typically use bufio.Scanner. The package documentation says the default maximum token is MaxScanTokenSize = 64 * 1024 unless you call Scanner.Buffer. It also says "Scanning stops unrecoverably at EOF, the first I/O error, or a token too large to fit in the Scanner.Buffer", with ErrTooLong as the error. A loop that never checks Scanner.Err() would stop at the first over-long line and report on a prefix of the file while looking complete. This is the same failure as the Nushell starter, reached a different way. ScanLines treats "one optional carriage return followed by one mandatory newline" as the line end. In encoding/json, Decoder.DisallowUnknownFields makes the decoder error on keys that match no field of the destination struct. That is the closed-record choice, and the opposite of this lesson's open contract.

jq. Two commands were run on the damaged file with jq 1.7.1, on the lab host (jq is not in the lab image). The first reads each line as text (-R) and parses it with fromjson?, then counts what came out. The second counts the lines without parsing:

$ jq -R "fromjson?" damaged.log | jq -s length
8
$ jq -Rn "[inputs] | length" damaged.log
12

The ? after fromjson suppresses errors. It dropped four unreadable lines (3, 4, 5 and 8) and kept lines 6 and 7, which parse as JSON and only fail the schema. Counting raw lines gives the true 12. The Nushell starter also reports 8 and keeps the same eight lines, because it applies the same rule: drop what fails to parse, keep what parses.

AWS CloudWatch. The CloudWatch Logs User Guide says "Filter patterns only return the log events that contain the terms you define". JSON patterns look like { $.eventType = "UpdateTrail" }, and { $.SomeOtherObject NOT EXISTS } returns JSON logs that do not contain a field. A filter is a query that returns matches. Events that match nothing are not returned, so a gap in your data is invisible unless you also ask for a total and compare. The DevOps Professional exam guide places log collection and analysis under Domain 4, "Monitoring and Logging" (Task Statements 4.1 and 4.2). This lesson is a local exercise and does not use AWS.

Kubernetes. The CKA curriculum lists "Manage and evaluate container output streams" under Troubleshooting. A container's output can mix structured log lines with stray text such as a crash message, so the same accounting question applies there: how many lines did you read, and how many could you not? (Hypothetical: a deployer running in a cluster and logging to its container output.)

Citations