Hedronite · Ops Lesson · α-Cognition/Synthesis-Lessons · 2026-06-11

Fat-Tail Geometry for Evaluation Distributions — Why Mean Scores Lie, and the Tail-Aware Promotion Gate

Lesson Class: ops
Filed: 2026-06-11
Shelf: α-Cognition/Synthesis-Lessons
Domains: AI, ML, AIOps, MLOps, QuantOps

<!-- hal:authoritative:yaml -->

§I — Frame

The Adversarial-Markets corpus learned a hard rule early: the mean of a return series is the least informative number in the room. Returns are fat-tailed. The average day says nothing about the day that ends the strategy. Every sizing rule, every kill-switch threshold, every pre-trade gate in that arc is built against the tail, because the tail is where strategies die.

Walk that rule across the pair boundary and look at an evaluation pipeline. The eval lesson built score distributions: a regression set producing hundreds of per-item scores, a canary window producing a moving metric, an adversarial bank producing pass/fail verdicts. Then the pipeline did what eval pipelines do by default. It averaged. One mean accuracy, one mean rubric score, compared against one threshold.

A model's failures are not spread evenly under that mean. They cluster. A candidate that scores 0.91 on average can carry a thin population of catastrophic responses: the refusal that answers, the citation invented with full confidence, the tool call that deletes instead of reads. The mean does not see them. The trader who sizes positions off average daily return and the operator who promotes models off average eval score are making the same mistake in different clothes.

This lesson gives the eval discipline the tail-reading it has been running without.

§II — Foundations

Four objects carry the argument; name them before using them.

The score distribution is the full set of per-item eval scores for one model on one set, kept as a distribution rather than collapsed to a summary. Everything below requires that the pipeline store per-item scores. A pipeline that discards them after averaging has already destroyed the evidence.

The moments describe the distribution's shape. The mean locates it; the variance spreads it; the skewness leans it; the kurtosis weighs its tails. Blitzstein's moments chapter builds these as a sequence, and the fourth one is the one this lesson turns on: kurtosis measures how much of the distribution's mass lives far from the mean. A Gaussian has kurtosis 3. Eval-failure distributions in production routinely run far above it, because failure modes are mechanism-driven, not noise-driven. A model fails catastrophically for a reason, and the reason repeats.

The tail metric reads the distribution where the damage lives. The simplest forms already do most of the work: the worst-decile mean (average score of the worst 10% of items), the p5 and p1 score percentiles, and the count of items below a catastrophe floor. The trading shelf knows the worst-decile mean as conditional value-at-risk; the name changes across the boundary, the arithmetic does not.

The tail-aware gate is a promotion gate that reads tail metrics beside the mean. The continuous-training lesson built the gate that a retrained candidate must clear; this lesson re-arms that gate so a candidate cannot buy a better mean by selling its tail.

§III — Mechanism

Why eval failures are fat-tailed by construction

A model's per-item scores are not independent draws from one quiet process. They are a mixture: a large population of items the model handles, and small populations of items that strike a specific defect. Mixtures of a dominant clean process and rare severe processes produce heavy tails; this is the same mechanism that makes market returns fat-tailed, where quiet regimes mix with crisis regimes. The geometry travels because the generating structure travels.

The practical consequence: tail mass concentrates in interpretable clusters. The worst decile of an eval run is rarely a random sample of the set. It is the model's defect map. Reading it item by item is the cheapest diagnostic in the whole pipeline.

The mean-gate failure, stated precisely

Take two candidates against a 600-item regression set with a promotion threshold of mean ≥ 0.90. Candidate A scores 0.905: tight distribution, worst decile at 0.74, nothing below 0.3. Candidate B scores 0.912: better mean, worst decile at 0.41, nine items below 0.2, and three of the nine are confident fabrications in a category that reaches customers. The mean gate promotes B over A. The tail gate reads B's worst decile and refuses. Every number the tail gate needed already existed in the per-item scores; the mean gate threw them away at the moment of averaging.

The tail-aware gate, concretely

Extend the gate the continuous-training lesson built with three checks beside the mean check. The candidate's worst-decile mean must hold within tolerance of the incumbent's worst-decile mean. The candidate's catastrophe count (items under the floor the operator set deliberately) must not exceed the incumbent's. And any new member of the worst decile that was not in the incumbent's worst decile gets read by a human before promotion, because a migrating tail item is a new defect, not noise.

The thresholds are the operator's Kelly question in eval clothing: how much tail are you willing to carry for how much mean? The trading arc answers it with sizing; the eval arc answers it with promotion. Neither answers it by ignoring the tail.

Kurtosis as the drift alarm before the drift

The canary metric the eval pipeline tracks is a mean over a moving window, and it inherits the mean's blindness. A model beginning to fail on a new traffic cluster shows tail growth before mean decay: kurtosis and catastrophe count move first, while the average still clears threshold. Tracking the canary window's fourth moment beside its first gives the retraining trigger an earlier tripwire. The inference-percentile habit is already normal on the serving side, where latency is managed at p99 because the mean latency of a system that occasionally hangs is a fiction. Score distributions deserve the percentile treatment latencies already get.

§IV — Worked Example

A document-extraction agent scores 0.93 mean accuracy on its 800-item regression set; the operator adds tail metrics to the same run. Worst-decile mean: 0.55. Catastrophe floor (score < 0.25): 11 items. Reading the 11, eight are one defect: documents with two-column layouts where the model interleaves columns and extracts confident garbage. The mean said ship. The tail said: there is a layout class this model cannot read, and it reaches 1.4% of production traffic.

The operator sets the gate: worst-decile mean ≥ 0.60, catastrophe count ≤ incumbent, human read on new tail entrants. The next retrained candidate posts mean 0.935 and worst-decile 0.51; the mean improved, the tail worsened; the gate refuses. The candidate after that, trained with two-column documents added to data assembly, posts mean 0.93 and worst-decile 0.71 with catastrophe count 3. It promotes. The mean never moved; the model got materially better. Only the tail saw it.

§V — Connection to Prior Lessons

The 2026-06-01 eval lesson built the score-producing machinery and three verdict patterns; this lesson changes what the verdicts read, from summaries to distributions. The 2026-06-08 continuous-training lesson built the promotion gate and shadow-eval discipline; the tail-aware gate drops into that shadow comparison without new infrastructure, because shadow evaluation already scores both models on identical inputs. And the Math-Grimoire's Random-Variables lesson carries the original refutation: the Gaussian assumption fails where the consequences concentrate. That refutation crossed from probability theory into market returns; today it crosses into eval scores. One geometry, three rooms.

§VI — Harmonic Ripples

This lesson ripples into QuantOps-Magus (the CVaR arithmetic runs unchanged on eval scores; an operator fluent in one reads the other for free), MLOps-Magus (data-assembly validation inherits the catastrophe-floor pattern for training-set quality), AIOps-Virtual-Machinist (serving-side percentile habits extend to quality percentiles beside latency percentiles), and SecOps-Magus (the adversarial bank is a deliberate tail sampler; its design quality decides whether the tail you measure is the tail that exists).

§VII — Closing

Pull the per-item scores out of your last eval run and plot them. Find the worst decile. Read every item in it. The mean told you whether the model is good on average; the tail tells you what the model does on the day it matters. Gate on both. Examine the worst decile well, down to the last item.

Related