🜃 Earth · DevOps Wiring · L1 Probability · L2 Statistics Cube

Availability and Compound Failure

TomesBlitzstein & Hwang Ch 4–5 (wiring) · Beyer, Jones, Petoff, Murphy Site Reliability Engineering (operative) Concept-ClassReliability arithmetic — series, parallel, the bottleneck StationVirtual Machinist → Magus (Earth-domain threshold) StageAlbedo · Citrinitas RipplesEarth (primary) · Air · Fire · Water · Aether
The foundation that holds is the foundation whose failure modes have been quantified. Hope is not a load-bearing material.

Intuition

Availability is not a property a system has — it is a property that emerges from the reliability of its components and the topology that joins them. Two servers in series are less available than either alone (both must work). Two servers in parallel are more available than either alone (one suffices). The mathematics of compound failure is the arithmetic by which an Earth-tier practitioner reasons about whether a foundation can bear its load.

Every claim of "five nines" or "99.9% uptime" is a probabilistic claim. The discipline is reading it as such. A 99% available server has roughly 3.65 days of downtime per year. Three such servers in series — say load-balancer, app, database, where any one going down takes the system down — yields combined availability $0.99 \cdot 0.99 \cdot 0.99 \approx 0.970$, or about eleven days of annual downtime. The series topology converts component-reliability into compound-fragility.

The Earth-tier shift in perception: stop treating "is the system available" as a yes/no question. Treat it as a continuous quantity that decomposes by topology. Once that becomes reflex, capacity planning, redundancy decisions, and SLO negotiations all become tractable. The wiring beneath this is the probability of independent events — L1 Probability, Blitzstein chapters 4 and 5.

The Formula

Availability of a single component
A  =  MTBF MTBF + MTTR

Where MTBF is the mean time between failures (how long the component runs healthy on average) and MTTR is the mean time to recovery (how long to restore after a failure). Availability is the steady-state fraction of time the component is healthy.

Series topology · all components must be up
Aseries  =  A1 · A2 · A3 · … · An  =  i=1n   Ai

Series availability multiplies. The combined availability is always lower than the weakest component. Six independent components each at 99.9% give a series total of about 99.4%. Series is where systems go to fail.

Parallel topology · any one component suffices
Aparallel  =  1  −  i=1n   (1 − Ai)

Parallel availability multiplies the probabilities of failure, not of success. Three independent 99% components in parallel yield $1 - (0.01)^3 = 0.999999$, six 9s — over thirteen orders of magnitude better than a single component. Parallel is where reliability is bought.

Expected annual downtime · the operational consequence
Tdown  =  (1 − A) · 8760 hours/year

Convert availability into time. 99.9% available = 8.76 hours of downtime/year. 99.99% = 52.6 minutes. 99.999% = 5.3 minutes. The leap from each nine to the next is a 10× reduction in downtime, and a roughly 10× increase in the engineering required to achieve it.

Doctrinal note · independence is the load-bearing assumption
The parallel formula is only this elegant when component failures are independent. Shared power, shared network, shared cooling, shared deployment-substrate — all break independence. When three Mac Studios share one Tailscale router, the router becomes a series bottleneck regardless of how parallel the Studios appear in compute. The Earth-tier discipline: identify the implicit serial elements in any nominally parallel architecture, and reason about common-mode failures as the dominant risk.

Worked Example

Hedronite scenario · k3s mesh availability across the operator substrate

The Hedronite operator mesh runs on three Mac Studios behind a single Tailscale relay. Each Mac Studio has an estimated MTBF of 6 months (4380 hours) and MTTR of 4 hours (Marci's measured average for the kosha-pod-restart-and-state-rehydrate path). Tailscale relay availability is published at 99.95%. What is the mesh's compound availability and expected annual downtime, and where is the bottleneck?

Step 1 · Component availability per Mac Studio

AStudio = 4380 / (4380 + 4) = 4380 / 4384 ≈ 0.999087

Each Mac Studio is independently available about 99.91% of the time. The 4-hour MTTR is what shifts a 6-month MTBF from five nines down to three nines — recovery time matters as much as failure rate.

Step 2 · Parallel availability of the three-Studio fleet

If any one Studio can absorb the operator load (which is the design intent of the redundant mesh), the fleet's availability is the parallel formula applied to three identical components:

Afleet = 1 − (1 − 0.999087)3 = 1 − (0.000913)3 = 1 − 7.6 × 10−101.000000

That is approximately seven 9s — about 24 milliseconds of expected fleet-level downtime per year. The parallel topology delivers a staggering reliability multiplication.

Step 3 · Adding the Tailscale relay in series

But the fleet does not stand alone — it sits behind a single Tailscale relay through which all operator traffic flows. Tailscale availability is 99.95%, in series with the fleet:

Atotal = Afleet · ATailscale = 0.999999999 · 0.9995 ≈ 0.9995

The seven-9s fleet has been throttled to the three-and-a-half-9s of its serial dependency.

Step 4 · Expected annual downtime and the bottleneck

Tdown = (1 − 0.9995) · 8760 = 0.0005 · 8760 ≈ 4.4 hours/year

The Tailscale relay accounts for essentially all of the downtime. The redundant Mac Studio mesh contributes negligibly. The bottleneck is the relay, not the fleet.

If higher availability is required, the leverage is on the relay — multi-relay redundancy, a self-hosted DERP, or a topology change that eliminates the serial dependency entirely. Throwing a fourth Mac Studio at the problem accomplishes nothing because the fleet isn't the constraint.

Operational note · Earth-tier discipline at work
This is the canonical pattern. Identify the implicit serial element in a nominally parallel architecture. Reliability investment outside the bottleneck is wasted; reliability investment in the bottleneck is multiplicative. Marci's chassis-domain decisions about Tailscale alternatives (PORTUNUS spec context) are made under exactly this kind of analysis.

Cross-Domain Ripples

Availability arithmetic is wiring at the Earth tier — it threads through every element where systems must run continuously and component failures compound. The shape of the math is the same; the surface of application differs.

Element Where availability arithmetic shows up
🜃 Earth · DevOps The native surface. k3s mesh availability, deployment platform reliability, CI/CD pipeline uptime. Every SLO/SLI calculation is availability arithmetic. The bottleneck-identification discipline above is the daily practice of Earth-tier mastery.
🜁 Air · Blockchain RPC node availability determines trade-execution reliability. Multi-node redundancy is parallel availability; serial dependencies (single bridge, single oracle) impose serial bottlenecks. Chain finality probability is itself an availability quantity — given network state, what's the probability a transaction settles within $t$ seconds?
🜂 Fire · AIOps Model-serving infrastructure availability, agent uptime, MoE-routing reliability. Retry budgets are quantitative availability investments — given retry-rate $r$ and per-call success rate $p$, the effective availability after $k$ retries is $1 - (1-p)^k$. Same parallel formula, different name.
🜄 Water · MLOps Training-run completion rate, pipeline-scheduler reliability, drift-detection availability. A retraining cadence whose pipeline succeeds 99% of the time fails to retrain on average 3.65 days/year — for fast-decaying models, those days are real edge degradation. Compound failure across pipeline stages is series-availability.
🜀 Aether · Quant Strategy execution uptime translates directly into edge capture. A latency-arb strategy with 99.5% uptime misses 0.5% of opportunities — meaningful at HFT-class trade rates, negligible at swing-class. Capacity for edge capture is computed from the parallel availability of the execution stack. AQS primitives that account for execution-uptime as a positional factor are Earth-wiring threaded through Aether.

The generalization: any time a master practitioner needs to reason about whether a continuously-running system will be running when it needs to be, the wiring beneath is availability arithmetic. The Magus station in Earth means this is reflex; the Archmagus title means the same reflex carries through all five elements.

Practice Problems

Problem 1 · Virtual Machinist tier

Two servers in series, each independently available 99.9% of the time. What is the combined availability, and what is the expected annual downtime?

Solution

Series availability: A = 0.999 · 0.999 = 0.998001 ≈ 99.80%

Annual downtime: (1 − 0.998001) · 8760 = 0.001999 · 8760 ≈ 17.5 hours/year

Two 99.9% components in series do not give you 99.9%. They give you 99.8% — twice the downtime of either component alone. This is the canonical demonstration that series multiplies vulnerabilities.

Problem 2 · Magus tier

Compute the compound availability and annual downtime for this Hedronite-shaped architecture: three Mac Studios (each 99% available) in parallel, behind a Tailscale relay (99.95% available), feeding a single RPC node (99.5% available) for chain interaction. Identify the bottleneck.

Solution

Parallel fleet (3 Mac Studios): Afleet = 1 − (1 − 0.99)3 = 1 − 0.000001 = 0.999999 (six 9s)

Series total: Atotal = Afleet · ATailscale · ARPC = 0.999999 · 0.9995 · 0.995 ≈ 0.9945

Annual downtime: (1 − 0.9945) · 8760 ≈ 48.2 hours/year

Bottleneck: the RPC node (99.5%) dominates by an order of magnitude over Tailscale (99.95%), and both swamp the parallel fleet's negligible contribution. The leverage for improvement is on the RPC layer — multi-RPC redundancy would yield the largest gain. Adding a fourth Mac Studio accomplishes nothing because the parallel fleet is already past the point of diminishing returns.

Problem 3 · Archmagus tier

Design a redundancy architecture for the Hedronite operator mesh that achieves 99.99% availability (about 52 minutes of downtime per year), given these component constraints: Mac Studio MTBF 6 months, MTTR 4 hours; Tailscale relay 99.95% baseline. You may add components, restructure topology, or invest in MTTR reduction. Identify the cheapest path to the target and the principle that justifies your choice.

Solution sketch (not unique — multiple architectures hit the target)

The bottleneck-first principle: the parallel fleet is already at seven 9s. The serial dependency (Tailscale at 99.95%) caps the system at 99.95%, regardless of fleet improvements. Investment outside the bottleneck cannot exceed the bottleneck. All paths to 99.99% must operate on the Tailscale layer.

Path A · Multi-relay redundancy. Run two independent Tailscale relays (or one Tailscale + one self-hosted DERP) in parallel. Combined relay availability: 1 − (1 − 0.9995)2 = 1 − 0.00000025 ≈ 0.99999975. Combined system: ≈ 0.99999975 (effectively limited by the now-tiny fleet contribution). Easily exceeds 99.99%. Cheapest path if a second relay is cheap.

Path B · MTTR reduction. If Tailscale offers an SLA upgrade with faster recovery, modeled MTBF/MTTR could shift the relay above 99.99%. Less leverage than redundancy; depends on vendor terms.

Path C · Eliminate the serial dependency. Replace single-Tailscale with multiple Tailnet routes per Studio (each Studio joins the Tailnet directly with its own routing), or move to a peer-to-peer mesh that doesn't require a relay for intra-cluster traffic. Topology change is the highest-leverage but highest-engineering-cost option. This is the PORTUNUS bridge design space.

The principle: reliability investment is always at the bottleneck. The architect's first move is bottleneck identification; the architect's second move is choosing between (a) redundancy at the bottleneck, (b) MTTR reduction at the bottleneck, or (c) topology change that removes the bottleneck. Each costs differently; the choice is engineering-economic. Investment outside the bottleneck wastes capital.

Further Reading