🜃 Earth · DevOps Wiring · L1 Probability · L2 Statistics Cube

Availability and Compound Failure

TomesBlitzstein & Hwang Ch 4–5 (wiring) · Beyer, Jones, Petoff, Murphy Site Reliability Engineering (operative) Concept-ClassReliability arithmetic — series, parallel, the bottleneck StationVirtual Machinist → Magus (Earth-domain threshold) StageAlbedo · Citrinitas RipplesEarth (primary) · Air · Fire · Water · Aether
The foundation that holds is the foundation whose failure modes have been quantified. Hope is not a load-bearing material.

Intuition

Availability is not a property a system has — it is a property that emerges from the reliability of its components and the topology that joins them. Two servers in series are less available than either alone (both must work). Two servers in parallel are more available than either alone (one suffices). The mathematics of compound failure is the arithmetic by which an Earth-tier practitioner reasons about whether a foundation can bear its load.

Every claim of "five nines" or "99.9% uptime" is a probabilistic claim. The discipline is reading it as such. A 99% available server has roughly 3.65 days of downtime per year. Three such servers in series — say load-balancer, app, database, where any one going down takes the system down — yields combined availability $0.99 \cdot 0.99 \cdot 0.99 \approx 0.970$, or about eleven days of annual downtime. The series topology converts component-reliability into compound-fragility.

The Earth-tier shift in perception: stop treating "is the system available" as a yes/no question. Treat it as a continuous quantity that decomposes by topology. Once that becomes reflex, capacity planning, redundancy decisions, and SLO negotiations all become tractable. The wiring beneath this is the probability of independent events — L1 Probability, Blitzstein chapters 4 and 5.

The Formula

Availability of a single component
A  =  MTBF MTBF + MTTR

Where MTBF is the mean time between failures (how long the component runs healthy on average) and MTTR is the mean time to recovery (how long to restore after a failure). Availability is the steady-state fraction of time the component is healthy.

Series topology · all components must be up
Aseries  =  A1 · A2 · A3 · … · An  =  ∏i=1n   Ai

Series availability multiplies. The combined availability is always lower than the weakest component. Six independent components each at 99.9% give a series total of about 99.4%. Series is where systems go to fail.

Parallel topology · any one component suffices
Aparallel  =  1  −  ∏i=1n   (1 − Ai)

Parallel availability multiplies the probabilities of failure, not of success. Three independent 99% components in parallel yield $1 - (0.01)^3 = 0.999999$, six 9s — over thirteen orders of magnitude better than a single component. Parallel is where reliability is bought.

Expected annual downtime · the operational consequence
Tdown  =  (1 − A) · 8760 hours/year

Convert availability into time. 99.9% available = 8.76 hours of downtime/year. 99.99% = 52.6 minutes. 99.999% = 5.3 minutes. The leap from each nine to the next is a 10× reduction in downtime, and a roughly 10× increase in the engineering required to achieve it.

Doctrinal note · independence is the load-bearing assumption
The parallel formula is only this elegant when component failures are independent. Shared power, shared network, shared cooling, shared deployment-substrate — all break independence. When node-a, node-b, and node-c share one mesh VPN relay, the relay becomes a series bottleneck regardless of how parallel the nodes appear in compute. The Earth-tier discipline: identify the implicit serial elements in any nominally parallel architecture, and reason about common-mode failures as the dominant risk.

Worked Example

Worked example · three nodes behind a mesh VPN relay

Three nodes — node-a, node-b, and node-c — sit behind a single mesh VPN relay. Each node has an estimated MTBF of 6 months (4380 hours) and an MTTR of 4 hours (the measured average to restart the workload and rehydrate its state). Relay availability is 99.95%. What is the compound availability and expected annual downtime, and where is the bottleneck?

Step 1 · Component availability per node

Anode = 4380 / (4380 + 4) = 4380 / 4384 ≈ 0.999087

Each node is independently available about 99.91% of the time. The 4-hour MTTR is what shifts a 6-month MTBF from five nines down to three nines — recovery time matters as much as failure rate.

Step 2 · Parallel availability of node-a, node-b, and node-c

If any one node can absorb the load (the design intent of the redundant set), availability is the parallel formula applied to three identical components:

Anodes = 1 − (1 − 0.999087)3 = 1 − (0.000913)3 = 1 − 7.6 × 10−10 ≈ 1.000000

That is approximately seven 9s — about 24 milliseconds of expected downtime per year across the three nodes. The parallel topology delivers a staggering reliability multiplication.

Step 3 · Adding the mesh VPN relay in series

The nodes do not stand alone — they sit behind a single mesh VPN relay through which all traffic flows. Relay availability is 99.95%, in series with the nodes:

Atotal = Anodes · Arelay = 0.999999999 · 0.9995 ≈ 0.9995

The seven-9s node set has been throttled to the three-and-a-half-9s of its serial dependency.

Step 4 · Expected annual downtime and the bottleneck

Tdown = (1 − 0.9995) · 8760 = 0.0005 · 8760 ≈ 4.4 hours/year

The mesh VPN relay accounts for essentially all of the downtime. The redundant nodes contribute negligibly. The bottleneck is the relay, not the nodes.

If higher availability is required, the leverage is on the relay — a second independent relay, or a topology change that eliminates the serial dependency entirely. Adding a fourth node accomplishes nothing because the nodes are not the constraint.

Operational note · Earth-tier discipline at work
This is the canonical pattern. Identify the implicit serial element in a nominally parallel architecture. Reliability investment outside the bottleneck is wasted; reliability investment in the bottleneck is multiplicative. Choosing a second relay, or removing the relay, is made under exactly this kind of analysis.

Cross-Domain Ripples

Availability arithmetic is wiring at the Earth tier — it threads through every element where systems must run continuously and component failures compound. The shape of the math is the same; the surface of application differs.

Element Where availability arithmetic shows up
🜃 Earth · DevOps The native surface. k3s mesh availability, deployment platform reliability, CI/CD pipeline uptime. Every SLO/SLI calculation is availability arithmetic. The bottleneck-identification discipline above is the daily practice of Earth-tier mastery.
🜁 Air · Blockchain RPC node availability determines trade-execution reliability. Multi-node redundancy is parallel availability; serial dependencies (single bridge, single oracle) impose serial bottlenecks. Chain finality probability is itself an availability quantity — given network state, what's the probability a transaction settles within $t$ seconds?
🜂 Fire · AIOps Model-serving infrastructure availability, agent uptime, MoE-routing reliability. Retry budgets are quantitative availability investments — given retry-rate $r$ and per-call success rate $p$, the effective availability after $k$ retries is $1 - (1-p)^k$. Same parallel formula, different name.
🜄 Water · MLOps Training-run completion rate, pipeline-scheduler reliability, drift-detection availability. A retraining cadence whose pipeline succeeds 99% of the time fails to retrain on average 3.65 days/year — for fast-decaying models, those days are real edge degradation. Compound failure across pipeline stages is series-availability.
🜀 Aether · Quant Strategy execution uptime translates directly into edge capture. A latency-arb strategy with 99.5% uptime misses 0.5% of opportunities — meaningful at HFT-class trade rates, negligible at swing-class. Capacity for edge capture is computed from the parallel availability of the execution stack. AQS primitives that account for execution-uptime as a positional factor are Earth-wiring threaded through Aether.

The generalization: any time a master practitioner needs to reason about whether a continuously-running system will be running when it needs to be, the wiring beneath is availability arithmetic. The Magus station in Earth means this is reflex; the Archmagus title means the same reflex carries through all five elements.

Practice Problems

Problem 1 · Virtual Machinist tier

Two servers in series, each independently available 99.9% of the time. What is the combined availability, and what is the expected annual downtime?

Solution

Series availability: A = 0.999 · 0.999 = 0.998001 ≈ 99.80%

Annual downtime: (1 − 0.998001) · 8760 = 0.001999 · 8760 ≈ 17.5 hours/year

Two 99.9% components in series do not give you 99.9%. They give you 99.8% — twice the downtime of either component alone. This is the canonical demonstration that series multiplies vulnerabilities.

Problem 2 · Magus tier

Compute the compound availability and annual downtime for this architecture: three nodes, node-a, node-b, and node-c (each 99% available) in parallel, behind a mesh VPN relay (99.95% available), feeding a single RPC node (99.5% available). Identify the bottleneck.

Solution

Parallel nodes (node-a, node-b, node-c): Anodes = 1 − (1 − 0.99)3 = 1 − 0.000001 = 0.999999 (six 9s)

Series total: Atotal = Anodes · Arelay · ARPC = 0.999999 · 0.9995 · 0.995 ≈ 0.9945

Annual downtime: (1 − 0.9945) · 8760 ≈ 48.2 hours/year

Bottleneck: the RPC node (99.5%) dominates by an order of magnitude over the relay (99.95%), and both swamp the parallel nodes' negligible contribution. The leverage for improvement is on the RPC layer — a second RPC path would yield the largest gain. Adding a fourth node accomplishes nothing because the parallel nodes are already past the point of diminishing returns.

Problem 3 · Archmagus tier

Design a redundancy architecture for three nodes behind one mesh VPN relay that achieves 99.99% availability (about 52 minutes of downtime per year), given these component constraints: each node MTBF 6 months, MTTR 4 hours; mesh VPN relay 99.95% baseline. You may add components, restructure topology, or invest in MTTR reduction. Identify the cheapest path to the target and the principle that justifies your choice.

Solution sketch (not unique — multiple architectures hit the target)

The bottleneck-first principle: the parallel nodes are already at seven 9s. The serial dependency (the relay at 99.95%) caps the system at 99.95%, regardless of node improvements. Investment outside the bottleneck cannot exceed the bottleneck. All paths to 99.99% must operate on the relay.

Path A · Multi-relay redundancy. Run two independent relays in parallel. Combined relay availability: 1 − (1 − 0.9995)2 = 1 − 0.00000025 ≈ 0.99999975. Combined system: ≈ 0.99999975 (effectively limited by the now-tiny contribution of the nodes). Easily exceeds 99.99%. Cheapest path if a second relay is cheap.

Path B · MTTR reduction. If the relay provider offers faster recovery, modeled MTBF/MTTR could shift the relay above 99.99%. Less leverage than redundancy; depends on the provider's terms.

Path C · Eliminate the serial dependency. Give each node its own route into the mesh (node-a, node-b, and node-c join directly, each with its own routing), or move intra-cluster traffic onto a peer-to-peer mesh that does not require a relay. Topology change is the highest-leverage but highest-engineering-cost option.

The principle: reliability investment is always at the bottleneck. The architect's first move is bottleneck identification; the architect's second move is choosing between (a) redundancy at the bottleneck, (b) MTTR reduction at the bottleneck, or (c) topology change that removes the bottleneck. Each costs differently; the choice is engineering-economic. Investment outside the bottleneck wastes capital.

Further Reading