Python for Validator-Set Telemetry: Async RPC Polling, Voting-Power Change Detection, and Pre-Upgrade Verification
Async RPC Polling, Voting-Power Change Detection, and Pre-Upgrade Verification
<!-- hal:authoritative:yaml -->
Async RPC Polling, Voting-Power Change Detection, and Pre-Upgrade Verification
§I — Frame
Today's Ops lesson named the four rehearsal steps for a coordinated chain upgrade. Each step ends in an audit question the operator must answer before proceeding. The answers come from a monitoring layer that watches the chain and surfaces the relevant state. This Dev lesson is about that layer.
Python is the right language for the layer because the work is shaped like research scripting at production cadence: poll many endpoints concurrently, normalize their responses, diff against prior state, surface the diffs that matter, and either alert or quietly log. The same script that an operator writes for ad-hoc investigation can graduate to a service by wrapping it in a long-running loop and forwarding its output to a metric sink. The language meets the workflow where the workflow lives.
The discipline that holds the work together is async I/O. A validator-set monitor polls every active validator on the chain. Cosmos Hub runs around 175 validators; Solana runs more than 1,400 active validators across testnet and mainnet; Ethereum's beacon chain runs more than a million. Sequential polling is not viable. Concurrent polling with bounded fan-out is. Python's asyncio gives you the primitive directly. The lesson walks the patterns that turn the primitive into a production telemetry layer for the chain-upgrade event class.
§II — Language Idiom: asyncio for I/O-Bound Fan-Out
Python's asyncio is purpose-built for the shape this lesson describes. Tens to thousands of network I/O operations, each waiting on a remote response, none CPU-bound. The event loop multiplexes the waits; a single Python process can hold thousands of in-flight RPC calls without spinning a thread per call.
The first primitive is asyncio.gather. It takes a collection of coroutines and runs them concurrently, returning when all of them complete. The second primitive is asyncio.Semaphore. It bounds the concurrency to a number the operator chose deliberately, so the script does not blow the chain's RPC endpoint or the operator's own network egress.
Here is the bounded fan-out pattern at its smallest:
import asyncio
import httpx
async def fetch_validator_status(client, validator_addr, sem):
async with sem:
response = await client.get(
f"https://rpc.cosmos.network/validators/{validator_addr}",
timeout=5.0,
)
response.raise_for_status()
return response.json()
async def poll_validator_set(addresses, max_concurrent=20):
sem = asyncio.Semaphore(max_concurrent)
async with httpx.AsyncClient() as client:
coros = [fetch_validator_status(client, addr, sem) for addr in addresses]
return await asyncio.gather(*coros, return_exceptions=True)The semaphore is the discipline. Without it, the script attempts to open one socket per validator and either the operator's machine runs out of file descriptors or the chain's RPC endpoint rate-limits the operator. The return_exceptions=True flag in asyncio.gather is the other discipline; one failing validator should not crash the entire poll. Failed coroutines return their exception as a value; the caller filters them out and handles them separately from the successful results.
The 2026-05-18 Python Iterator Protocol lesson named the streaming-data shape; this lesson generalizes the same idea to concurrent network I/O. The 2026-05-21 contextvars lesson named the discipline of per-task state; the same primitive is useful here when the operator wants to attach per-validator logging context that propagates through the async stack without being passed as an argument at every layer.
§III — Code Worked Example: A Validator-Set Diff Monitor
The complete monitor watches the active validator set on a Cosmos-class chain, polls every validator's status concurrently, computes the diff against the prior snapshot, and surfaces voting-power changes that exceed a threshold. The shape composes the asyncio primitive of §II with a state-diff loop and a metrics emission step.
import asyncio
import json
import logging
from dataclasses import dataclass
from pathlib import Path
import httpx
@dataclass
class ValidatorSnapshot:
address: str
moniker: str
voting_power: int
status: str
commission_rate: float
jailed: bool
@dataclass
class ValidatorDiff:
address: str
moniker: str
field: str
before: object
after: object
async def fetch_validator(client, address, sem):
async with sem:
try:
resp = await client.get(
f"https://rpc.cosmos.network/cosmos/staking/v1beta1/validators/{address}",
timeout=5.0,
)
resp.raise_for_status()
data = resp.json()["validator"]
return ValidatorSnapshot(
address=address,
moniker=data["description"]["moniker"],
voting_power=int(data["tokens"]),
status=data["status"],
commission_rate=float(data["commission"]["commission_rates"]["rate"]),
jailed=bool(data.get("jailed", False)),
)
except (httpx.HTTPError, KeyError, ValueError) as e:
logging.warning("validator_fetch_failed address=%s err=%s", address, e)
return None
async def poll_active_set(client, max_concurrent=20):
sem = asyncio.Semaphore(max_concurrent)
resp = await client.get(
"https://rpc.cosmos.network/cosmos/staking/v1beta1/validators?status=BOND_STATUS_BONDED&pagination.limit=200",
timeout=10.0,
)
resp.raise_for_status()
addresses = [v["operator_address"] for v in resp.json()["validators"]]
coros = [fetch_validator(client, addr, sem) for addr in addresses]
results = await asyncio.gather(*coros, return_exceptions=False)
return {v.address: v for v in results if v is not None}The fetch function is bounded by the semaphore and returns either a snapshot or None on failure. The poll function reads the bonded validator set first, then fans out concurrent fetches. The output is a dictionary keyed by validator address, which is the natural shape for diffing.
The diff layer follows. It compares two snapshots and emits diff records for any field that changed by more than the operator's threshold.
def diff_snapshots(prior, current, voting_power_threshold_pct=5.0):
diffs = []
for addr, after in current.items():
before = prior.get(addr)
if before is None:
diffs.append(ValidatorDiff(addr, after.moniker, "joined", None, after.voting_power))
continue
if before.jailed != after.jailed:
diffs.append(ValidatorDiff(addr, after.moniker, "jailed", before.jailed, after.jailed))
if before.status != after.status:
diffs.append(ValidatorDiff(addr, after.moniker, "status", before.status, after.status))
if before.voting_power > 0:
pct_change = abs(after.voting_power - before.voting_power) / before.voting_power * 100
if pct_change >= voting_power_threshold_pct:
diffs.append(ValidatorDiff(addr, after.moniker, "voting_power", before.voting_power, after.voting_power))
for addr, before in prior.items():
if addr not in current:
diffs.append(ValidatorDiff(addr, before.moniker, "left", before.voting_power, None))
return diffsThe diff function handles three cases. New validators that appeared in the current snapshot but not the prior one. Existing validators whose state changed past the threshold. Validators that disappeared from the current snapshot, which typically means they have been jailed and removed from the active set.
The orchestration loop runs the poll at a cadence, persists the snapshot, computes the diff against the previous one, and emits the diff records to a metric sink. The cadence depends on the chain's epoch length: Cosmos Hub epochs are ~6 seconds, so polling every 30 seconds captures every 5 blocks; Solana epochs are 2 days, so polling every few minutes is appropriate. The cadence should be tuned to the chain's pace and the operator's signal-noise tolerance.
async def monitor_loop(snapshot_dir, interval_seconds=30):
snapshot_dir.mkdir(parents=True, exist_ok=True)
prior_path = snapshot_dir / "latest.json"
prior = load_prior(prior_path)
async with httpx.AsyncClient() as client:
while True:
try:
current = await poll_active_set(client)
diffs = diff_snapshots(prior, current)
for d in diffs:
emit_diff_metric(d)
save_snapshot(prior_path, current)
prior = current
except Exception as e:
logging.error("monitor_loop_iteration_failed err=%s", e)
await asyncio.sleep(interval_seconds)The loop is intentionally simple. Pull, diff, emit, save, sleep. The complexity lives in the pull and diff steps; the loop itself stays out of the way. The try/except around the iteration prevents one bad pull from killing the long-running service.
§IV — Connection to Today's Ops Lesson
Today's Ops lesson named four rehearsal-step audit questions. Each one is a query the monitor described here is positioned to answer.
Does my binary hash match? The monitor extends to track the validator's reported software version. Cosmos's node_info endpoint surfaces the version; Solana's getVersion RPC does the same. Comparing the operator's own validator against the rest of the active set surfaces the upgrade lag: which validators have upgraded and which have not. A pre-activation chart of upgrade-readiness across the active set tells the operator how prepared the network as a whole is for the activation height.
Does the binary keep up with the network at the operator's current hardware sizing? The monitor extends to track per-validator block-signing latency. Tendermint's consensus_state endpoint and Solana's getVoteAccounts surface signing timing. Plotting the operator's own signing latency against the network median, both pre- and post-test-network-upgrade, shows whether the operator's hardware is in danger of falling behind.
Does my upgrade procedure work end-to-end? The monitor watches the test network through its rehearsal upgrade and records the operator's validator state across the activation. The pre-upgrade snapshot, the at-upgrade-height halt, the migration window, the post-upgrade resumption, and the first signed block on the new binary all appear in the snapshot history. The operator reviews the history after the rehearsal to confirm each phase completed as expected.
Is the post-upgrade steady-state what the release notes said it would be? The monitor's diff loop surfaces any unexpected change in voting-power distribution, commission rates, or validator-set membership in the days after the test-network upgrade. Anything that diverges from the release notes' expected behavior is a signal worth raising before the mainnet activation.
The monitor described here is the implementation tier of the audit gates the Ops lesson named. The two lessons are designed to be read together; the Ops lesson tells you what to watch for, and the Dev lesson shows you how to watch.
§V — Prior-Lesson Reach
The 2026-05-18 Iterator Protocol lesson named the streaming-data primitive of Python. The fetch-fan-out pattern of this lesson generalizes the same idea to network I/O. Both treat data as something you process as it arrives, not something you accumulate and then process. The asyncio primitive is the iterator-protocol-equivalent for concurrent I/O.
The 2026-05-21 contextvars lesson named per-task state propagation. Validator-monitor work benefits from the same discipline when the monitor grows to track multiple chains concurrently — per-chain logging context, per-chain metric prefixes, per-chain auth tokens all propagate cleanly through contextvars.ContextVar without polluting function signatures.
The 2026-05-27 secrets-and-cryptography lesson named the discipline of handling sensitive material in Python without it leaking through normal flow paths. A validator-monitor that also handles the operator's signing-key custody — even just to verify the signing-key is present on the expected machine — must apply the same discipline to the key material. The monitor should never log a key, never serialize a key into a snapshot file, never pass a key as a function argument it does not need. The same restrictive-handling reflex applies.
The 2026-05-23 Validator Operations lesson defined Identity, Liveness, Safety. The monitor watches all three at once. Identity, by tracking which key is signing for the validator. Liveness, by tracking signing rate and missed-block counts. Safety, by watching for any double-sign evidence the chain has emitted, which surfaces in slash-related events.
§VI — Closing
A monitor for a chain-upgrade event is a Python script that grows up into a service. The asyncio primitive is the spine; the diff loop is the discipline; the metric emission is the surface the operator actually reads. None of it is exotic. All of it is the difference between a validator that ships clean through an upgrade and a validator that does not.
The pattern travels. Cosmos, Solana, Substrate, Ethereum beacon-chain — the RPC surfaces differ, but the fan-out-and-diff shape is the same. Build the monitor once for one chain; the second chain is mostly a substitution exercise. By the third chain, the monitor is generic and the per-chain code is the adapter.
Read the Ops lesson before this one. Read the Cert lesson after.