Cosmos SDK Migrations and the IBC Protocol Versioning Surface: Sovereign Chain Coordination Across Consensus Upgrades

Sovereign Chain Coordination Across Consensus Upgrades

<!-- hal:authoritative:yaml -->

Sovereign Chain Coordination Across Consensus Upgrades

§I — Frame

A Cosmos chain that upgrades its consensus rules cannot upgrade in isolation. The chain has IBC channels open to counterparty chains, each of which depends on a stable picture of the upgrading chain's state-transition rules to verify cross-chain packets. An upgrade that changes the rules without coordinating with the channel counterparties either breaks the channels, breaks the chains the counterparties run, or both. The Cosmos SDK exposes a specific module to make this coordination tractable. This lesson is about that module, the upgrade-handler discipline that sits inside it, and the IBC-channel versioning model the module's design respects.

This is the inaugural Interchain cert lesson. The Cosmos ecosystem is built on the sovereignty principle — each chain is its own state machine with its own validator set and its own upgrade authority. The IBC protocol is what lets sovereign chains coordinate without yielding sovereignty to a shared authority. The upgrade module and the IBC versioning surface are where these two principles meet most concretely. An Interchain developer who understands the two together is positioned to reason about every consequential operational decision a Cosmos chain makes.

The lesson grounds in three artifacts: the Cosmos SDK's x/upgrade module, the upgrade-handler pattern that lives inside it, and the IBC protocol's versioning surfaces at the channel, connection, and client tiers. Each section names what the artifact does, what its API looks like, and how an Interchain developer interacts with it during a real coordinated upgrade.

§II — Foundations: The x/upgrade Module

The Cosmos SDK ships an x/upgrade module that every chain in the ecosystem uses to coordinate consensus upgrades. The module's job is to halt the chain at a specified height, store the upgrade's planned state, and re-enter block production once the new binary is in place with the migrations applied.

The module's data model is two structures. An UpgradePlan records the upgrade's name, the target height (or block time), an optional info string, and the upgrade-handler key. A Plan is created by a governance proposal of type MsgSoftwareUpgrade, voted on by the validator set, and stored on-chain if it passes. The module's state-machine logic checks the current block height against any stored Plan at every block; when the height reaches the Plan's target, the module triggers a panic-style halt that brings the chain to a clean stop with the post-upgrade work persisted in the chain's halt-info file.

The post-halt work is the second half. The chain binary that takes over after the halt — the new version — reads the halt-info file, finds the Plan, and looks up the upgrade-handler registered for the Plan's name. The handler runs in a special context where it has access to the chain's state-transition functions but is not yet producing blocks. The handler's responsibility is to migrate the state from the pre-upgrade format to the post-upgrade format and to register any new modules the upgrade introduces. When the handler returns successfully, the chain resumes block production at the height immediately after the halt.

The handler is the artifact the upgrading chain's developers author. It is the most consequential code an Interchain developer writes, because every running validator's chain state will pass through it at exactly the same height. A bug in the handler that survived test-network rehearsal will manifest on every validator simultaneously and will produce divergent state if the handler is non-deterministic. The handler's discipline mirrors the discipline of a database migration: idempotent where possible, deterministic always, audited carefully, rehearsed thoroughly.

§III — Mechanism: The Upgrade Handler

A Cosmos SDK upgrade handler is a Go function registered with the upgrade keeper during the chain's app.go initialization. The function takes the chain's context, the upgrade Plan, and the current module-version map; it returns the new module-version map and an error. The chain's binary holds a handler for every named upgrade it knows about; at the halt height, the binary looks up the handler whose name matches the stored Plan and invokes it.

Here is the canonical handler shape from a recent Cosmos Hub release:

app.UpgradeKeeper.SetUpgradeHandler(
    "v17",
    func(ctx sdk.Context, plan upgradetypes.Plan, fromVM module.VersionMap) (module.VersionMap, error) {
        ctx.Logger().Info("starting v17 upgrade migration")
        newVM, err := app.mm.RunMigrations(ctx, app.configurator, fromVM)
        if err != nil {
            return newVM, err
        }
        if err := migrateValidatorSetExtraFields(ctx, app.StakingKeeper); err != nil {
            return newVM, err
        }
        if err := initializeNewIBCMiddlewareParams(ctx, app.IBCKeeper); err != nil {
            return newVM, err
        }
        ctx.Logger().Info("v17 upgrade migration complete")
        return newVM, nil
    },
)

The handler has three responsibilities. The first is to run module migrations through app.mm.RunMigrations, which walks every registered module and invokes its migration handler if its on-chain version is behind its in-binary version. The second is to perform any custom state-modification work that the standard migration framework does not handle. The third is to return the new module-version map so the upgrade keeper can persist it for the next upgrade's fromVM baseline.

Each line in the handler is a potential failure point. A migration that errors leaves the chain in an inconsistent state; the operator must roll back the binary, the state, and the upgrade-keeper state in coordination. The standard mitigation is to bracket the handler's risky work in panic recovery and to use the SDK's store-key snapshotting to checkpoint state at points where rollback is possible. Production handlers in mature chains often run to several hundred lines for major upgrades; the Cosmos Hub's Gaia v17 handler is publicly readable in the gaia repository and rewards detailed study.

The upgrade-handler discipline pairs with the validator-operator discipline named in today's Ops lesson. The operator's four rehearsal steps test the handler from the outside: the test-network upgrade exercises the handler against real chain state, and any divergence between the test-network handler run and the mainnet handler run is the failure-mode the rehearsal exists to catch. From the developer's side, the rehearsal is the validation that the handler has been authored correctly; from the operator's side, the same rehearsal is the validation that the upgrade procedure works end-to-end.

§IV — IBC Protocol Versioning: Three Tiers

The IBC protocol is layered. A light-client on chain A tracks the consensus state of chain B. A connection between the two chains binds two light-clients together and carries the protocol-version negotiation. A channel on top of a connection scopes the connection to a specific application protocol — token transfers, interchain accounts, interchain queries, or any custom protocol the chain teams have agreed on. Each tier has its own versioning surface, and each tier is affected differently by a chain upgrade.

Tier 1 — Light-client versioning. The IBC light-client is the on-chain proof system that verifies counterparty consensus. When a chain upgrades in a way that changes how its consensus state should be serialized — a new validator-set encoding, a new commitment-root scheme, a new key-rotation primitive — the counterparty's light-client must be updated to match. The x/upgrade module's UpgradePlan carries an optional UpgradedClientState field exactly for this case. The upgrading chain commits its post-upgrade client state into the Plan at proposal time; counterparty chains relay this client state into their own light-client of the upgrading chain at the upgrade height. If the counterparty does not update the light-client, IBC packets between the two chains stall.

Tier 2 — Connection versioning. A connection between two chains is negotiated when the connection is established, and the negotiated version is stored as part of the connection's on-chain state. The version field is a structured identifier that names the connection's feature set: ordered packet delivery, packet flush behavior, fee middleware, channel-upgrade support. A chain upgrade that introduces a new connection-version feature does not break existing connections, but new connections opened after the upgrade can negotiate the new version. Existing connections can upgrade their version through the channel-upgradeability flow added in IBC v8; this is one of the cleanest examples of how the protocol absorbed an extensibility surface that earlier versions lacked.

Tier 3 — Channel versioning. A channel between two chains is opened on top of a connection and carries its own version field, which scopes the application protocol. The version is negotiated when the channel is opened; for token transfers it is typically ics20-1 or ics20-2; for interchain accounts it is a JSON-encoded structure carrying the host-chain's metadata. A chain upgrade that introduces a new channel-version of an application protocol — say, ics20-2 adding metadata-passthrough that ics20-1 did not have — must coordinate with counterparty chains running the same application. The upgrade either opens new channels at the new version while keeping old channels at the old version, or it triggers a channel-upgrade flow to migrate existing channels to the new version.

The three tiers compose. An IBC packet flowing across a channel is verified against the connection, which is verified against the light-client, which is verified against the counterparty consensus. Every tier in the stack must agree on its version with its counterparty for the packet to deliver. A coordinated upgrade that affects any tier without coordinating the counterparty's matching change breaks the stack at that tier and stalls every packet that depends on it.

§V — Connection to Today's Ops + Dev Lessons

Today's Ops lesson named the four pre-activation rehearsal steps and walked the example of a Cosmos Hub v17 upgrade. The upgrade-handler artifact named in this lesson is the developer-side artifact that the operator's rehearsal exercises. The two lessons describe the same upgrade event from two angles: the operator's preparation discipline and the developer's authoring discipline. An operations team and a development team that read both lessons together arrive at the next upgrade with a shared vocabulary.

Today's Dev lesson covered Python async RPC polling for validator-set telemetry. The monitor described there extends to IBC-tier observability: polling the chain's ibc query endpoints surfaces light-client update height, connection state, and channel state. Pre-upgrade verification benefits from polling not just the validator set but also the IBC tier — confirming that counterparty light-clients are updated, that connections are healthy, and that channels are flowing packets, before the upgrade height arrives. The async polling pattern of the Dev lesson applies directly; the only change is the endpoint set.

§VI — Practice Questions

A. The chain resumes from height 18,000,000 with the v16 binary because the halt is informational. B. The v16 binary detects the unresolved upgrade Plan and refuses to start, halting again immediately. C. The chain reorganizes around the v16 binary because the upgrade governance is non-binding. D. The v16 binary signs a v17 block, double-signing the v16 block at the same height and getting slashed.

*Answer: B. The x/upgrade module persists the upgrade Plan into the halt-info file, and the v16 binary checks the file at startup. Detecting an unresolved upgrade, it refuses to proceed. The operator must run the v17 binary to clear the Plan and resume the chain.*

A. Packets continue to flow with no degradation. B. Outgoing packets from the upgrading chain to the counterparty stall because the counterparty's light-client cannot verify the new consensus state. C. The counterparty's chain halts. D. The IBC connection is terminated and must be re-established from scratch.

Answer: B. Packets stall at the counterparty's verification step. The relayer must update the counterparty's light-client with the upgraded client state before packet flow resumes; this is a recoverable failure mode, not a terminal one.

A. Fully rolled back to the pre-upgrade state. B. Partially migrated — modules processed before the error reflect the new schema; modules processed after the error retain the old schema. C. Fully migrated, with the error logged for follow-up. D. The chain halts permanently and requires manual state reconstruction.

*Answer: B. Module migrations within RunMigrations are processed sequentially; modules completed before the failure are persisted, and the failure leaves the remainder unmigrated. Recovery requires either a corrected handler that detects partial state and completes the migration or a coordinated rollback of the binary and the persisted state. This is the failure mode the test-network rehearsal exists to catch before mainnet.*

§VII — Closing

The Cosmos SDK's upgrade module and the IBC versioning surface are two halves of the same coordination problem: how do sovereign chains change their rules without breaking the cross-chain mesh they participate in. The upgrade module solves the in-chain half. The IBC versioning model solves the cross-chain half. An Interchain developer who can move fluently between the two is positioned to author or operate any chain in the ecosystem.

The Interchain cert path covers many surfaces; this lesson stakes the territory at the operational core. The next Interchain lessons in the cert-prep arc will deepen into specific modules — staking, governance, distribution, mint, slashing — and into the specific IBC application protocols. Each future lesson references the upgrade-and-versioning frame established here.

Read the Ops lesson before this one to ground the operational frame. Read the Dev lesson alongside this one to see the monitoring infrastructure that the operator and developer both depend on.