Secret Management and Credential Lifecycle for Multi-Agent Production Systems

Vaulting, rotation, and the blast-radius discipline.

§ IFrame

The prior β-Trust lesson named who an agent is. SPIFFE issues an identity, SPIRE attests it, mTLS proves it on every connection. The agent now stands inside the system with a name the system trusts.

This lesson takes up what that name carries. Identity is the envelope; secrets are the letters inside. An agent that authenticates to a database still needs the database password. An agent that calls a model still needs the API key. An agent that signs a Solana transaction still needs the validator key. Identity opens the door; the secret does the work inside the room.

Production systems mishandle secrets in three predictable ways. They put them in environment variables and leak them in process listings. They commit them to git and discover the leak months later when someone scrapes the public repo. They rotate them never, then rotate them all at once when an audit lands, and watch half the fleet break for the duration of the change. Each failure is a credential-lifecycle failure, not a cryptography failure. The keys are strong. The handling is sloppy.

The operator who treats secrets as static configuration inherits whichever leak path the deployment opens. The operator who treats secrets as having a lifecycle (issued, scoped, rotated, revoked, audited) closes the leak paths before they open. The difference shows up the first time a developer's laptop is compromised, or the first time a vendor announces a breach upstream, or the first time the security team asks how long a particular key has been valid and the only honest answer is we don't know.

§ IIFoundations

Four primitives carry secret management. Name them; reason about them after.

The four compose into a single picture. The vault issues a scoped secret to an authenticated agent, the secret expires on a rotation schedule, and the blast radius of any compromise is bounded by the scoping decision the operator made at issuance.

§ IIIMechanism

How each primitive works in a production system.

The vault as the source of truth

The vault sits as a network service with its own identity and its own authentication path. HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Doppler are the common implementations; the architecture is the same across all of them. An agent presents its workload identity (the SPIFFE SVID from the prior lesson, or the IAM role from a cloud-native deployment) and receives back a short-lived credential scoped to a particular secret. The agent never sees the master copy. The vault never sees the agent's request to use the secret. It sees only the request to fetch it.

The authentication path to the vault is the foundation of the discipline. If the vault accepts a static token to authenticate agents to itself, the vault is itself a secret-handling problem, and the operator has only pushed the leak point one layer back. The pattern that closes this is workload-identity-based vault authentication: the agent proves who it is through the same identity mechanism the rest of the system trusts, and the vault checks that identity against a policy. The vault's own master key, the seal key, is held by the operator and never touches the agent fleet.

Rotation as a scheduled vault process

Rotation runs as a scheduled process owned by the vault. Modern vaults speak directly to the upstream provider (to PostgreSQL for database credentials, to AWS for IAM access keys, to GitHub for personal access tokens) and execute the rotation on the upstream side, then publish the new value through the same fetch endpoint the agents already use. The agent does not know rotation happened. It refetches its credential at every TTL boundary and receives whichever value the vault holds at that moment. The rotation is invisible to the agent and unavoidable for the attacker who held the prior credential.

Scoping as policy at issuance time

Scoping happens at issuance time. The vault holds a policy that maps workload identities to the set of secrets they may fetch. A model-router agent may fetch the OpenAI key but not the Anthropic key. A backup agent may fetch read-only database credentials but not write credentials. A trading agent may fetch the testnet exchange key but not the mainnet exchange key. Mainnet scope expansion requires explicit Sovereign authorization at the policy tier. Scoping is where the principle of least authority becomes a configurable, auditable thing.

Blast radius as the consequence

Blast radius emerges as the consequence of scoping and rotation decisions. A widely-scoped key rotated annually has a large blast radius for a long time. A narrowly-scoped key rotated hourly has a small blast radius for a short window. The operator chooses the trade-off explicitly: tighter scoping costs configuration complexity; shorter rotation costs reissue overhead. The right setting depends on what the secret protects.

For exchange API keys at the trading-operator tier (the freshly canonized Gemstone Blvd grouping running CryptoOps and QuantOps), the answer leans toward tight scoping and frequent rotation, because the loss is denominated in dollars and irrecoverable. For internal model API keys at the cognition tier, the answer can be looser, because the loss is denominated in API credits and rate-limited recoverable. The discipline is to name the trade-off rather than inherit a default.

§ IVWorked Example — Trading Operator with Five Credentials

A trading operator inside the Gemstone Blvd grouping needs to place orders on three exchanges: Binance, Kraken, and a Solana DEX through a wallet. The operator runs as a process inside a container, scheduled on the local OrbStack cluster on the Sovereign-Mac during development and on the production K8s cluster when promoted. The secret-handling problem has five distinct credentials with five distinct blast radii.

The vault holds all five. The trading-operator's workload identity grants fetch on the five secrets and nothing else. The signing-key fetch carries an additional check: the vault policy requires a fresh Sovereign-issued signed attestation within the last hour before issuing the wallet key, so that a compromised operator-process cannot pull the signing key without a Sovereign action having occurred recently. The other four secrets do not carry this constraint, because the blast radius does not warrant the friction. The discipline is per-secret, not per-fleet.

Each secret fetch is logged. The audit log carries the workload identity, the secret name, the timestamp, and the source IP. Anomalous patterns (a fetch from an unexpected IP, a fetch outside the trading hours window, a burst of fetches in a short period) trigger an alert before they trigger a loss. The audit is the rear-view mirror; the rotation is the windshield. Both are required.

§ VConnection to Prior Lessons

The Workload Identity lesson (2026-05-20) established SPIFFE/SPIRE/mTLS as the answer to who an agent is. This lesson takes that identity as a given and asks what it can fetch. The vault's authentication policy is written in terms of SVID URIs. spiffe://hedronite.local/trading/binance-trader may fetch the Binance key, and nothing else may. Identity without authorization is a name without a permission set; authorization without identity is a permission set with no one to grant it to. The two complete each other.

The Validator Operations lesson (2026-05-23) named the validator signing-key lifecycle as a hardest case. A validator key cannot be rotated cheaply because every chain participant has to learn the new public key, and the rotation window is itself a slashing-attack opportunity. The discipline there was to hold the signing key in a remote signer that the validator process talks to over a local socket, so the key material never sits in the validator's memory. The vault pattern in this lesson is the same architecture at a different scale: the secret stays in one place, the consumer fetches a usage rather than a copy.

The Model-Serving Topology lesson (2026-05-25) treated routing, pooling, versioning, and cost surface as the four primitives of inference economics. The cost surface assumes the model API key is correctly accounted to the calling agent. If the key is shared across agents, the cost surface cannot disambiguate. Per-agent scoping at the vault tier is what makes per-agent cost accounting possible at the application tier. The two layers compose.

§ VIConnection to Today's Dev Lesson

The Rust lesson takes up the secret-handling problem at the type level. A secret is a piece of memory the program should never accidentally log, never accidentally serialize, never accidentally let escape a narrow scope. Rust's ownership and type system can express each of these constraints at compile time, so the compiler refuses to build a program that violates them. The lesson covers the secrecy crate's Secret<T> wrapper that redacts Debug output, the type-state pattern that forces a fetch-and-use sequence rather than a fetch-and-hold sequence, and the explicit zeroization discipline that returns secret memory to a known state when the value drops.

The Ops lesson keeps secrets out of the wrong place by architecture. The Dev lesson keeps secrets out of the wrong place by type. Both disciplines compose: a vault-fetched secret arriving into a Rust process is held in a Secret<String>, used inside a scoped closure, and zeroized when the closure exits. The blast radius is bounded twice: once at issuance by the vault policy, again at use by the type system.

Paired Dev lesson → Polyglot-Dev/Rust/2026-05-26-rusts-ownership-discipline-applied-to-secret-handling-types-that-cannot-leak

§ VIIConnection to Today's Cert Lesson

Today's cert lesson treats AWS credential architecture as the unified ground for both the SAP (Solutions Architect Pro) and DOP (DevOps Engineer Pro) credentials. The vault-rotation-scoping-blast-radius primitives in this lesson are the platform-agnostic disciplines; AWS Secrets Manager, IAM, and KMS are AWS's managed implementation of them. The cert lesson shows the platform; this lesson shows the principles the platform implements.

The relationship matters. An operator who understands the principles can evaluate AWS Secrets Manager's defaults against the topology they need, decide which knobs to keep and which to override, and decide when self-hosted Vault on EKS earns its complexity over the managed service. The cert credential confirms platform fluency; the principles confirm the operator can use the platform deliberately rather than ceremonially.

Paired Cert lesson → Archmagus-Stack/09-Tomes/Cert Prep/AWS/2026-05-26-aws-credential-architecture-iam-secrets-manager-and-kms-across-sap-and-dop

§ VIIIClosing

A secret is a key to a room. The vault is the locksmith. Rotation is the schedule on which the lock changes. Scoping is the rule that says which keys open which rooms. Blast radius is the count of rooms a single key would open if found in the wrong hand.

The operator who builds the four primitives as a discipline keeps the count low and the schedule frequent. The operator who builds none of them is one developer-laptop theft away from explaining to the Sovereign which positions were liquidated and which wallets were drained.

Examine the credential paths in any system being designed. For each secret in the system, name where it sits, who fetches it, on what cadence it rotates, and what its blast radius would be if found tomorrow. The system that cannot answer those four questions for every secret is the system that holds an unmeasured liability.

Reflect on this.