Supply-Chain Security for Multi-Agent Production Systems — SBOMs, Signed Artifacts, and Admission-Time Provenance Verification

The bill names what is inside. The signature names who authorized it. The attestation names how it was built.

§ IFrame

Three Wednesdays of the β-Trust arc have been spent inside the cluster. Workload identity gave each agent a cryptographically-attestable name. Admission control gave the cluster a place to reject a workload before it ran. Runtime detection gave the operator eyes that watched the process tree after the workload landed. Each of these defenses assumed that the container image, once it arrived at the kubelet, was the image the engineer had reviewed.

The assumption is wrong as often as not. The image that the kubelet pulls is fetched from a registry by digest if the operator is careful, by tag if they are not, and by an automated chain of CI hand-offs that have re-tagged, re-pushed, and re-mirrored the bytes between the engineer's last review and the cluster's first pull. Every link in that chain is a place where the image could have been altered. An adversary who controls one link controls every cluster downstream of it.

This lesson sits on the link before the kubelet, the supply chain, and gives it the same kind of audit the runtime gets. Three named techniques carry the lesson: the bill of materials, which catalogs what is in the image; the signed artifact, which proves who put it there; and the provenance gate, which the cluster opens only when both the bill and the signature check out. The Wed-arc closes here. Identity, admission, runtime, and now provenance: four eyes on the production workload, each looking at a different seam.

§ IIFoundations

A container image is a tarball of layers, a manifest that names those layers, and a config blob that says how to run them. A registry stores all three under a content-addressable identifier called a digest. The digest is the cryptographic name of the bytes. Two images are the same image if and only if their digests match. Tags such as :latest, :v1.2.0, :prod are mutable pointers into a digest space. Trust a tag and trust the pusher; trust a digest and trust the bytes.

These three — bill, signature, attestation — are the documents that travel with the image. The bill names what is inside. The signature names who authorized it. The attestation names how it was built. A cluster that admits only images with all three documents has closed the supply-chain seam.

§ IIIMechanism

The mechanism has four moving parts: a build that produces the documents, a registry that stores them next to the image, a key infrastructure that the cluster trusts, and an admission gate that reads the documents before the workload runs. Each part has failure modes the operator should know cold.

The build

A CI pipeline compiles the application, packages it into an image, generates an SBOM from the package manifest of the chosen base, signs the image digest, and emits a provenance attestation describing the build. The Sigstore project provides the canonical tooling: cosign for signing, syft for SBOM generation, slsa-github-generator or its peers for provenance. The pipeline's responsibility is to do all four before pushing. A pipeline that pushes the image and signs it later opens a window of time where the image is pullable but unsigned. Adversaries that watch CI watch for that window.

The registry

A signature in the OCI 1.1 specification is itself an artifact stored alongside the image, addressed by the digest of the image it signs. The pull-by-signature query asks the registry for all artifacts that reference this digest with media-type signature. The registry returns zero, one, or many. The cluster validates each against its trusted public keys and accepts the workload if at least one signature is valid under at least one trusted key. The same pattern holds for SBOMs and attestations.

The keys

Three patterns exist for managing the verifier's public-key set. Long-lived key pairs stored in HSMs or KMS systems, common in enterprise pipelines, give the operator a stable verification anchor but require explicit rotation. Keyless signing via Fulcio, common in OSS, mints short-lived X.509 certificates bound to OIDC identities and logs the binding to a public transparency log called Rekor; the cluster verifies the OIDC issuer rather than a static key. The third pattern is hybrid: keyless for upstream OSS dependencies, long-lived keys for internally-built workloads. The hybrid pattern fits a fleet that runs both kinds of code.

The admission gate

Kubernetes admits a pod through the admission controller chain. A validating webhook can inspect any pod-create request, fetch the image's signature and attestation from the registry, verify them against the cluster's policy, and admit or reject. Three implementations dominate the production market: Sigstore's policy-controller, Kyverno's verify-images policy, and Connaisseur. All three follow the same general pattern. Each lets the operator name an image-pattern, the allowed signers for that pattern, and the required attestation predicates. A pod whose image matches a pattern but whose signature does not match the allowed signers is rejected at admission time, never reaching the kubelet.

§ IVWorked Example

A fleet runs a multi-agent inference workload. The workload pulls model weights from an internal registry, processes user queries through a routing agent, and emits outputs to an audit pipeline that was built in the prior Wed lesson. The fleet adopts cosign for signing, Syft for SBOMs, the GitHub-hosted SLSA generator for provenance attestations, and Sigstore's policy-controller for admission. Walk the lifecycle.

A developer pushes a commit that updates the routing agent. GitHub Actions builds the image and tags it with the commit SHA. Before the push, the workflow runs syft against the staging-stage image and emits a CycloneDX SBOM. The workflow then pushes the image to the registry. After the push completes and the registry returns the immutable digest, the workflow runs cosign sign against that digest, using a workload identity issued by the GitHub OIDC provider. Cosign requests a short-lived signing certificate from Fulcio, signs the digest, and uploads the signature back to the registry as an OCI artifact referencing the digest. The same workflow runs the SLSA generator, which emits a provenance attestation naming the source commit, the workflow file, the build runner, and the resulting digest; the attestation is also signed by Fulcio and uploaded.

A cluster operator deploys the routing agent. The deployment manifest names the image by tag. The kubelet resolves the tag to a digest at pull time, but the validating webhook intercepts the pod-create request first. The webhook fetches the signature artifact from the registry, verifies it against the Fulcio root and the configured OIDC issuer claim (must equal https://token.actions.githubusercontent.com), and verifies the workflow identity (must equal https://github.com/fleet-org/agent-router/.github/workflows/release.yml@refs/heads/main). The webhook then fetches the attestation, parses its SLSA predicate, and checks that the source commit lives on the main branch. Every check passes; the pod is admitted; the kubelet pulls; the workload runs.

A week later, a security researcher discovers a vulnerability in the model-loading library the routing agent uses. The vulnerability has a CVE and a patched version. The operator queries the fleet's SBOM index — every signed image has a stored SBOM, indexed by digest — for any image that includes the affected library version. Three images come back. Two are running in production; one is dormant. The operator pages the routing-agent team, who rebuild against the patched library, push, sign, attest, and deploy. The dormant image is removed from the registry. The fleet is patched in under four hours. Without the SBOM, the same query would have required pulling each image, extracting its filesystem, and grepping for the library — a workflow that takes days and misses dormant images entirely.

Now consider the failure mode. An adversary compromises a maintainer's GitHub account and pushes a malicious commit. CI runs the malicious code, produces a signed image, and emits a clean-looking provenance attestation that names the malicious commit. The admission webhook verifies the signature; it is technically valid; and admits the pod. The supply-chain controls did not prevent the attack; they did, however, create a permanent forensic record. The Rekor transparency log entry names the malicious commit, the workflow file, and the time of signing. The post-incident audit identifies the compromise in minutes, not weeks. Supply-chain security buys two things: it raises the cost of attack, and it shortens the time from detection to attribution.

§ VConnection to Prior Lessons

Three earlier Wed lessons compose with this one to form a complete defense.

SPIFFE/SPIRE workload identity (2026-05-20) gave each running workload a cryptographic name attested at runtime. That name answers who is this workload talking to me right now. Image signing answers a different question: who authorized the bytes this workload is built from. The two identities are distinct. SPIFFE attests the running process; cosign attests the artifact. A pod whose image is signed but whose runtime identity is wrong is still untrustworthy; a pod whose runtime identity is right but whose image was never signed has skipped a check the cluster owed itself.

Admission control with External Secrets Operator (2026-05-27) gave the cluster a webhook chain that runs before the kubelet pulls. The image-signing webhook is the same chain, the same architecture, with a different question. The 2026-05-27 lesson taught how to write the webhook; this lesson teaches what one specific webhook in that chain should ask. A cluster that runs both is running two checks: is this pod's secrets reference resolvable and is this pod's image signed by an authorized signer. Both run before the workload starts.

Runtime threat detection (2026-05-28) gave eyes inside the running pod. The runtime sees what the image actually does once the process is alive. Image signing controls what enters the pod; runtime detection controls what happens after it does. The two are complements. An attacker who slipped past admission (perhaps via an exploitable input) is caught at runtime. An attacker who never had a chance to slip past admission, because the image was rejected, never reaches runtime. The cost of compromise rises with each gate that closes.

The four gates together — identity, admission, signing, runtime — close the seams a multi-agent production system has between the build and the production read. Each gate is necessary; none is sufficient.

§ VIConnection to Today's Dev + Cert Lessons

The Go dev lesson today builds the admission webhook from scratch: go-containerregistry to fetch the registry artifacts, crypto/ed25519 for the signature primitive, an HTTP handler that the cluster posts pod-create reviews to, and the JSON encoding the webhook protocol expects. Where this Ops lesson described what the webhook checks, the Go lesson shows the wire-level mechanics of how to perform that check in code.

The cert lesson today maps the same architecture onto CKA + CKS exam scope. CKA owns the cluster-operations side: deploying the policy-controller, managing the ClusterImagePolicy CRDs, debugging when admission fails. CKS owns the security-policy side: writing the policy CRDs, choosing the OIDC issuers, integrating Sigstore into the org's SDLC. The two exam lenses converge at the same artifact this lesson describes.

§ VIIClosing

The supply chain is not a separate concern from the cluster. It is the cluster's past tense — the history of where every running byte came from. A cluster that admits unsigned images is a cluster whose past tense is anything could have happened. A cluster that requires signed images, SBOMs, and provenance attestations is a cluster whose past tense is here is who authorized this, here is what they built, here is the commit that produced it. The operator who walks into the post-incident review with the second answer is the operator who closes the incident the same day.

Four gates close the Wed-arc. Identity names the worker. Admission names the policy. Runtime names the behavior. Provenance names the build. Examine each seam well; the system lives where the gates meet.