Python and the Image Admission Gate — the gate that phones out

Every Pod creation in the cluster is about to wait on your HTTP call.

Five prior lessons ran beside the cluster, on their own clock. This one runs inside somebody else's.

§IFrame

The Ops lesson filed this morning ends with a Gatekeeper constraint that refuses any image reference lacking an @sha256: digest. That constraint is a string check. It tells an engineer that a tag is not allowed; it cannot tell the engineer which digest the tag currently points at.

Somebody has to make that call, and the call goes over the network to a registry.

Five prior lessons on this track wrote Python that talks to the API server. The controller reconciled. The webhook validated a Pod spec it already held in hand. The watch loop consumed a stream. The kopf operator drove desired state. The preflight asked the authorization API whether an action was permitted before attempting it. All five ran beside the cluster, on their own clock.

This one runs inside somebody else's clock. A validating webhook is called synchronously during admission, before the object reaches etcd, and Burns and co-authors are explicit in the policy chapter that the request blocks until the webhook answers. Add an outbound registry call to that path and every kubectl apply in the cluster now depends on a third party's availability.

Call it the gate that phones out. The design problem is not the HTTP request. The design problem is what happens when the request is slow.

§IILanguage Idiom

Three Python idioms carry this lesson.

A session, not a request. requests.Session keeps the TCP connection and the TLS handshake alive across calls to the same registry host. On a gate that fires on every Pod creation, connection reuse is the difference between a two-millisecond resolution and a ninety-millisecond one. The session also holds the bearer token, so the auth dance happens once rather than per image.

HEAD where the body is not wanted. The registry returns the manifest digest in the Docker-Content-Digest response header. A GET fetches the whole manifest body to read a header. A HEAD fetches the header. Poulton's digest section describes the client-side effect of this without naming the verb; the verb is where the Python saves the bytes.

A TTL cache, not lru_cache. functools.lru_cache never expires an entry. On tag-to-digest resolution that is wrong, because the whole reason tags are dangerous is that they move. A small dict of {(repo, tag): (digest, expires_at)} with a sixty-second life gives the gate its speed and still notices a re-push within a minute. This is the same shape as the min_interval guard on the token refresher from Monday's lesson, applied to a different clock.

The auth flow deserves its own paragraph because it surprises people the first time. Request a manifest without credentials and the registry answers 401 with a WWW-Authenticate: Bearer realm=...,service=...,scope=... header. That header is an instruction. Call the realm URL with the service and scope as query parameters, receive a short-lived token, retry the original request with it. Two round trips, and the token is scoped to one repository, which is why the session must hold a token per repository rather than one token for the registry.

§IIICode Worked Example

The resolver. It performs the challenge dance, caches the token per repository, and resolves a tag to a digest with a hard timeout.

import time
import requests

ACCEPT = ", ".join([
    "application/vnd.oci.image.index.v1+json",
    "application/vnd.oci.image.manifest.v1+json",
    "application/vnd.docker.distribution.manifest.list.v2+json",
    "application/vnd.docker.distribution.manifest.v2+json",
])


class RegistryResolver:
    def __init__(self, host, timeout=1.5, ttl=60.0):
        self.host = host
        self.timeout = timeout
        self.ttl = ttl
        self.session = requests.Session()
        self._tokens = {}
        self._digests = {}

    def _token(self, repo):
        cached = self._tokens.get(repo)
        if cached and cached[1] > time.monotonic():
            return cached[0]
        probe = self.session.get(
            f"https://{self.host}/v2/{repo}/manifests/latest",
            timeout=self.timeout,
        )
        if probe.status_code != 401:
            return None
        challenge = probe.headers["WWW-Authenticate"]
        parts = dict(
            p.split("=", 1) for p in challenge[len("Bearer "):].split(",")
        )
        realm = parts["realm"].strip('"')
        params = {
            "service": parts["service"].strip('"'),
            "scope": f"repository:{repo}:pull",
        }
        resp = self.session.get(realm, params=params, timeout=self.timeout)
        resp.raise_for_status()
        token = resp.json()["token"]
        self._tokens[repo] = (token, time.monotonic() + 240.0)
        return token

    def digest(self, repo, tag):
        key = (repo, tag)
        cached = self._digests.get(key)
        if cached and cached[1] > time.monotonic():
            return cached[0]
        headers = {"Accept": ACCEPT}
        token = self._token(repo)
        if token:
            headers["Authorization"] = f"Bearer {token}"
        resp = self.session.head(
            f"https://{self.host}/v2/{repo}/manifests/{tag}",
            headers=headers,
            timeout=self.timeout,
        )
        resp.raise_for_status()
        digest = resp.headers["Docker-Content-Digest"]
        self._digests[key] = (digest, time.monotonic() + self.ttl)
        return digest

The Accept header is the part worth reading twice. A multi-arch image is an index that points at one manifest per platform, and Poulton's multi-architecture section describes the two-level shape. Ask for the index media type and the registry returns the index digest. Ask only for the platform manifest type and it returns a platform digest. Those are different sha256: strings for the same v1.4.0 tag, and pinning the wrong one produces a Deployment that works on your laptop's architecture and fails on the node pool. Accept the index. The kubelet resolves the platform underneath.

The webhook. It reads the AdmissionReview request, checks each container image, and answers.

import os
from fastapi import FastAPI, Request

app = FastAPI()
REGISTRY = os.environ["TRUSTED_REGISTRY"]
resolver = RegistryResolver(REGISTRY)


def review(uid, allowed, message=None):
    status = {"message": message} if message else {}
    return {
        "apiVersion": "admission.k8s.io/v1",
        "kind": "AdmissionReview",
        "response": {"uid": uid, "allowed": allowed, "status": status},
    }


@app.post("/validate")
async def validate(request: Request):
    body = await request.json()
    uid = body["request"]["uid"]
    spec = body["request"]["object"]["spec"]
    containers = spec.get("containers", []) + spec.get("initContainers", [])

    for c in containers:
        image = c["image"]
        if not image.startswith(f"{REGISTRY}/"):
            return review(uid, False, f"{image}: registry not trusted")
        if "@sha256:" in image:
            continue
        repo, _, tag = image[len(REGISTRY) + 1:].partition(":")
        try:
            digest = resolver.digest(repo, tag or "latest")
        except Exception as exc:
            return review(uid, False, f"{image}: digest unresolved ({exc})")
        return review(
            uid, False,
            f"{image}: pin the digest. Use {REGISTRY}/{repo}@{digest}",
        )

    return review(uid, True)

The rejection message carries the digest the author should have written. A gate that says no teaches nothing. A gate that says no, and here is the string gets obeyed, because obedience costs one copy-paste rather than one shell session.

§IVConnection to Today's Ops Lesson

The Ops lesson names the registry boundary and the digest requirement. This webhook is that boundary made executable, and it closes the gap the Gatekeeper constraint leaves open.

The two also disagree in a useful way. The Gatekeeper constraint is pure evaluation with no I/O, so it cannot fail from a network partition. This webhook resolves digests, so it can. That trade is the reason the registration below sets a short timeout and, on the trading namespace, failurePolicy: Fail:

webhooks:
  - name: digest.hedronite.io
    timeoutSeconds: 3
    failurePolicy: Fail
    namespaceSelector:
      matchLabels:
        supply-chain: enforced

Fail means a registry outage stops deployments into that namespace. That is the correct answer for a trading namespace and the wrong answer for a cluster-wide policy, which is what the namespaceSelector is for. Scope the strictness to the ground that deserves it.

§VPrior-Lesson Reach

Monday's preflight lesson built a TokenRefresher with a minimum interval so a six-hour process never hammers the token endpoint. The _tokens dict here is that guard again, keyed by repository because registry tokens are scoped narrower than ServiceAccount tokens.

Wednesday's configuration lesson argued for resolving every external value once at startup rather than on the hot path. This webhook cannot obey that rule, because the values it needs depend on the request. The cache is the compromise, and naming it as a compromise rather than a design is the honest reading.

The kopf operator from the first of the month reconciles on its own timer and can retry forever. This code gets three seconds. Same cluster, same Python, opposite budget.

§VIClosing

An admission webhook is a promise to answer quickly. Anything the webhook does on the request path becomes a dependency of every write to the cluster, and a registry is a fine thing to depend on until the morning it is not.

So build the resolver, cache what it returns, bound the wait, and scope the strict failure policy to the namespaces where a stalled deploy is cheaper than an unverified image.

Then read your own rejection message. If it does not hand the author the exact string to paste, write it again.