A Python kopf Operator for RuntimeClass Enforcement — the patch that arrives too late

A mutation after start cannot choose the kernel. The honest timer never patches runtimeClassName on a live Pod.

§I — Frame

The Ops lesson put one field between a process and the host kernel: runtimeClassName: gvisor on the Pod spec. Type it into every manifest and it works until someone ships a Deployment that forgets.

The Bootcamp's Question 10 makes the forgetfulness the whole task. Create a RuntimeClass named sandboxed with handler runsc, then patch every Deployment in a namespace so the Pod template carries the name. Candidates who patch spec instead of spec.template.spec watch the roll stall and blame gVisor. The exam does that work by hand, once, under a clock. Production does it every time a namespace opts in.

Today the operator writes the field. No CustomResourceDefinition. No FastAPI process. Two kopf handlers and a Namespace label.

The label is the switch. The mutate is the decision. The timer is the audit. The failure the timer keeps finding has a name: the patch that arrives too late. Hold it. The rest of this lesson is the map from that name to a 422, an immutable field, and a process that already talked to Container-Optimized OS.

§II — Language Idiom: kopf owns the loop; you own the delta

Dobies and Wood open by calling operators software SREs: a controller that watches a kind you care about and keeps the cluster matching the intent you stored there (Ch. 1, Operators Are Software SREs / How Operators Work, pp. 4-6). The 08-01 lesson took that sentence and made the kind a CRD. That was the right move for a policy object you invented. It is the wrong move for a field Kubernetes already has.

RuntimeClass is a built-in. The Pod spec already carries runtimeClassName. The Namespace already carries labels. Inventing NamespaceBaseline a second time would be a new API for a switch that labels already are.

kopf's own README is blunt about the scope: handlers register on custom resources and on built-in kinds (pods, namespaces); mutating admission is a first-class handler, not an extra server; timers tick for as long as the resource exists; filters match on labels (Features). The 07-26 FastAPI lesson built the AdmissionReview contract, the uid echo, and the fail-closed JSON by hand. The 08-07 lesson did it again so the webhook could phone the registry. kopf owns that HTTP. You write the delta.

Two decorators, counted.

1. **@kopf.on.mutate.** Runs in the API server's admission chain, before the object is persisted. A return or a patch dict becomes the mutation. If the handler is down and failurePolicy is Fail, the create is refused. If it is Ignore, the create lands unstamped. That second setting is how unsandboxed Pods get into a labeled namespace while you are looking at the timer.

1. **@kopf.timer.** Runs on a clock against objects that already exist. A timer does not sit on the request path. It cannot un-start a container. It can notice, emit, and (if you insist) delete so a recreate hits the mutate path. Delete is a recreate. Recreate is a new process. The old process already ran.

Chozanshi's chain, written out: if the guest kernel is chosen when the container starts, then a mutation after start cannot choose it, thus the honest timer never patches runtimeClassName on a live Pod. For this reason the timer's job is inventory, not repair.

Dobies and Wood put the same shape in controller language. A custom controller watches, then creates, updates, or deletes other objects (Ch. 3, Custom Controllers, pp. 27-29). Update is available. Update is not always legal. Pod spec is almost entirely immutable. runtimeClassName is not in the small set of fields a live Pod will accept (image, activeDeadlineSeconds, a handful of others). The API tells you so with a 422. The operator that treats 422 as a retryable error will retry forever.

§III — Code Worked Example: label the namespace, stamp on create, refuse to patch the running

Trading cluster from the Ops lesson. Namespace trading is about to take untrusted batch work. The sandbox pool exists. The gvisor RuntimeClass exists. The missing piece is the vote on every Pod.

The switch is a label, not a CRD:

kubectl label namespace trading sandbox.hedronite.io/enforce=gvisor

The mutate: a decision at create

kopf's label filter on @kopf.on.mutate('pods', labels=...) matches Pod labels. The opt-in lives on the Namespace. The webhook configuration carries a namespaceSelector. That selector is the whole policy. The handler itself stays small.

import kopf

LABEL = "sandbox.hedronite.io/enforce"
WANT = "gvisor"

@kopf.on.mutate("pods", operations=["CREATE"])
def stamp_runtimeclass(spec, patch, namespace, **_):
    if spec.get("runtimeClassName") == WANT:
        return
    patch.spec["runtimeClassName"] = WANT

Three facts in that block.

The handler does not look up the Namespace. The API server already did, because the MutatingWebhookConfiguration carries:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sandbox-runtimeclass
webhooks:
  - name: stamp.sandbox.hedronite.io
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions: ["v1"]
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
    namespaceSelector:
      matchLabels:
        sandbox.hedronite.io/enforce: gvisor
    clientConfig:
      service:
        namespace: sandbox-system
        name: sandbox-operator
        path: /mutate/pods

failurePolicy: Fail is the 08-07 defaultAllow: false in a different object. An unreachable stamper blocks creates in labeled namespaces. Ignore is how a labeled namespace fills with unsandboxed Pods during a rollout of the operator itself.

operations: ["CREATE"] is the other non-negotiable. An UPDATE mutate that tried to stamp a live Pod would be refused by the same immutability rule, and kopf would retry it. Limit the verb. CREATE is the only verb that can still choose a kernel.

kopf in-cluster generates most of that webhook object when you start the operator with the admission server enabled. The namespaceSelector is the piece you must not let the generator omit. Without it the stamper fires on every Pod in the cluster, including kube-system, and the Ops lesson already named what happens when you sandbox kube-dns: the cluster cannot finish installing itself.

The process that serves that webhook is one command:

kopf run sandbox_operator.py --standalone --liveness=http://0.0.0.0:8080/healthz

--standalone skips peering, which is correct for a single replica that must not pause because a laptop is also running kopf run --dev. Dev-mode tunneling is how you iterate on the handler without installing a certificate; production is a Deployment, a Service, and a cert-manager Certificate whose CA bundle is written into clientConfig. The 07-26 lesson spent a section on that TLS path. Reuse it. Do not re-derive it.

RBAC is three verbs on two kinds, and no others.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: sandbox-operator
rules:
  - apiGroups: [""]
    resources: ["namespaces", "pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["namespaces/status"]
    verbs: ["patch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create"]

get/list/watch on Pods and Namespaces is the timer. patch on namespaces/status is the count the timer writes back. create on Events is how kopf publishes logger.warning so kubectl describe namespace trading shows the late Pods. There is no patch on Pods. That omission is the discipline, encoded as a 403 if a later commit adds the 422-producing call. Dobies and Wood treat the Role as part of the operator's surface, not as an afterthought (Ch. 3, Operator Scopes, and Appendix C on RBAC). Scope the Role to what the handler actually does, then let the missing verb catch the next mistake.

The 08-04 lesson's SelfSubjectAccessReview preflight belongs here as a startup check: on boot, can_i against pods list and namespaces/status patch. Exit 78 if either is missing. An operator that discovers its RBAC is wrong at first timer tick has already missed every create that happened while it was crashing.

The timer: inventory of what already started

Labeling trading does nothing to the Pods that are already Running. They started against the host kernel. The timer's job is to say so, every thirty seconds, until a human (or a delete) makes a new create.

from kubernetes import client

v1 = client.CoreV1Api()

@kopf.timer("namespaces", interval=30.0, labels={LABEL: WANT})
def catch_unsandboxed(name, logger, **_):
    pods = v1.list_namespaced_pod(name)
    late = []
    for pod in pods.items:
        if pod.spec.runtime_class_name == WANT:
            continue
        late.append(pod.metadata.name)
        logger.warning(
            "pod %s/%s running with runtimeClassName=%r; "
            "spec.runtimeClassName is immutable, a patch cannot sandbox it",
            name,
            pod.metadata.name,
            pod.spec.runtime_class_name,
        )
    return {"unsandboxed": late, "count": len(late)}

kopf persists that return value onto the Namespace status. kubectl get namespace trading -o jsonpath='{.status}' then shows the count. A count of zero means every Pod in the namespace took the mutate path. A count above zero is the patch that arrived too late, named as inventory.

The patch people reach for anyway:

v1.patch_namespaced_pod(
    pod.metadata.name,
    name,
    {"spec": {"runtimeClassName": "gvisor"}},
)

The API answers 422. Forbidden: pod updates may not change fields other than spec.containers[*].image, spec.containers[*].resources, .... runtimeClassName is not on the list. Retrying the 422 is how an operator burns its rate limit on a law of the API.

Delete-and-recreate is the remaining move, and it is a different decision. A Deployment will make a new Pod. The new Pod hits the mutate. The new container talks to runsc. The old container, between the delete signal and the process exit, still talks to the host kernel. That window is the Cert lesson's territory: Falco is what fires on the syscall the guest kernel would have refused. The operator cannot close a window that already opened. It can refuse to pretend a patch closed it.

What the operator does not do

It does not create the RuntimeClass. GKE already has gvisor. The Bootcamp asks you to create sandboxed with handler runsc because the exam cluster does not. Detect the environment; do not fight it.

It does not taint nodes or create the sandbox pool. Those are cluster-admin acts from the Ops lesson. An operator that tries to be the pool will sandbox the default pool and take kube-system with it.

It does not install a NetworkPolicy. That was 08-01, and the CRD there was the right shape because the policy objects did not exist until the operator wrote them. RuntimeClass already exists. The field already exists. The operator writes one string onto one field, at the one moment the string still works.

§IV — Connection to Prior Lessons

The 08-01 kopf lesson installed a default-deny pair from a NamespaceBaseline CRD. The reconcile target was an object the cluster would not have otherwise. Today's reconcile target is a field on an object the cluster already admits. Reusing the CRD shape would add a second source of truth next to the Namespace label, and the two would drift. The label is enough.

The 07-26 FastAPI webhook stood in the request path and judged. The 08-07 webhook stood in the same path and phoned the registry. Both spent most of their lines on AdmissionReview marshalling, uid echo, and the 200-with-allowed-false contract. kopf's mutate handler is those lines in the framework. The remaining code is the one assignment to patch.spec. If you find yourself parsing AdmissionReview today, you have rebuilt 07-26 by accident.

The 08-10 watch lesson built a generator that survives 410 Gone and correlates two event streams. A timer is not a watch. A timer is a poll. Thirty seconds of lag is acceptable for inventory and fatal for admission. Do not fold the stamper into the timer to "simplify." The stamper is on the create path because that is the only path that still chooses a kernel.

The 08-04 in-cluster config and projected-token refresh still apply. This operator runs for days. config.load_incluster_config() once at startup, and the projected token's refresh, are what keep the timer's list calls from going 401 at the one-hour mark. The 08-01 timer had the same dependency; it did not get any less true.

§V — Connection to Today's Ops and Cert Lessons

Ops named the placement. Autopilot votes per Pod. Standard votes per pool, with a taint the RuntimeClass tolerates. Privileged, hostPath, custom seccomp, port-forward, NET_RAW, and the metadata server sit on the other side of the name. The operator writes the name. It does not negotiate the incompatibility list. A Pod that asks for privileged: true and gets runtimeClassName: gvisor stamped on it will still be refused by GKE. That refusal is correct. The operator's job is to stamp, not to strip privileged. Stripping is a different mutate, and mixing the two in one handler hides which rule fired.

Cert is the other half of the same wall. RuntimeClass is create-time isolation. Falco, the audit policy, and readOnlyRootFilesystem are what you still need after the process is running, because the guest kernel is a smaller kernel, not an omniscient one. The timer's inventory of unsandboxed Pods is the input to that half: those are the processes whose syscalls still reach the host. The syscall that should not have happened is the Cert lesson's name for the event Falco exists to catch, and it is also the event the late patch failed to prevent.

Question 22 of the Bootcamp is the proof the mutate is aiming for. dmesg inside the Pod prints Starting gVisor instead of the COS boot log. The operator does not run dmesg. The operator writes the field that makes dmesg come out that way. Verify on a canary Pod after the first stamp, the way the Ops lesson did, before you label every namespace.

The verification sequence is four commands, in order, and the fourth is the only one that proves isolation rather than placement.

kubectl get pod batch-untrusted -n trading -o jsonpath='{.spec.runtimeClassName}'
kubectl get pod batch-untrusted -n trading -o jsonpath='{.spec.nodeName}'
kubectl get namespace trading -o jsonpath='{.status}'
kubectl exec -n trading batch-untrusted -- dmesg | head

The first should print gvisor. The second should be a sandbox-pool node. The third should show unsandboxed: []. The fourth should print Starting gVisor. If the first is gvisor and the fourth prints the COS kernel, the RuntimeClass name exists and the handler does not, which is the Ops lesson's "name with no handler on that node is a Pod that will never start," except here it started, which means it started on a node that ignored the name. Stop. Check the pool taint before you label another namespace.

§VI — Closing

The Namespace label is the switch. The mutate is the only write that still works. The timer is a count of writes that no longer would.

Do not patch a running Pod's runtimeClassName. The API will refuse, and kopf will retry a refusal that cannot succeed. Delete if you mean to recreate. Recreate if you mean to sandbox. Accept that the original process already ran.

Set failurePolicy: Fail on the webhook. Set operations: ["CREATE"]. Put the namespaceSelector on the webhook, not a Pod-label filter on the handler. Then label one namespace, create one Pod, and read dmesg until it says gVisor.

Examine the 422. Then check the timer's count.

Related

• Prior arc: A Kubernetes Operator in Python with kopf
• Language hub: Cross-References/dev-languages/Python
• Grounding tome: Kubernetes Operators (Dobies and Wood) (Ch. 1 How Operators Work, pp. 4-6)
• Paired Ops: Kubernetes Runtime Isolation on GKE
• Paired Cert: CKS Runtime Security

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura Filed 2026-08-13 at Fajr. Trio #88, sprint day 22, K8s track.