Python and the Kubernetes Authorization Surface — ask before you act

Four controllers into this arc, and not one of them ever asked whether it was allowed.

§ IFrame

Look at what the K8s-day Dev slot has built. A kopf controller that patches a Tenant CRD. A FastAPI admission webhook that rejects privileged pods. A watch-driven service registry following EndpointSlices. An operator reconciling a NetworkPolicy baseline into labeled namespaces.

Four programs, four credentials, zero questions asked. Every one called config.load_incluster_config() and started working, and every one would have discovered a missing RBAC verb the same way: a 403 mid-reconcile, in whatever hour the workload hit that code path.

Today's Ops lesson made identity the subject. This lesson takes it into the client and answers three questions in code: where the credential comes from, how it stays valid across a long-running watch, and how a program asks am I allowed before it acts. The coin: ask before you act.

§ IIWhere the Credential Comes From

Burns et al. describe two loading paths (ch. 13, pp. 219-220). Outside the cluster the client reads ~/.kube/config, resolves the current context, and builds credentials from what that context names. Inside the cluster it reads the projected ServiceAccount volume the kubelet mounted at a fixed path.

from kubernetes import client, config
from kubernetes.config.config_exception import ConfigException


def load_kube() -> client.ApiClient:
    try:
        config.load_incluster_config()
    except ConfigException:
        config.load_kube_config()
    return client.ApiClient()

The ordering matters. In-cluster first means a program deployed to a cluster never picks up a developer's kubeconfig baked into the image.

What the in-cluster loader reads is three files and two environment variables, all under /var/run/secrets/kubernetes.io/serviceaccount/: the token, the cluster CA at ca.crt, and the pod's namespace. The API server address comes from KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, injected by the kubelet. Nothing here is magic and all of it is readable, which is why the Ops lesson's automountServiceAccountToken: false matters for workloads with no business holding a credential (Burns et al., ch. 14, p. 247).

§ IIIThe Refresh Problem

A projected token is bound to a lifetime, commonly one hour, and the kubelet rewrites the file in place before expiry. The Python client reads the token once at load time and holds the string on the configuration object. A process running six hours on a one-hour token starts getting 401 Unauthorized in hour two, and the traceback points at whatever call happened to be next rather than at the credential.

Recent client versions rotate internally on the standard in-cluster path. Relying on that silently is the mistake; an explicit refresh is four lines and removes the class.

import time
from pathlib import Path

TOKEN_PATH = Path("/var/run/secrets/kubernetes.io/serviceaccount/token")


class TokenRefresher:
    def __init__(self, cfg: client.Configuration, min_interval: float = 300.0):
        self._cfg = cfg
        self._min_interval = min_interval
        self._last = 0.0

    def refresh(self) -> None:
        now = time.monotonic()
        if now - self._last < self._min_interval:
            return
        if TOKEN_PATH.exists():
            self._cfg.api_key["authorization"] = "Bearer " + TOKEN_PATH.read_text().strip()
            self._last = now

Call refresh() at the top of each reconcile pass and the held credential never drifts more than five minutes behind the file on disk. The min_interval guard keeps a hot loop from re-reading thousands of times a second, which matters because the 07-29 watch loop and the 08-01 timer both run at rates where an unguarded read shows up in a flame graph.

The model to keep is the Ops lesson's: the token is a lease, not a key. Code treating it as a key works in testing and breaks after an hour of uptime.

§ IVWorked Example — The Startup Preflight

kubectl auth can-i is not a client-side computation. It posts a SelfSubjectAccessReview to the API server, which runs its own authorizer against the caller's identity and returns a verdict. Poulton's pages describe the union-of-bindings evaluation producing that verdict (pp. 178-180); the review API is how a program reads the result instead of reimplementing the union.

from kubernetes import client


def can_i(verb: str, resource: str, namespace: str, group: str = "") -> bool:
    api = client.AuthorizationV1Api()
    review = client.V1SelfSubjectAccessReview(
        spec=client.V1SelfSubjectAccessReviewSpec(
            resource_attributes=client.V1ResourceAttributes(
                group=group,
                resource=resource,
                verb=verb,
                namespace=namespace,
            )
        )
    )
    result = api.create_self_subject_access_review(review)
    return bool(result.status.allowed)

Two properties make this safe to call anywhere. The review always evaluates the caller's own identity, so no impersonation and no elevated permission are involved. And every authenticated principal may create one, so the preflight never needs an RBAC grant of its own. A program can always ask what it may do.

import logging
import sys

REQUIRED = [
    ("get", "networkpolicies", "networking.k8s.io"),
    ("create", "networkpolicies", "networking.k8s.io"),
    ("patch", "networkpolicies", "networking.k8s.io"),
    ("list", "namespaces", ""),
    ("patch", "tenantbaselines", "hedronite.io"),
]

log = logging.getLogger("preflight")


def preflight(namespace: str) -> None:
    missing = [
        f"{verb} {resource}"
        for verb, resource, group in REQUIRED
        if not can_i(verb, resource, namespace, group)
    ]
    if missing:
        for item in missing:
            log.error("rbac_missing", extra={"permission": item, "namespace": namespace})
        sys.exit(78)
    log.info("preflight_ok", extra={"checked": len(REQUIRED), "namespace": namespace})

REQUIRED is the interesting artifact, more than the function around it. It is the controller's RBAC contract written in the controller's own repository, beside the code depending on it, in a form that runs. The Role YAML in the Helm chart drifts the first time somebody adds an API call and forgets the chart. This list cannot drift, because a mismatch stops the process on the next deploy with the exact missing verb in the log line.

Exit code 78 is EX_CONFIG from sysexits.h: the configuration is wrong, not the program. A Deployment with that code lands in CrashLoopBackOff with a log line naming the missing permission. Compare that against a 403 at 3am inside a handler that fails closed. Same failure. One of them tells you what to fix, at the moment you deployed the thing that broke it.

This is the 08-02 three-witnesses discipline arriving on the K8s track: the log line, the exit code, and the preflight producing both before any work begins.

§ VTwo Traps

SelfSubjectRulesReview is the wrong tool for a checklist. It returns every rule applying to the caller in a namespace, which sounds better and is worse. The returned rules carry wildcards and aggregated ClusterRole expansions, so matching them yourself means reimplementing the authorizer's union logic in Python. The docs say plainly the result is for display, not for access decisions. Render a debug endpoint with it; decide with SelfSubjectAccessReview.

Namespace-scoped checks say nothing about cluster scope. Calling can_i("list", "namespaces", "trading") asks a malformed question: namespaces is cluster-scoped and the namespace field is ignored for it. Pass an empty namespace for cluster-scoped resources and read the answer as cluster-wide. Backwards, this produces a preflight that passes and a controller that still cannot list namespaces.

§ VIClosing

Four controllers assumed. The fifth asks. Three lines of change to any of the prior four: load config through the fall-back helper, refresh the token at the top of the loop, run preflight() before the first watch opens. What comes back is a program that states its permission requirements, verifies them against the live authorizer, and refuses to start rather than half-run.

Take the operator you built on Saturday. Write its REQUIRED list from the API calls in its handlers, without looking at its Role YAML. Then compare the two. The delta is what has been over-granted or under-declared since the day it shipped.