Kubernetes Admission Control and External Secrets Operator — Defense in Depth for Trust-Coupled Workloads

The cluster remembers the trust posture so the team does not have to.

§ IFrame

A multi-agent production system needs two things that pull in opposite directions. It needs secrets (API keys, database credentials, signing keys) distributed to workloads at the point of use. It also needs a guarantee that no workload can run that violates the trust posture: no privileged containers slipping through, no host-path mounts leaking the node filesystem, no images from unsigned registries. Both must hold under the operational pressure of routine deploys, emergency rollbacks, and the steady churn of agent-pod-restart cycles.

Kubernetes provides two coordinated surfaces for this. Admission control is the gate between an API request and the etcd state, the cluster-level checkpoint that decides whether a manifest is allowed to manifest. External Secrets Operator (ESO) is the bridge between cluster Secrets and an upstream vault, the mechanism that holds the credential lifecycle outside the cluster while keeping the consumption pattern Kubernetes-native. Used together, they encode defense-in-depth at the workload-creation boundary: the secrets path is mediated by an out-of-cluster authority, and every pod that consumes those secrets passes through admission gates that enforce the trust posture.

This lesson refracts Tuesday's secret-management lesson (vaulting, rotation, blast-radius) and last week's workload-identity lesson (SPIFFE, SPIRE, mTLS) through the Kubernetes admission surface. The pairing matters because admission control without secrets discipline produces a hardened cluster that still leaks credentials, and external-secrets without admission control produces a clean credential pipe that runs into pods nobody validated.

§ IIFoundations

The admission control pipeline

A request to the Kubernetes API server passes through four phases before it lands in etcd: authentication, authorization, admission, and validation. The admission phase splits into two sub-phases, mutating admission and validating admission, run in that order. Mutating admission can rewrite the object (inject sidecars, default fields, attach labels). Validating admission can only accept or reject. Both phases run a pipeline of plugins; some are compiled into the API server (always-on), others are attached as webhooks (cluster-operator-configured).

The cluster operator's lever is the webhook. A MutatingWebhookConfiguration or ValidatingWebhookConfiguration resource registers an HTTPS endpoint that the API server calls for objects matching a rule (group, version, resource, operation). The webhook receives an AdmissionReview JSON object, returns an AdmissionReview with allowed: true|false and optional patches, and the API server proceeds or rejects accordingly. The webhook is the place where cluster-specific policy lives.

The model is consequential because it makes policy enforcement a runtime concern of the cluster, not a build-time concern of the application. A team can ship a Helm chart; the cluster's admission webhooks decide whether the chart's pods are allowed to run. The team does not need to remember the rules; the cluster remembers for them. This shift, from remember the rules to the cluster remembers for you, is the operator-discipline coining that distinguishes admission-controlled clusters from convention-controlled clusters.

Pod Security Standards as baseline admission

Kubernetes ships three Pod Security Standards: privileged (no restrictions), baseline (denies known-bad privilege escalations), and restricted (denies anything that has not proven it needs the privilege). Each can be applied at three enforcement levels (enforce, audit, warn) per-namespace via labels:

pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: v1.29
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted

The namespace label is the policy. The API server's built-in PodSecurity admission plugin reads the label at pod creation and rejects, audits, or warns accordingly. No webhook needed. The plugin is GA and compiled in.

Operational discipline: every namespace gets labels at creation. The default (no label) means privileged, which is the wrong default. The cluster operator's first admission policy is to enforce baseline cluster-wide via a Kyverno or OPA Gatekeeper rule that requires the labels, with restricted for production namespaces and baseline for development. This produces a cluster where the trust posture is namespace-visible and audit-friendly.

Validating admission policies (the in-cluster CEL surface)

As of Kubernetes 1.30, the API server supports ValidatingAdmissionPolicy as an in-cluster alternative to validating webhooks. Policies are written in CEL (Common Expression Language) and stored as Kubernetes resources. The API server evaluates the CEL expression at admission time without an out-of-process webhook call, which removes the webhook-availability dependency and reduces admission latency.

A policy that rejects pods with host-path mounts:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: deny-hostpath
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: "!object.spec.volumes.exists(v, has(v.hostPath))"
      message: "host-path volumes are denied in this cluster"

The CEL expression is evaluated server-side at admission. A pod with a host-path volume fails the expression and the API server rejects the request before the pod ever reaches etcd.

The tradeoff against webhook-based policy engines (Kyverno, OPA Gatekeeper) is expressiveness: CEL is intentionally limited (no loops, no recursion, bounded evaluation time), which makes it predictable but constrains the policy shape. For policy that needs cross-resource lookups, external data, or recursive reasoning, webhook engines remain the right tool. For the bulk of cluster trust posture (deny host-path, deny privileged, require image-registry allowlist) CEL is sufficient and cheaper than a webhook.

§ IIIMechanism

External Secrets Operator architecture

ESO runs as a controller in the cluster, watching ExternalSecret custom resources. An ExternalSecret references a SecretStore (or ClusterSecretStore) that defines the upstream: AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault, Azure Key Vault, and others. The controller authenticates to the upstream using a SecretStore-defined identity (IRSA on AWS, Workload Identity on GCP, Vault Kubernetes auth, etc.), fetches the secret values, and writes a native Kubernetes Secret with the fetched data.

The chain has four links and each one carries a security property.

The pod consumer pattern is unchanged from a native-Secrets model. This is the key operational property: application code does not change when the credential source shifts. The cluster's secret-handling code stays Kubernetes-native; the operational discipline shifts to the SecretStore configuration and the IAM policy that scopes the upstream-fetch identity.

Rotation and the refresh interval

ExternalSecret resources carry a refreshInterval field. ESO re-fetches from the upstream at that cadence and updates the cluster Secret. Pods consuming the Secret via env-var do not automatically pick up the change (env vars are set at pod start). Pods consuming via mounted files do see the new content (the kubelet refreshes the projected volume), with a propagation delay of up to a minute.

Rotation discipline: pair refreshInterval with a downstream rotation strategy. For env-var consumers, use a sidecar that watches the Secret and restarts the pod on change (Reloader is a common tool). For file consumers, applications must re-read the file on a signal (SIGHUP) or on a watch. Design the application to consume credentials from files specifically because the watch-and-reread pattern is the only zero-downtime rotation path Kubernetes natively supports.

For ephemeral credentials with short TTLs (Vault dynamic database creds, AWS STS tokens), the refreshInterval is set well below the TTL, typically 1/3 to 1/2 of the lifetime, so the Secret is always fresh enough that consumers see a valid credential. The Sovereign's blast-radius discipline from Tuesday's lesson finds its operational expression here: shorter TTLs require tighter refresh intervals which require disciplined consumer-side rotation patterns.

Encryption at rest

A Kubernetes Secret is base64-encoded, not encrypted. Anyone with read access to etcd reads the plaintext. The cluster operator's responsibility is to configure encryption at rest via an EncryptionConfiguration resource that defines an envelope encryption provider (KMS, age, AES-CBC, or aescbc). The recommended posture is KMS-backed envelope encryption: the cluster generates a data encryption key per Secret, encrypts the Secret with the DEK, encrypts the DEK with a KMS-held key encryption key, and stores both. Disk access alone gives an attacker no plaintext; they would also need KMS-access on the KEK.

ESO does not replace encryption at rest. ESO controls the credential source-of-truth (upstream vault); encryption at rest controls etcd confidentiality (cluster-local). Both must be configured. A common deployment error is to ship ESO under the assumption that secrets are in Vault, etcd does not matter, but the cluster Secret ESO writes is plaintext in etcd unless encryption at rest is on.

§ IVWorked Example

Consider a multi-agent trading system with three workload classes: a market-data ingester that needs database write credentials, an inference-service that needs model-API tokens, and a signing service that needs a private key. The trust posture is:

• Database creds rotate every 24 hours (Vault dynamic role)
• Model-API tokens rotate every 7 days (manual upstream rotation)
• Signing key never rotates (long-lived; protected by separate KMS)
• All pods run under restricted Pod Security Standard
• No pod may mount the host filesystem
• No pod may run privileged
• Only images from registry.hedronite.internal are allowed

The cluster gets four admission policies. First, a Kyverno ClusterPolicy requires Pod Security Standards labels on every namespace and rejects namespace creation without them. Second, a ValidatingAdmissionPolicy written in CEL rejects pods with host-path volumes:

validations:
  - expression: "!object.spec.volumes.exists(v, has(v.hostPath))"

Third, a ValidatingAdmissionPolicy rejects pods with privileged: true:

validations:
  - expression: "!object.spec.containers.exists(c, c.securityContext.privileged == true)"

Fourth, a Kyverno ClusterPolicy rewrites image: fields to require the registry.hedronite.internal prefix; this one needs Kyverno because CEL cannot do string-prefix mutation cleanly.

The cluster gets three SecretStore resources. The first authenticates to Vault via the cluster's service-account JWT and the Vault Kubernetes auth backend. The second authenticates to AWS Secrets Manager via IRSA. The third authenticates to a KMS-backed key store via Workload Identity.

The cluster gets nine ExternalSecret resources, three per workload class, one per credential. The database-cred ExternalSecret has refreshInterval: 6h (well below the 24h TTL). The model-API token has refreshInterval: 24h (well below the 7-day TTL). The signing key has refreshInterval: 168h (weekly; conservative for the long-lived key). All consumers mount the Secret as files; all run a sidecar that watches the file and signals the main container on change.

§ VConnection to Prior Lessons

This lesson sits as the third arrival in the β-Trust week-2 progression. The first (2026-05-20 Workload Identity for Multi-Agent Pipelines) established that workloads must have identities (SPIFFE IDs, mTLS certificates), and that identity is the prerequisite for trust decisions. The second (2026-05-26 Secret Management and Credential Lifecycle) established that credentials must vault, rotate, and limit their blast radius. This third lesson closes the triangle by showing how Kubernetes itself enforces these disciplines at the workload-creation boundary: admission control as the cluster's memory of the trust posture, ESO as the bridge between vault and pod.

The Backward-Synergy-Reach is tight because the three lessons interlock. The workload identity from Lesson 1 becomes the IRSA / Workload-Identity binding that ESO uses to authenticate to the upstream. The credential-rotation discipline from Lesson 2 becomes the refreshInterval and the sidecar-reload pattern. The admission controls from this lesson are the cluster-resident memory that ensures those disciplines hold even when the application team is not paying attention.

§ VIConnection to Today's Dev Lesson

Today's Python lesson examines the secrets and cryptography modules, which sit on the consumer side of the chain: the application code that reads the credential ESO wrote to the Secret, derives keys from it, or compares it against another value in constant time. The Python lesson is the application-side discipline that pairs with this cluster-side discipline; together they describe the full path from upstream vault through cluster admission through application consumption. The cluster controls how the credential arrives; the application controls how the credential is handled once it lands.

Paired Dev lesson → Polyglot-Dev/Python/2026-05-27-pythons-secrets-module-and-cryptography-library-for-trust-coupled-workloads

§ VIIConnection to Today's Cert Lesson

The Wed CKA + CKS combined cert lesson treats the same admission and secrets-at-rest surface from the exam-readiness angle. CKA candidates must know how to enable the built-in PodSecurity plugin and configure EncryptionConfiguration on the kube-apiserver; CKS candidates extend this to author ValidatingWebhookConfiguration resources and design KMS-backed encryption-provider chains with graceful migration paths. The cert lesson is the exam-flavored expression of the same cluster-side mechanisms this Ops lesson described from the operator-discipline angle.

Paired Cert lesson → Archmagus-Stack/09-Tomes/Cert Prep/CNCF/2026-05-27-cka-and-cks-admission-controllers-pod-security-standards-and-secrets-encryption-at-rest

§ VIIIClosing

Admission control and external secrets are a pair because they encode complementary trust-posture decisions. Admission control encodes what may run; external secrets encode what runs may know. Run one without the other and the trust posture has a hole, either a hardened cluster leaking credentials, or a clean credential pipe feeding pods nobody validated. Run both and the cluster carries the trust posture in its own configuration, not in the operational habits of the teams shipping into it. The cluster remembers for them.

Reflect on this.