Kubernetes Storage on EKS — three Pendings, three owners

A Pod says Pending. That one word covers three unrelated failures owned by three different controllers.

§I — Frame

A Pod says Pending. That single word covers three unrelated failures owned by three different controllers, and the operator who treats them as one failure will fix the wrong thing.

The scheduler can be unable to place the Pod. The provisioner can be unable to create the disk. The attacher can be unable to bolt the disk onto the node the scheduler picked. Same word on the screen. Three owners, three logs, three fixes.

Call them three Pendings, three owners. Hold the name; the rest of this lesson is the map from each Pending to its owner.

Storage is the domain this arc has walked past six times. The arc took the network at 07-29 and again at 08-01, identity at 08-04, image provenance at 08-07. Every one of those lessons handled a Pod that was running. Today handles the Pod that never starts, and storage is where it most often stops.

§II — Foundations: the persistent volume subsystem

Poulton frames the whole design in one line: Kubernetes gives you a persistent volume subsystem with three objects, and the objects exist so that a workload can ask for storage without naming a disk (Ch. 10, The Kubernetes persistent volume subsystem, p. 131).

Three objects, counted:

1. PersistentVolume (PV). A piece of storage in the cluster. Either an administrator created it by hand, or a provisioner created it in response to a claim.
2. PersistentVolumeClaim (PVC). A request for storage by a workload. Capacity, access mode, and optionally a class. The Pod mounts the claim, never the volume.
3. StorageClass (SC). A named recipe for making PVs on demand. Provisioner, parameters, reclaim policy, binding mode.

The separation is the whole point. A Deployment references a claim by name. That claim resolves to an EBS gp3 volume on EKS, a Persistent Disk on GKE, a Managed Disk on AKS, or a hostPath on a laptop, and the Deployment YAML does not change. This is the same discipline the 08-04 lesson applied to identity: the workload names a ServiceAccount, and the binding to a cloud IAM principal happens outside the manifest.

The Container Storage Interface (CSI) is what makes the recipe portable. Before CSI, every storage vendor's driver was compiled into the Kubernetes tree, which meant a storage bug shipped on the Kubernetes release cycle. CSI moved the vendor code out (Poulton Ch. 10, The big picture, pp. 129-130). On EKS, the AWS EBS CSI driver runs as a Deployment (the controller half) plus a DaemonSet (the node half), and those two halves do genuinely different jobs. Remember the split. It is the difference between the second Pending and the third.

Access modes matter more than candidates expect:

Read the RWO row again. The unit is the node, not the Pod. Two Pods on the same node can share an RWO volume. Two Pods on different nodes cannot, which is why a Deployment with replicas: 3 and one RWO claim gives you one running Pod and two Pending ones on a multi-node cluster.

§III — Mechanism: bind, provision, attach, mount

The claim-to-running path has four steps and each one has a different owner. Trace it once, slowly.

Bind. A PVC is created. The kube-controller-manager PV controller looks for a PV that satisfies it: capacity at least the requested size, matching access mode, matching class. If one exists, the controller writes the binding. The PVC goes Pending to Bound. If none exists and the PVC names a StorageClass with a provisioner, the controller hands off to the provisioner instead.

Provision. The EBS CSI controller sees the claim, calls CreateVolume against the EC2 API, and gets back a volume ID. It writes a PV object describing that volume and binds it to the claim. The AWS-side call is where IAM appears: the controller needs ec2:CreateVolume, and on a properly built cluster it holds that permission through an IAM Roles for Service Accounts binding, exactly the mechanism the 08-04 lesson took apart on GKE's Workload Identity.

Attach. The Pod is scheduled to a node. The AttachDetachController creates a VolumeAttachment object. The EBS CSI controller sees it and calls AttachVolume against EC2, bolting the block device onto that specific EC2 instance.

Mount. The node-half DaemonSet formats the device if it is raw, then mounts it into the kubelet's directory tree and bind-mounts it into the container.

Now the part that decides whether any of this works: **volumeBindingMode**.

An EBS volume lives in exactly one Availability Zone and can attach only to an instance in that same zone. That single sentence is the whole problem.

With volumeBindingMode: Immediate, provisioning happens the moment the claim is created. The provisioner has no idea where the Pod will run, so it picks a zone. If it picks us-east-1a and the scheduler later finds that the only node with room is in us-east-1b, the Pod is unschedulable forever. Nobody made a mistake. The two decisions were made in the wrong order.

Call it the eager disk. The disk was created before anyone asked where the work would happen, so the work must now happen where the disk landed, and it cannot.

With volumeBindingMode: WaitForFirstConsumer, the provisioner does nothing until a Pod referencing the claim is scheduled. The scheduler picks a node using all its normal constraints. Only then does provisioning fire, in that node's zone, by definition correct. The Bootcamp CKA set asks for exactly this construction in Question 14: create a StorageClass with WaitForFirstConsumer, then patch it to become the cluster default, then ensure it is the only default.

That last clause carries weight. Two StorageClasses both annotated storageclass.kubernetes.io/is-default-class: "true" is not an error the API server rejects. It is an ambiguity that resolves unpredictably for every claim that omits a class name.

§IV — Worked Example: three Pendings on one EKS cluster

One cluster. Three node groups, one per AZ. The EBS CSI driver installed as an EKS add-on. Watch the same word mean three things.

Pending the first: the scheduler could not place the Pod

$ kubectl get pod api-7d9f -o wide
NAME        READY   STATUS    NODE
api-7d9f    0/1     Pending   <none>

$ kubectl describe pod api-7d9f | tail -6
Events:
  Type     Reason            From               Message
  ----     ------            ----               -------
  Warning  FailedScheduling  default-scheduler  0/9 nodes are available:
           3 node(s) had volume node affinity conflict,
           6 Insufficient cpu.

volume node affinity conflict is the eager disk, named by the scheduler in plain text. The PV carries a nodeAffinity stanza pinning it to one zone, and no node in that zone has capacity. Owner: the scheduler. Fix: recreate the class with WaitForFirstConsumer, or add capacity in the zone the disk already occupies.

Pending the second: the provisioner could not create the disk

$ kubectl get pvc data-api
NAME       STATUS    VOLUME   CAPACITY   STORAGECLASS   AGE
data-api   Pending                       gp3            4m

$ kubectl describe pvc data-api | tail -5
Events:
  Type     Reason                From                        Message
  ----     ------                ----                        -------
  Warning  ProvisioningFailed    ebs.csi.aws.com_ebs-csi-...  failed to provision volume:
           rpc error: code = Internal desc = Could not create volume "pvc-8c1e":
           UnauthorizedOperation: You are not authorized to perform this operation

The Pod is Pending because the claim is Pending. The claim is Pending because an EC2 API call was refused. Owner: the CSI controller, and behind it, IAM. Note where the evidence lives. describe pod shows only that the Pod waits on a claim; the actual cause is on the PVC's event stream, one object down.

This is the single most useful habit in the whole domain: when a Pod waits on storage, describe the claim, not the Pod.

Pending the third: the disk exists and will not attach

$ kubectl describe pod api-7d9f | tail -5
Events:
  Warning  FailedAttachVolume  attachdetach-controller
           Multi-Attach error for volume "pvc-8c1e" Volume is already exclusively attached
           to one node and can't be attached to another

Here the claim is Bound and the PV exists. The volume is RWO and already attached to the node running the old ReplicaSet Pod, which has not terminated. Rolling a Deployment with an RWO claim produces this reliably, because the default RollingUpdate strategy starts the new Pod before the old one is gone. Owner: the attach/detach controller. Fix: strategy: Recreate, or a StatefulSet, which the Poulton chapter on StatefulSets builds precisely to give each replica its own claim (Ch. 12, pp. 165-167).

Three Pendings. Scheduler, provisioner, attacher. The word on the screen never changed.

A working class for EKS, for reference:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete
parameters:
  type: gp3
  encrypted: "true"

reclaimPolicy: Delete deserves one sentence of caution. Delete the PVC, and the EBS volume is destroyed with it. The Bootcamp's Question 1 exists because of the opposite setting: a Deployment was deleted, the PV was Retain, and the recovery is to write a new PVC that matches the surviving volume's capacity and access mode closely enough to re-bind. Retain is what turns an accident into an inconvenience.

§V — Connection to Prior Lessons

The 08-01 lesson established default-deny NetworkPolicy on EKS and made the argument that an allow-list is only honest when the enforcement point is named. Storage repeats the shape at a different layer: the claim is a request, the class is the policy that answers it, and the cluster default is the answer given to every workload that forgot to ask.

The 08-04 GKE lesson took Workload Identity Federation apart to show a Pod's ServiceAccount reaching a cloud IAM principal. The EBS CSI controller is that same pattern running as infrastructure rather than as application: its ServiceAccount holds an IRSA annotation, and the second Pending above is what a missing IAM permission looks like from inside the cluster.

The 08-07 AKS lesson pinned images to digests so that the thing admitted is the thing that was verified. Storage carries an analogous identity question that admission control cannot answer: a PV is a pointer to state that outlives every Pod that ever mounts it. Nothing in the cluster verifies that the data behind the claim is the data you left there.

§VI — Connection to Today's Dev Lesson

Every diagnosis above was performed the same way: run describe against an object, read the tail of its event list, decide which controller is speaking. Three objects, three event streams, one operator reading them in sequence.

That does not scale past one cluster and one impatient morning. Today's Dev lesson builds the reader: a Python tool that watches the Kubernetes event stream with resourceVersion continuation, correlates events on Pods against events on their claims, and reports which of the three Pendings a workload is actually sitting in. The Ops lesson names the three cases. The Dev lesson makes the naming automatic.

Today's Cert lesson takes the general form of the same skill, since CKA Troubleshooting is thirty percent of the exam and storage is only one of the places the cluster stops.

§VII — Closing

The storage domain is ten percent of the CKA blueprint and considerably more than ten percent of the incidents. The asymmetry comes from the four-step path: bind, provision, attach, mount. Four steps, four owners, one status word covering all of them.

Set WaitForFirstConsumer on every class you create on a multi-AZ cluster and the eager disk stops happening. Describe the claim before the Pod and the second Pending stops hiding. Use Recreate or a StatefulSet whenever an RWO volume meets a rolling update and the third stops surprising you.

Examine the four steps well. Then go look at what your own default StorageClass says, and check that there is exactly one of it.

Related

• Prior arc: Kubernetes Supply Chain Security on AKS
• Domain hub: Cross-References/domains/01-Earth-DevOps
• Grounding tome: The Kubernetes Book (Poulton) (Ch. 10 Kubernetes storage, pp. 129-141)

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura Filed 2026-08-10 at Fajr. Trio #85, sprint day 19, K8s track rung 7.