Kubernetes Storage on GKE — the volume that survived the claim

The claim is gone. The disk is still in Compute Engine. kubectl get pv still names it.

<!-- hal:authoritative:yaml -->

The claim is gone. The disk is still in Compute Engine. kubectl get pv still names it.

§I — Frame

Monday 08-10 opened storage on EKS and named three Pendings with three owners. Saturday 08-16 named the Pending that is not storage: a taint, a request, a PriorityClass. Wednesday 08-19 encrypted Secrets on EKS. Today the arc returns to storage, and the failure is no longer Pending.

You deleted the PVC. The Pod is gone. kubectl get pvc is empty. On a GKE cluster whose class is premium-rwo or standard-rwo, the Persistent Disk is gone too. Delete is the default reclaim. Poulton is blunt: Delete is the most dangerous policy, and it is the default for PVs created dynamically via StorageClasses unless you specify otherwise (Ch. 10, Reclaim policy, p. 128).

Change one field. reclaimPolicy: Retain. Delete the same PVC. kubectl get pv still lists the volume. Google Cloud Console, Compute Engine, Disks still shows the 20G regional disk. That is Poulton's clean-up on a regional GKE cluster (Ch. 10, p. 133). The claim died. The volume lived.

Coin it: the volume that survived the claim.

This is not 08-10. That lesson isolated the owner of a Pending. Today isolates the owner of a leftover. The leftover has two halves. A PV in Released still carries claimRef for a claim that does not exist. A GCE Persistent Disk still bills. A new PVC that "matches" stays Pending until you clear the leftover pointer. The exam names this Q1. The cloud names it a disk you forgot to delete.

§II — Foundations: two policies, one leftover pointer

Poulton counts two reclaim policies (Ch. 10, p. 128). Recycle is gone from the current API. Do not write it on a manifest.

Delete. When the last PVC that bound the PV is removed, Kubernetes deletes the PV object and asks the CSI driver to delete the back-end asset. On GKE that asset is a Compute Engine Persistent Disk. pd.csi.storage.gke.io is the provisioner on premium-rwo and standard-rwo. The in-tree kubernetes.io/gce-pd still sits on the class named standard. All three that GKE creates at cluster birth use Delete (Ch. 10, p. 129). Data is gone. The bill is gone. That is the success case for a throwaway environment and the failure case for a database.

Retain. The PV object stays. The disk stays. Other PVCs are prevented from using it. Manual clean-up is required. Poulton's new class sc-fast-repl is this policy on purpose: pd.csi.storage.gke.io, type: pd-ssd, replication-type: regional-pd, volumeBindingMode: WaitForFirstConsumer, reclaimPolicy: Retain (Ch. 10, p. 132).

The leftover is not only the disk. After the PVC is deleted, the PV phase becomes Released. spec.claimRef still names the dead claim: namespace, name, uid. The binder will not give that PV to a new PVC while claimRef is set. Q1 LabSetUp patches it away so the candidate can bind again. Production does the same patch, or deletes the PV and the GCE disk by hand, or leaves both as a backup you intended.

K8sUR3 Ch. 12 Caution is the sibling warning. Automatic provisioning is easy. The default reclamation binds the volume's life to the life of the thing that created it. Delete a Pod the wrong way and the volume can go with it (pp. 203-204). Their wording names the PersistentVolumeClaim as the policy holder. The API stores persistentVolumeReclaimPolicy on the PV. StorageClass reclaimPolicy is what the provisioner copies onto the PV at create. Read the PV. Do not hunt the claim for a policy the claim does not own.

Three objects still hold, as 08-10 already taught.

1. PersistentVolume. Cluster-scoped. Capacity, access mode, reclaim policy, class, and either a static handle or a CSI volume handle.
2. PersistentVolumeClaim. Namespaced. The ticket the Pod mounts. Never the disk name.
3. StorageClass. The recipe. Provisioner, parameters, reclaim policy, binding mode, default annotation.

Today's work is the first object's life after the second object dies.

Access modes still constrain the disk you can keep. Poulton lists RWO, RWM, ROM (Ch. 10, p. 128). A GCE Persistent Disk is RWO. Two Pods on different nodes cannot share it. Regional PD replicates across zones; it does not become RWX. If the second replica needs the same bytes, you are on Filestore or you are on a single writer. Retain does not change the mode.

§III — Mechanism: GKE PD CSI and the class you actually have

Poulton's hands-on "will only work on a regional Google Kubernetes Engine (GKE) cluster with the CSI plugin installed" (Ch. 10, p. 129). That sentence is today's overlay. The last two K8s Ops visits sat on AKS (08-16) and EKS (08-19). GKE is the rebalance, and it is the cluster the tome actually types.

kubectl get sc on a typical GKE cluster:

NAME            PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE
premium-rwo     pd.csi.storage.gke.io          Delete          WaitForFirstConsumer
standard        kubernetes.io/gce-pd           Delete          Immediate
standard-rwo    pd.csi.storage.gke.io          Delete          WaitForFirstConsumer

Read four columns.

Provisioner. Two CSI classes, one leftover in-tree class. New work uses pd.csi.storage.gke.io. The in-tree kubernetes.io/gce-pd on standard is the old driver. Do not write new YAML against it unless the question shows that name.

Reclaim. Delete on all three. A PVC against premium-rwo that you delete will take the SSD with it. If the data must survive the claim, you need a class you created, like Poulton's sc-fast-repl, or a static PV whose persistentVolumeReclaimPolicy is already Retain.

Binding mode. WaitForFirstConsumer on the CSI classes. Immediate on standard. 08-10 taught this as the owner of a Pending on EKS. On GKE it is also why a PVC against premium-rwo sits Pending until a Pod mounts it (Ch. 10, pp. 130-131). That Pending is not today's failure. Today's failure starts after Bound.

Default. standard is marked default on Poulton's cluster. A PVC that omits storageClassName lands there: in-tree, Immediate, Delete. Q14 is the exam form of this column. Two defaults is a broken cluster. Zero defaults plus an omitted class is a Pending claim. One default is a loaded gun. The class you did not name still runs.

kubectl describe sc premium-rwo adds the parameters (Ch. 10, p. 130): type=pd-ssd, AllowVolumeExpansion: True, ReclaimPolicy: Delete, VolumeBindingMode: WaitForFirstConsumer. The regional replica class adds replication-type: regional-pd. A zonal pd-balanced disk cannot move with a Pod that scheduled in the other zone. That is volume node affinity, the Storage sentence 08-16 Q6 told you not to spend the scheduling 15% on. Describe the PVC. Then look at the disk's zone.

The CSI split still matters, even though 08-10 already named it. Controller half creates and deletes disks. Node half attaches and mounts. Retain stops the controller half from deleting. It does not stop a node from detaching when the Pod dies. The disk is unattached and still present. Compute Engine will keep billing it.

K8sUR3 isolates the Pod from the disk on purpose (Ch. 12, pp. 199-200). A claim keeps the Pod spec cloud-agnostic. Swap nfs for gcePersistentDisk and the Pod YAML does not change. Dynamic provisioning is the same isolation with the class as the cloud-specific half (pp. 202-203). Their Azure default-class example is a different vendor and the same gun. Today's referent stays GKE because that is the cluster Poulton's YAML will apply on.

§IV — Worked example: delete the claim, keep the disk

Start from Poulton's sc-fast-repl (Ch. 10, p. 132).

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: sc-fast-repl
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd
  replication-type: regional-pd
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain

Apply it on a regional GKE cluster. Deploy a 20Gi PVC that names sc-fast-repl and a Pod that mounts the claim. WaitForFirstConsumer holds the PVC Pending until the Pod exists. Then the provisioner creates a regional Persistent Disk and a PV. kubectl get pv shows RECLAIM POLICY Retain and STATUS Bound. Console, Compute Engine, Disks shows the 20G regional disk with the same name as the PV (Ch. 10, p. 131 for the premium-rwo analog, p. 133 for this class).

Delete the Pod and the PVC together.

kubectl delete -f vol-app.yml

kubectl get pvc is empty. kubectl get pv is not. The PV is Released. spec.claimRef still points at default/pvc2 (or whatever name you used). The disk is still in the project.

A new PVC that requests 20Gi, RWO, and storageClassName: sc-fast-repl will not bind. The leftover pointer is the reason. The binder treats a Released PV with a claimRef as taken.

Two recoveries.

Reuse. Patch away claimRef.

kubectl patch pv <name> --type=json -p '[{"op":"remove","path":"/spec/claimRef"}]'

Phase becomes Available. Create a new PVC. Optionally set spec.volumeName to the PV name so the binder cannot pick a different volume. Q1 does this for MariaDB: 250Mi, RWO, reuse the one retained PV, fill claimName on the Deployment. The data in /var/lib/mysql is still on the volume because Retain kept the bytes. LabSetUp already performed the patch so the candidate starts from Available. In production you will do that patch yourself, or you will discover a Pending PVC and a Released PV that still names last Tuesday's claim.

Abandon. Delete the PV object, then delete the GCE disk in the console or with gcloud compute disks delete. Failure to delete the regional disk "may result in unexpected charges" (Poulton Ch. 10, p. 133). The volume that survived the claim is also the invoice that survived the claim.

If the class had been premium-rwo, both recoveries are unavailable. Delete already removed the PV and the disk. There is nothing to patch. That is why the class field is the decision, and why you read kubectl get sc before you delete a PVC you care about.

K8sUR3's static NFS example is the other create path (Ch. 12, Example 12-4). An administrator writes a PV by hand, gcePersistentDisk or a CSI volume handle filled in, persistentVolumeReclaimPolicy: Retain set on the PV itself. No StorageClass required. Q1 LabSetUp is this path: a hostPath PV with Retain, a default class name copied on so a PVC without storageClassName can still bind. The exam cluster is not GKE. The leftover pointer is the same object.

A third failure sits between the two recoveries. You delete the PV object and leave the GCE disk. Kubernetes no longer has a handle. Compute Engine still does. kubectl get pv is a clean empty. The bill is not. The day's Dev lesson can list Released PVs. It cannot see a disk whose PV you already deleted. The third list is still Console, Compute Engine, Disks.

Four ways this fails in the same hour.

You read the PVC, not the class. kubectl get pvc is empty, so you tell the room the data is gone. The class was Retain. The PV is Released. The disk is in the project. The room is now restoring from a snapshot you did not need.

You patch claimRef and omit volumeName. A second unused PV in the same class is Available. The binder gives the new PVC that empty volume. MariaDB starts clean. The survived bytes sit on the first PV, still Released, still unnamed by any claim. Q1 SolutionNotes sets volumeName: mariadb-pv for this reason.

You keep the default class. The PVC never named sc-fast-repl. It landed on standard, Immediate, Delete, in-tree. Retain YAML you applied last week is a class nobody used. The disk is already gone.

You expand and then delete. AllowVolumeExpansion: True on premium-rwo lets you grow a Bound PVC. Growing does not change reclaim. A 200Gi disk that survived the claim costs more than the 20Gi you meant to keep. Expansion is a size decision. Reclaim is a life decision. They do not commute.

§V — Connection to prior lessons

08-10 on EKS taught volumeBindingMode as the owner of a Pending. The Pod was waiting for a disk that did not exist yet, or a disk that existed in the wrong AZ. Today's GKE CSI classes default to WaitForFirstConsumer, so that Pending still happens. It is the front half of the same object. The back half is reclaim. Do not spend this fire re-teaching Immediate versus Wait. Spend it on what happens after Bound.

08-16 on AKS taught the Pending that is not storage. Q6 in that Cert file asked which domain owns had volume node affinity conflict. The answer was Storage, taken on 08-10 Ops. A GKE zonal PD in us-central1-a with a Pod that can only land in us-central1-b is that event. Regional PD (replication-type: regional-pd) is the class-level escape Poulton chose for sc-fast-repl. It is not a scheduler fix.

08-19 on EKS taught a leftover of a different kind: a Secret whose etcd bytes were never rewritten. Same shape. An enablement (encryptionConfig, reclaim Retain) does not rewrite the past by itself. Encryption needs replace. Retain needs a patch or a disk delete. The console saying "encrypted" was not a census. kubectl get pvc saying empty is not a census of Compute Engine.

08-13 GKE Sandbox and 08-04 GKE Workload Identity stay adjacent. They are GKE, not storage. 08-21 is Terraform Azure. 08-20 is a Python MIG. Neither is this fire.

§VI — Connection to today's Dev and Cert

Dev is a Python reader. CoreV1Api.list_persistent_volume for status.phase, spec.persistent_volume_reclaim_policy, and spec.claim_ref. It prints Released volumes whose claim is gone. It does not patch. It does not call compute.disks.list unless you hand it a project and ask for a second census. The Kubernetes object is not the GCE disk. A checker that only lists PVs can miss a disk whose PV was deleted by hand.

Cert is CKA Storage 10%, the domain 08-16 left for a Cert revisit. Q1 is reuse: create the PVC, optionally volumeName, fill claimName, MariaDB comes back on the same 250Mi. Q14 is the default class: create local-storage with rancher.io/local-path and WaitForFirstConsumer, do not make it default in the first apply, then patch so it is the only default. The exam will not say GKE. The default-class gun and the leftover pointer are the same two facts.

§VII — Close

Storage on this arc now has both halves. 08-10 owned the Pending. Today owns the leftover. The class you choose is the decision. Delete spends the disk with the claim. Retain leaves the volume that survived the claim, a Released PV with a stale claimRef, and a GCE disk that still bills.

Read kubectl get sc before you delete a PVC. Read kubectl get pv after. Then open Compute Engine, Disks. The third list is the one that charges.

Examine well. The pointer is on the PV. The disk is in the project. Neither list is the other.

Related

• Prior arc: Kubernetes Storage on EKS (2026-08-10)
• Pair hub: Cross-References/domains/01-Earth-DevOps
• Grounding tome: Poulton Ch.10, Reclaim policy / GKE hands-on / Retain clean-up, pp. 128-133