Python and the Kubernetes PV Surface — the volume that survived the claim
The client lists PersistentVolumes. Released still names a claim that is gone. Your loop is not a patch.
<!-- hal:authoritative:yaml -->
The client lists PersistentVolumes. Released still names a claim that is gone. Your loop is not a patch.
§I — Frame
The day's Ops lesson puts reclaimPolicy: Retain on a GKE StorageClass and names the failure: the volume that survived the claim. Delete the PVC. The PV stays. Compute Engine still has the disk. spec.claimRef still points at Tuesday. A new PVC that matches capacity and class stays Pending.
This slot is Python touching K8s. k8s_day_counter is even, so Cert is CKA Storage, and Dev stays on the client. 08-10 already watched events for a Pending Pod. 08-16 already predicted a scheduler no and refused to bind. 08-19 already read encryptionConfig and could not see etcd. 08-13 already mutated a RuntimeClass with kopf. None of those tools answer the question Ops just asked: which PersistentVolumes are Released, which still name a dead claim, and which reclaim policy made that leftover legal?
Call the failure the checker that must not patch. A first draft will list Released PVs and patch away claimRef so the next PVC can bind. That is Q1 LabSetUp. It is not a census. LabSetUp clears the pointer so a candidate can practice the bind. A cluster-wide loop that does the same thing rebinds every leftover volume to the next matching claim. MariaDB lands on the empty disk. The survived bytes sit unnamed.
Today's tool does one read. CoreV1Api.list_persistent_volume for status.phase, spec.persistent_volume_reclaim_policy, and spec.claim_ref. It prints Released volumes. It flags a claimRef whose PVC is gone. It does not patch. It does not create. It does not open a watch. It does not call Compute Engine unless you hand it a project and ask for a second census, and even then the second census is a different client.
§II — Language idiom: the generated CoreV1 object
K8sUR3 Ch. 13 is blunt about how you talk to this API. The OpenAPI spec is the source. The generated client libraries are how a language consumes it (p. 216). Python's kubernetes package is that generated client. CoreV1Api is the core group. PersistentVolume is a core v1 object, cluster-scoped, same group as Pod and Node. There is no StorageV1Api.list_persistent_volume. StorageClass lives on StorageV1Api. Do not mix the two clients and then wonder why the method is missing.
Load config the way 08-04 already taught. In-cluster uses the projected token. Out-of-cluster uses kubeconfig. Today's tool does not care which, so long as the ServiceAccount can get and list persistentvolumes. A write verb is a bug.
from kubernetes import client, config
def core() -> client.CoreV1Api:
try:
config.load_incluster_config()
except config.ConfigException:
config.load_kube_config()
return client.CoreV1Api()The list call returns V1PersistentVolumeList. Each item is a V1PersistentVolume. The fields you need are attributes, not dict keys, unless you call to_dict(). Prefer the attributes. They fail loud when the generated class changes. A dict silently returns None for a typo.
from dataclasses import dataclass
from typing import Optional
@dataclass(frozen=True)
class Leftover:
name: str
phase: str
reclaim: str
claim_ns: Optional[str]
claim_name: Optional[str]
storage_class: Optional[str]
capacity: Optional[str]
claim_missing: boolclaim_missing is the finding. A Released PV with a claimRef is expected after a Retain delete. A Released PV whose PVC still exists is a race. A Bound PV whose PVC is gone is a lie you should not see. The tool reports the third as loudly as the first.
StorageClass is the other read, and it is a different API object.
def classes(api: client.StorageV1Api) -> dict[str, str]:
out = {}
for sc in api.list_storage_class().items:
policy = sc.reclaim_policy or ""
out[sc.metadata.name] = policy
return outThe class policy is the recipe. The PV policy is the copy the provisioner wrote at create. If they disagree, someone patched one of them later. Report both. Do not "fix" them.
Pagination is not optional on a cluster that has lived a year. list_persistent_volume returns a page. metadata._continue is the token. Loop until it is empty. A one-shot list that prints three leftovers and misses the other forty is a worse report than no report.
def list_all_pvs(api: client.CoreV1Api) -> list:
items = []
_continue = None
while True:
kwargs = {"limit": 100}
if _continue:
kwargs["_continue"] = _continue
page = api.list_persistent_volume(**kwargs)
items.extend(page.items)
_continue = page.metadata._continue
if not _continue:
break
return itemslimit without the loop is how you hide the oldest Released PV. The oldest one is the disk that has been billing since March.
ResourceVersion belongs to 08-10. This tool does not watch. If you add a watch because you want "live leftovers," you inherit 08-10's contract: start from a list, then watch from that resourceVersion, then relist on a 410. Do not add it today. A leftover is not a stream. It sits.
to_dict() looks convenient and lies about names. The generated field is persistent_volume_reclaim_policy in Python and persistentVolumeReclaimPolicy in JSON. to_dict() may emit either depending on package version. Read the attribute. When you print, you pick the JSON key on purpose, once.
A V1ObjectReference for claim_ref has name, namespace, uid, resource_version, and sometimes api_version plus kind. Name plus namespace is the human string. Uid is the identity. If a candidate recreates mariadb/mariadb after deleting the first PVC, the name pair matches and the uid does not. read_namespaced_persistent_volume_claim then returns 200. claim_missing is false. The PV is still Released and still points at the old uid. Print the uid. The false claim_missing is not a bug. It is the new claim, which still cannot bind until the pointer dies.
§III — Code worked example: list the leftovers
The read itself is small. The discipline is in what you refuse to do after it.
def leftovers(core_api: client.CoreV1Api) -> list[Leftover]:
pvs = core_api.list_persistent_volume().items
found = []
for pv in pvs:
ref = pv.spec.claim_ref
claim_ns = ref.namespace if ref else None
claim_name = ref.name if ref else None
missing = False
if claim_ns and claim_name:
try:
core_api.read_namespaced_persistent_volume_claim(
claim_name, claim_ns
)
except client.ApiException as exc:
if exc.status == 404:
missing = True
else:
raise
cap = None
if pv.spec.capacity:
cap = str(pv.spec.capacity.get("storage"))
found.append(
Leftover(
name=pv.metadata.name,
phase=(pv.status.phase or ""),
reclaim=(pv.spec.persistent_volume_reclaim_policy or ""),
claim_ns=claim_ns,
claim_name=claim_name,
storage_class=pv.spec.storage_class_name,
capacity=cap,
claim_missing=missing,
)
)
return foundWalk the result with three predicates.
Released and claim missing. This is the volume that survived the claim. Reclaim should be Retain. If reclaim is Delete and the PV is still here, the CSI delete hung or an administrator intervened. Print it either way. Do not decide.
Released and claim present. The PVC delete has not landed, or a new PVC reused the same name in the same namespace. Names are not uids. claim_ref.uid is the field that distinguishes "same name, new object" from "the original claim." Read it if you need that distinction. Q1 LabSetUp uses a fresh PVC with the old name after the old uid is gone. The leftover pointer still blocks until it is cleared.
Bound and claim missing. The API should not show this for long. Report it. A controller is mid-update, or you are looking at a stale informer. This tool does not use an informer. It lists once.
The printer is the interface. JSON lines, one leftover per line, only the rows that are claim_missing or phase == "Released". A full dump of every Bound PV is noise. 08-19's checker printed Secret metadata and refused to print data. This checker prints the pointer and refuses to print a patch.
import json
def report(rows: list[Leftover]) -> None:
for row in rows:
if row.phase != "Released" and not row.claim_missing:
continue
print(json.dumps({
"name": row.name,
"phase": row.phase,
"reclaim": row.reclaim,
"claim": (
f"{row.claim_ns}/{row.claim_name}"
if row.claim_ns and row.claim_name
else None
),
"claim_missing": row.claim_missing,
"storage_class": row.storage_class,
"capacity": row.capacity,
}))That is the whole tool. A later function named clear_claim_ref would be one patch_persistent_volume with a JSON remove on /spec/claimRef. Do not write it in this file. Q1 LabSetUp already has the patch. Cert will type it by hand. A Python loop that applies it is how you give the next matching PVC a disk it did not ask to inherit.
Walk one object the way you would on the exam cluster, then on GKE.
On Q1 the leftover is mariadb-pv. Phase starts Bound while MariaDB is up. LabSetUp deletes the Deployment and the PVC, then patches claimRef so the candidate does not waste the clock on Released. If you run this tool before that patch, you should print:
{"name": "mariadb-pv", "phase": "Released", "reclaim": "Retain", "claim": "mariadb/mariadb", "claim_missing": true, "storage_class": "standard", "capacity": "250Mi"}If you run it after the patch, the row disappears from the report (phase is Available, claim_missing is false because there is no ref). The candidate's job is the new PVC. Your job was the row. The row is gone because LabSetUp decided. Your tool did not decide.
On GKE the leftover is a CSI volume whose name starts with pvc- and a uuid. storage_class is sc-fast-repl. reclaim is Retain. capacity is 20Gi. claim is default/pvc2. Compute Engine has a regional disk whose name matches. Your JSON does not mention the disk. That omission is the point.
§IV — What the client cannot see
Poulton's clean-up has a third list: Google Cloud Console, Compute Engine, Disks (Ch. 10, p. 133). The Kubernetes client does not have that list.
pv.spec.csi.volume_handle on a GKE CSI volume is a clue. It often contains the disk name. It is still a Kubernetes field. It is not a Compute Engine API response. A disk whose PV you already deleted will not appear here. A disk whose delete hung will not appear here either if the PV object is gone.
If you want the third list, you open a second client. google.cloud.compute_v1.DisksClient. You pass a project and a zone or a region. Regional PD lives in a region, not a zone. Poulton's sc-fast-repl is replication-type: regional-pd. disks.list on a single zone will miss it. region_disks.list is the call.
Do not add that client to the leftover tool unless the operator passed --project. Default-on cloud calls from a Kubernetes checker are how 08-19's first draft tried to open etcd. Same shape. The Kubernetes object is the Kubernetes object. The disk is the disk. Two clients, two findings, one report that says they can disagree.
kubectl get pvc is empty. The Python list is empty. The invoice is not. That sentence is why the second client exists, and why it is optional.
RBAC is the other wall. A Role in a namespace cannot list PersistentVolumes. PVs are cluster-scoped. You need a ClusterRole with persistentvolumes get/list, and persistentvolumeclaims get/list if you want claim_missing to be real. Without PVC get, you can still print Released plus claimRef. You cannot know whether the claim is gone. Print claim_missing: null and leave. Do not infer 404 from a 403.
Types earn their keep on the claimRef branch. ref is V1ObjectReference | None. A Released PV with Retain should have a ref. A Released PV with the ref already removed is Available in a correct cluster and Released-and-empty only if someone half-patched. if ref: is the gate. ref.name without the gate is an AttributeError, and that exception is better than printing claim: None/None as if it were a finding.
Do not catch ApiException broadly and continue. 403 on one PVC get means your RBAC is short. Raising is the report. Swallowing turns a permissions problem into claim_missing: true for every Bound volume you were not allowed to see. That report would tell an operator to patch production claimRefs that are still live.
§V — Connection to today's Ops lesson
Ops is the GKE class and the disk. Dev is the list. The coined name is the same leftover.
Ops applies sc-fast-repl with reclaimPolicy: Retain on pd.csi.storage.gke.io. A 20Gi PVC binds. The Pod dies. The PVC dies. The PV is Released. Ops then shows the two recoveries: patch claimRef, or delete the PV and the GCE disk. Dev stops before both recoveries. It prints the Released row. A human chooses reuse or abandon.
If the class was premium-rwo, Dev's list is empty after the delete. Delete already removed the PV. The checker saying "no leftovers" is correct for the API and wrong for a disk only if the CSI delete hung. That hung case is the optional Compute Engine read, not a Kubernetes retry.
Ops also named the default-class gun. A PVC that omitted storageClassName landed on standard, in-tree, Immediate, Delete. Dev can print storage_class on each leftover. If the field is standard and reclaim is Delete, you are looking at a hung delete, not a Retain success. Do not patch a Delete leftover into Available. You would rebind a volume the provisioner is trying to destroy.
§VI — Prior-lesson reach
08-19's checker listed Secret metadata and refused to treat data: as a store audit. Today's checker lists PV metadata and refuses to treat a Released phase as a license to patch. Same refusal, different object.
08-16's fit-checker printed a scheduler no and did not bind. Today's leftover-checker prints a binder no (Released plus claimRef) and does not clear the pointer. Same refusal, different API.
08-10's event tool watched a Pending Pod and named the owner. A Pending PVC whose matching PV is Released is the storage owner of that Pending. This tool does not watch. It lists. If you need the event, go back to 08-10. If you need the pointer, stay here.
08-04 already loaded in-cluster config and asked SelfSubjectAccessReview. If you want to know whether this tool can list PVs before it lists them, reuse that preflight. Do not add it unless the operator asked. A 403 on list_persistent_volume is already an answer.
08-13 kopf mutated. This file does not.
The pub test for this file is one sentence. Would you run a default-on patch from a leftover report? No. You would print the row and open a ticket, or you would type the Q1 patch on one named PV. The generated client will do either. Only one of those belongs in this program.
§VII — Close
The generated client will list every PersistentVolume you can see. Released plus a dead claimRef is the volume that survived the claim. Retain made that leftover legal. The patch that makes it bindable is a human decision, and Q1's. Your loop reports. It does not decide.
Examine well. Print the pointer. Leave the disk. Leave the patch.
Related
- Prior arc: Python and the EKS Encryption Config (2026-08-19)
- Language hub: Cross-References/dev-languages/Python
- Grounding tome: K8sUR3 Ch.13, generated client libraries, p. 216