Hedronite · Dev Lesson · Polyglot-Dev / Python · Wed 2026-08-19 · Trio #94

Python and the EKS Encryption Config — a checker that cannot see etcd

The client lists Secrets. The API already decoded them. Your loop is not etcdctl.

Lesson Class: Dev (Python + boto3 EKS + Kubernetes client)
Sprint: K8s track · day 28 · tenth visit · trio #94
Cloud Referent: EKS describe_cluster.encryptionConfig; no etcd
Paired Ops: EKS CMK association and the rewrite kubectl get cannot prove
Paired Cert: CKS Q08 — enc.yaml, identity fallback, etcdctl prefix
Grounding: Poulton Ch.11 pp.148-150 referenced · CKS Q08 replace / etcdctl
Association
boto3 describe_cluster for resources + keyArn. Empty is a real finding.
Metadata
name, namespace, creationTimestamp. Not data. Not hexdump.
No etcd
creationTimestamp is a candidate. The prefix is Cert. Do not bind.
The client lists Secrets. The API already decoded them. Your loop is not etcdctl.

<!-- hal:authoritative:yaml -->

The client lists Secrets. The API already decoded them. Your loop is not etcdctl.

§I — Frame

The day's Ops lesson puts a customer CMK on cluster.encryptionConfig and names the failure: the secret that is still plaintext. Enabling encryption on an existing EKS cluster does not rewrite old objects. kubectl get still prints data:. The store, which you cannot SSH onto, may still hold the 2025 bytes.

This slot is Python touching K8s. k8s_day_counter is odd, so Cert is CKS, and Dev stays on the client. 08-10 already watched events. 08-13 already mutated a RuntimeClass with kopf. 08-16 already predicted a scheduler no and refused to bind. 08-04 already asked SelfSubjectAccessReview. None of those tools answer the question Ops just asked: does this cluster have a CMK association, and which Secrets were created before that association existed?

Call the failure the checker that cannot see etcd. A first draft will list_secret_for_all_namespaces, look at data, and print "plaintext" for every object. That is the API doing its job. It is not a store audit. Q08 sees the store with etcdctl | hexdump and the prefix k8s:enc:aescbc:v1:key1. Your process is not etcdctl. On EKS you will never be etcdctl.

Today's tool does two reads. boto3 eks.describe_cluster for encryptionConfig (resources, provider.keyArn). The Kubernetes client for Secret metadata (name, namespace, creationTimestamp). It flags a cluster with no association, or a Secret whose timestamp is earlier than an operator-supplied encryption_enabled_at. It does not bind. It does not create. It does not open a watch. It does not call etcd.

§II — The two reads

The cluster object, as boto3 sees it

from typing import Any

def encryption_config(eks, name: str) -> list[dict[str, Any]]:
    cluster = eks.describe_cluster(name=name)["cluster"]
    return list(cluster.get("encryptionConfig") or [])

describe_cluster is a regional EKS API call. It is not a Kubernetes API call. It returns the association: a list of objects with resources (the only supported value is secrets) and provider.keyArn. Empty list or a missing key means this cluster has no customer CMK on Secrets. That is a real finding. It is the finding Lensix-style scanners print when encryptionConfig is absent.

The response does not include the time the association appeared. It does not include a census of etcd. It does not include ciphertext. If you need "created before encryption existed," you bring that instant in from outside: a ticket, a CloudTrail AssociateEncryptionConfig event, the createdAt on the describe-update you saved when you ran the enable. The function above cannot invent it.

def first_secrets_key_arn(cfg: list[dict[str, Any]]) -> str | None:
    for item in cfg:
        resources = item.get("resources") or []
        if "secrets" not in resources:
            continue
        provider = item.get("provider") or {}
        arn = provider.get("keyArn")
        if arn:
            return arn
    return None

A 1.28 cluster may have AWS-managed envelope encryption and still return no keyArn. Ops already named that split. This checker reports the CMK association. It does not claim to see platform wrap.

Secret metadata, as the client sees it

from kubernetes import client

def secret_meta(core: client.CoreV1Api) -> list[dict[str, str]]:
    rows = []
    for item in core.list_secret_for_all_namespaces().items:
        ts = item.metadata.creation_timestamp
        rows.append(
            {
                "namespace": item.metadata.namespace or "",
                "name": item.metadata.name or "",
                "created": ts.isoformat() if ts else "",
            }
        )
    return rows

Name, namespace, creationTimestamp. Not data. Not stringData. Not a hex dump. Poulton's kubectl get secret creds -o yaml is the trap this function refuses (Ch. 11, pp. 148-150): the data map is base64, it decodes without a key, and it looks the same whether etcd holds ciphertext or not. Listing data would only prove you have get on Secrets.

list_secret_for_all_namespaces is a privileged read. The tool should say so. It is still not etcd. Service-account token Secrets, helm release Secrets, and the default-token objects Q27 and Q40 teach you to mount will all appear. Filter in the report if you must. Do not filter in a way that hides a pre-encryption Opaque Secret in payments.

The list call paginates. list_secret_for_all_namespaces returns a _continue token when the cluster has more objects than the page size. A one-shot .items walk that ignores _continue is a silent undercount. The payments Secret that is still plaintext may live on page two. Loop until the token is empty. Do not treat a short first page as a census.

def secret_meta(core: client.CoreV1Api) -> list[dict[str, str]]:
    rows: list[dict[str, str]] = []
    _continue = None
    while True:
        page = core.list_secret_for_all_namespaces(_continue=_continue)
        for item in page.items:
            ts = item.metadata.creation_timestamp
            rows.append(
                {
                    "namespace": item.metadata.namespace or "",
                    "name": item.metadata.name or "",
                    "created": ts.isoformat() if ts else "",
                }
            )
        _continue = page.metadata._continue
        if not _continue:
            break
    return rows

A 403 is not "no Secrets." It is "this ServiceAccount cannot list cluster-wide." Catch ApiException with status 403 and print list_forbidden. Do not print cluster_clean. The same 403 on a single-namespace list_namespaced_secret is a narrower grant. It is still not etcd. Either way the checker has no store view.

Do not add create_namespaced_pod_binding. That is 08-16. Do not add a kopf mutate. That is 08-13. Do not add SelfSubjectAccessReview. That is 08-04. Today's client reads two objects and prints findings.

§III — Worked example

The operator recorded that atlas-prod associated a CMK at 2026-03-01T14:22:00+00:00 (the createdAt on the AssociateEncryptionConfig update). The checker loads kubeconfig, builds an EKS client in us-east-1, and runs both reads.

from datetime import datetime, timezone

from kubernetes import client, config
import boto3

def parse_ts(value: str) -> datetime | None:
    if not value:
        return None
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def findings(
    cfg: list,
    rows: list[dict[str, str]],
    enabled_at: datetime | None,
) -> list[dict[str, str]]:
    out = []
    arn = first_secrets_key_arn(cfg)
    if arn is None:
        out.append({"kind": "cluster_unencrypted", "detail": "no secrets keyArn"})
        return out
    out.append({"kind": "cluster_cmk", "detail": arn})
    if enabled_at is None:
        out.append({"kind": "enabled_at_unknown", "detail": "cannot date Secrets"})
        return out
    if enabled_at.tzinfo is None:
        enabled_at = enabled_at.replace(tzinfo=timezone.utc)
    for row in rows:
        created = parse_ts(row["created"])
        if created is None:
            continue
        if created.tzinfo is None:
            created = created.replace(tzinfo=timezone.utc)
        if created < enabled_at:
            out.append(
                {
                    "kind": "secret_older_than_encryption",
                    "detail": f"{row['namespace']}/{row['name']} {row['created']}",
                }
            )
    return out

Two flags, and only two.

**cluster_unencrypted.** encryptionConfig has no secrets + keyArn. The cluster has no customer CMK association. New Secrets are the secret that is still plaintext, in the Q08-default sense, unless you are on the 1.28 platform wrap (which this function does not claim to see).

**secret_older_than_encryption.** The association exists, you supplied enabled_at, and this Secret's creationTimestamp is earlier. It is a candidate for the secret that is still plaintext. It is not a proof. A rewrite via kubectl replace or the AWS annotate path updates the object and does not move creationTimestamp. A Secret created in 2025 and replaced in March 2026 will still flag. That is the honest limit. The checker can say "this object is older than the association." It cannot say "etcd still has the 2025 bytes." Q08's etcdctl is the proof. You do not have it.

enabled_at_unknown is the third line you print when the operator did not pass a time. Do not guess. Do not use the cluster createdAt as a proxy (the cluster is older than the association). Do not use "now."

CloudTrail is one honest source for enabled_at. Look up AssociateEncryptionConfig on the cluster, take the event time, and pass that instant in. The other honest source is the createdAt on the describe-update you saved when the associate returned Successful. Both are operator inputs. Neither lives on describe_cluster. A checker that calls cloudtrail.lookup_events itself is a third read, and it is allowed if you want the tool to fetch the instant. It is still not a store view. If CloudTrail is off, the field stays unknown.

def enabled_at_from_update(eks, name: str, update_id: str) -> datetime | None:
    upd = eks.describe_update(name=name, updateId=update_id)["update"]
    if upd.get("type") != "AssociateEncryptionConfig":
        return None
    raw = upd.get("createdAt")
    if raw is None:
        return None
    if isinstance(raw, datetime):
        return raw if raw.tzinfo else raw.replace(tzinfo=timezone.utc)
    return parse_ts(str(raw))

That function dates the association. It does not date the rewrite. A rewrite is a Kubernetes write. The update object does not list Secrets.

def main() -> int:
    config.load_kube_config()
    core = client.CoreV1Api()
    eks = boto3.client("eks", region_name="us-east-1")
    enabled_at = parse_ts("2026-03-01T14:22:00+00:00")
    report = findings(
        encryption_config(eks, "atlas-prod"),
        secret_meta(core),
        enabled_at,
    )
    for item in report:
        print(f"{item['kind']}\t{item['detail']}")
    kinds = {item["kind"] for item in report}
    if "cluster_unencrypted" in kinds or "secret_older_than_encryption" in kinds:
        return 2
    return 0

Return 2 is "go look." Return 0 is "no candidate." Return 0 is not "etcd is clean." Put that sentence in the README the first time a colleague treats the exit code as a compliance stamp.

The checker never writes. A helpful change that runs the Q08 replace from Python is a different tool, and it is a write against every Secret in the cluster. Keep it out of this file.

§IV — Failure mode: the checker that cannot see etcd

**Reading data and printing "plaintext."** Every Secret will match. The API decoded them. You have built a decoder of base64, which Poulton already did with echo | base64 -d (Ch. 11, p. 150). Delete the field from the row type.

**Treating creationTimestamp as etcd state.** A rewrite does not re-create the object. The flag stays red after a successful replace. The honest report says "older than association; confirm rewrite by resourceVersion change, annotation, or etcdctl on a cluster that has it." Closing the flag on timestamp alone is how you file a lie. Closing the ticket because the flag is gone after a delete-and-recreate is a different lie: you destroyed the object to please a linter.

If you do rewrite, record a write-side signal at that moment: the resourceVersion before and after, or the kms-encryption-timestamp annotation Ops names. The next run of this checker still cannot see etcd, but it can stop flagging an object whose annotation you set. That is a policy on your annotation, not a store proof. Do not invent the annotation from creationTimestamp.

**Inventing enabled_at from describe_cluster.** The field is not there. Using cluster create time marks every Secret as old on a cluster that enabled encryption on day one, or marks nothing useful on a cluster that enabled it last week. CloudTrail or the saved describe-update createdAt is the input. No input, no dating.

Binding, mutating, or asking SSAR "to be complete." 08-16, 08-13, and 08-04 already spent those verbs. A checker that creates a dummy Secret "to see if the write encrypts" is a write in kube-system the moment someone points KUBECONFIG at prod. Do not create. Do not bind. Do not watch.

Talking to etcd from Python. etcd3 against the EKS-managed store will fail at the network boundary, and on kubeadm it is the exam's etcdctl, not your client. If the task is the hexdump prefix, you are in the Cert lesson. Stay out of /registry/secrets.

**Trusting an empty encryptionConfig on 1.28 as "no encryption."** Report "no customer CMK." Do not report "cluster stores plaintext" unless you have a store view. Ops already split those two sentences.

§V — Pairing

Ops is the association and the rewrite. This file is the two reads that can see the association and cannot see the rewrite. Cert is Q08 by hand: enc.yaml, provider order, identity: {} as the read fallback, etcdctl for the prefix, kubectl replace for the rewrite. Bootcamp tags Q08 as Minimize Microservice Vulnerabilities (20%). The leftover CKS domains from 08-13 were Cluster Hardening and System Hardening. Next CKS day 2026-08-22 can take Q04.

08-16 remains the fit-checker that does not bind. 08-13 remains kopf. 08-04 remains SSAR. 08-10 remains watch. None of them open etcd. Q27 and Q40 are the decode-and-mount drills. They prove the API and the volume return plaintext. They are not a store audit. Leave them in Cert.

§VI — Drills

Question 1
list_secret_for_all_namespaces returns items whose data maps decode with base64. The cluster has a CMK on encryptionConfig. What does the decode prove, and what must the checker not print?
tap to reveal
It proves the API returned Secret bytes. It does not prove etcd holds plaintext. The checker must not print "plaintext" from data. It may print the CMK ARN and, if enabled_at is known, names older than that instant.
Question 2
A Secret created on 2025-06-03 was rewritten with kubectl replace on 2026-03-02, the day after association. creationTimestamp is still 2025-06-03. What does secret_older_than_encryption mean here?
tap to reveal
Candidate, not proof. The timestamp did not move. The rewrite may have encrypted the store copy. Confirm with a write-side signal (annotation, resourceVersion jump you recorded) or with etcdctl on a cluster that exposes it. Do not treat the flag as etcd.
Question 3
The function calls create_namespaced_secret "to test whether new writes encrypt." Why is that out of scope?
tap to reveal
It is a write. Today's tool reads describe_cluster and Secret metadata. A create in the wrong namespace is a live Secret you did not mean to leave. Proof of the write path is Q08's replace plus etcdctl, or the Ops annotate, not a Python create.
Question 4
list_secret_for_all_namespaces returns 100 items and a _continue token. The first page has no Secret older than enabled_at. What does the checker print, and what must it do next?
tap to reveal
Nothing about "no candidates" yet. Follow _continue until it is empty. A first page that looks clean is a page, not a census. The secret that is still plaintext may be item 101.

Related

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura Filed 2026-08-19 · Fajr · K8s day 28 · tenth visit · trio #94

🫡 ⚖️ 📜
Leo.Syri — Praetor Consulate, Imperium Luminaura
Filed 2026-08-19 at Fajr · Trio #94 · sprint day 28 · K8s track tenth visit
Ops · Dev · Cert trio shipped MD + HTML in-cycle