EKS Secrets Encryption and KMS — the secret that is still plaintext

The console says encrypted. etcd still has the old bytes. kubectl get will not tell you.

<!-- hal:authoritative:yaml -->

The console says encrypted. etcd still has the old bytes. kubectl get will not tell you.

§I — Frame

Three days ago this arc asked kube-scheduler for a vote on AKS. Six days ago it put a guest kernel under a Pod on GKE. Nine days ago it put a disk under a Pod on EKS. Those lessons ended on a Pending, a sandbox, and a volume. None of them asked what etcd holds when you type kubectl create secret.

Today the owner is the write path into the cluster store.

A Secret is admitted. kubectl get secret db-pass -o yaml prints the keys. The EKS page shows a KMS ARN under Secrets encryption. An auditor asks whether the password is encrypted at rest. The honest answer is: maybe the new ones are. db-pass was created fourteen months ago, before anyone called associate-encryption-config. Enabling encryption on an existing cluster does not rewrite old objects. The API still returns plaintext for every Secret it can decrypt, including the ones it never encrypted.

Call the failure the secret that is still plaintext. The control plane accepted a KMS key. New writes wrap a data-encryption key and send that wrapped key to KMS. The old object in etcd is the same base64 blob it was on the day it was created. kubectl get cannot see the difference, because the API server decodes before it answers.

EKS is the overlay on purpose. 08-16 was AKS. 08-13 was GKE. 08-10 already spent EKS on disks. Today's EKS surface is cluster.encryptionConfig: a resources list whose only supported value is secrets, and a provider.keyArn pointing at a customer-managed KMS key in the same Region. Envelope encryption is the shape. KMS wraps the DEK. The Secret object the API returns is still plaintext.

Poulton says the short answer first. Secrets are not secure. Kubernetes does not encrypt them. It obscures them as base64 values anyone can decode. EncryptionConfiguration is how you change the store, and even then the Secret is mounted into the container as plain text so the app can read it (Ch. 11, Are Secrets secure, pp. 148-150). Rice says the same opt-in: encryption at rest is available and is not on by default; etcd stores base64, not ciphertext, until you configure it (Ch. 12, Kubernetes Secrets, pp. 145-146). Today's lesson is what that opt-in looks like when the cluster is EKS and the key is a CMK, and what remains in the clear after you flip the switch.

§II — Foundations

The store is etcd; the default is cleartext

Every Secret is an API object. The API server serializes it and writes it to etcd. Bootcamp Q08 opens on that sentence: by default, Kubernetes stores Secrets in etcd in plaintext. Poulton's workflow is the same fact as a list. The Secret is created and persisted un-encrypted. It is transferred to the node. The kubelet mounts it on a tmpfs and decodes base64 to plain text. The app consumes it. When the Pod dies, the node copy goes away. The etcd copy does not (Ch. 11, p. 148).

Base64 is not a key. echo UGFzc3dvcmQxMjM= | base64 -d prints Password123 with no secret material (Ch. 11, p. 150). An etcd snapshot or a volume backup of the control plane is enough. EKS hides etcd from you. That hiding is not encryption. It is an access boundary around a store that, until you configure a provider, still holds the object in the clear.

Rice names the commercial turn: enterprises put the key in a cloud KMS or Vault, and Rancher's docs show AWS KMS as the encryption-at-rest example (Ch. 12, p. 146). EKS is that example as a product. The cluster's encryptionConfig is how you attach the CMK. The API server, which you do not SSH onto, is the process that calls KMS.

Envelope: a DEK in the cluster, a KEK in KMS

Poulton's threat model is why the cloud key exists. Kubernetes 1.7 introduced encryption of Secrets and did not enable it by default. Even when encryption is on, the data-encryption key lives on the same node as the Secret. Access to that node bypasses the wrap. Version 1.11 let the key-encryption key live outside the cluster, as an HSM or a cloud KMS (Ch. 15, Protecting cluster data, p. 204).

EKS implements that outside key. The API server generates or unwraps a DEK. The DEK encrypts the Secret bytes. KMS encrypts the DEK. The blob that lands in etcd is ciphertext plus enough metadata to ask KMS for the DEK again. The CMK never sees the Secret. The CMK sees the DEK.

Two consequences follow.

First, deleting the CMK is not a cleanup. AWS warns that a cluster whose encryption key is gone is not recoverable. Rotation is a new key and a rewrite, not a delete.

Second, kubectl get secret is not a ciphertext viewer. The API server decrypts on the way out. A compliant get that prints data: is the designed behavior. It is not proof the store is encrypted, and it is not proof the store is clear. It is proof you are talking to the API, not to etcd.

What describe-cluster actually returns

aws eks describe-cluster --name atlas-prod --query cluster.encryptionConfig

A cluster that has a customer CMK associated answers with a list. Each item has resources (only secrets is supported) and provider.keyArn (the CMK ARN or an alias ARN in the same Region). A cluster that has never had associate-encryption-config called, and that was not created with --encryption-config, answers with null or an empty list.

That field is the association. It is not a census of etcd. It does not list Secrets. It does not return the time you enabled encryption. It does not tell you whether objects that already existed were rewritten. The Dev lesson is the checker that sees this field and cannot see the store.

Currency note: EKS 1.28 and higher documents default envelope encryption for Kubernetes API data, and associate-encryption-config is scoped to 1.27 and lower. Platform wrap is not encryptionConfig.keyArn. Today's referent is the CMK association, the enable-later path, and the Secret that existed before it. The exam is kubeadm plus aescbc. Neither makes kubectl get print ciphertext.

§III — Worked example

Cluster atlas-prod is two years old. It was created without --encryption-config. The CMK arn:aws:kms:us-east-1:111122223333:key/abcd1234-... should now wrap Secrets. The key is symmetric, in us-east-1, and the cluster role can kms:DescribeKey and kms:CreateGrant.

Enable is one API call.

aws eks associate-encryption-config \
  --cluster-name atlas-prod \
  --encryption-config '[{"resources":["secrets"],"provider":{"keyArn":"arn:aws:kms:us-east-1:111122223333:key/abcd1234-..."}}]'

The response is an update of type AssociateEncryptionConfig with status InProgress. You wait on describe-update until the status is Successful. Then describe-cluster shows the ARN. The action is irreversible. You cannot disassociate. You can rotate to a new key. You cannot turn the feature off.

At this moment the console is green and the store is mixed. Every Secret created after the update writes through the KMS provider. Every Secret created before the update is the object it was. Q08 step 5 is the same sentence on a kubeadm control plane: without a rewrite, old secrets remain in plaintext in etcd. EKS did not invent a different physics. It hid etcd and left the rewrite to you.

Two rewrite paths exist. Do not confuse them.

The exam hand, Q08 SolutionNotes, is a replace of every Secret:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Replace reads each object through the API (plaintext) and writes it back. The write hits the first provider. On EKS that provider is kms. On the exam it is aescbc. The bytes in etcd change. The string a client sees does not.

AWS's enable-kms page, for the CLI path after associate, documents an annotate instead of a replace:

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - kms-encryption-timestamp="2026-08-19T10:45:00Z"

Annotate is also a write. A write is the point. eksctl utils enable-secrets-encryption will do the rewrite for you unless you pass --encrypt-existing-secrets=false. The false flag is how you reproduce today's failure on purpose: association on, store still mixed.

You cannot confirm the rewrite with kubectl get. On kubeadm, Q08 uses etcdctl get /registry/secrets/default/test-unencrypted | hexdump -C and looks for k8s:enc:aescbc:v1:key1. On EKS you do not have etcdctl. You have the rewrite command's exit code, the annotation or resourceVersion change, and (if CloudTrail is on) a KMS Encrypt trail that starts after the rewrite, not after the associate. The Dev checker will flag Secrets whose creationTimestamp is earlier than the moment you recorded for the association. It will not open etcd.

A create-time --encryption-config wraps Secrets written after the cluster exists. A restore that talks to the API writes a new object and will encrypt.

§IV — Failure mode: the secret that is still plaintext

Enable without rewrite. associate-encryption-config returns Successful. The console badge changes. A ticket is closed. db-pass in payments was created on 2025-06-03. Nobody replaced it. An etcd-level copy still has the old bytes. Q08's SolutionNotes wrote this down so the exam would not forget it. EKS's enable-kms page writes the same order: after you enabled encryption, you must encrypt all existing secrets with the new key.

**Trusting kubectl get.** An engineer pastes kubectl get secret db-pass -o yaml and points at the data map. That map is plaintext-as-the-API-returns-it. It looked the same before the CMK existed. Poulton's decode of UGFzc3dvcmQxMjM= is this paste (Ch. 11, p. 150). The API is doing its job. The job is not a store audit.

The identity-first config, on a cluster you can edit. The Cert lesson is the exam form. If the first provider is identity: {}, every new write is plaintext and the aescbc or kms provider is only a reader of old ciphertext. EKS will not let you paste that file. A kubeadm cluster will. The failure is the same name: the secret that is still plaintext, now as a default write path you chose.

A deleted CMK. Encryption is on. The rewrite happened. Someone deletes the key to finish a rotation. Decrypt fails. AWS says the cluster is permanently degraded. Rotation is encrypt under a new key plus a rewrite, then a waiting period, then a disable of the old key. Delete is not a step.

A CMK in the wrong account without a grant. Associate or create fails, or the API server later cannot unwrap. kms:CreateGrant on the calling principal is the usual miss. The condition kms:GrantIsForAWSResource is not supported on CreateCluster.

Treating platform default envelope as the CMK. A 1.28 cluster has AWS-managed wrap. That is not encryptionConfig.keyArn. A customer-CMK finding is still red.

§V — Pairing

Today's Dev lesson is a Python checker that calls eks.describe_cluster for encryptionConfig and lists Secret metadata with the Kubernetes client. It can flag a cluster with no CMK association, or a Secret whose creationTimestamp is earlier than the moment you recorded for that association. It cannot see etcd. It does not bind. It is not the 08-16 fit-checker, not the 08-13 kopf mutate, and not the 08-04 SelfSubjectAccessReview.

Today's Cert lesson is CKS Q08 as a hand exam. The file is /etc/kubernetes/enc/enc.yaml. The first provider encrypts. identity: {} is the read fallback. etcdctl plus hexdump is how you see k8s:enc:aescbc:v1:key1. The replace is step 5. Bootcamp files Q08 under Minimize Microservice Vulnerabilities (20%). That is the honest domain tag. 08-13 left Cluster Hardening and System Hardening on the leftover list. Today's CKS fire is this control because it is the one the EKS overlay binds. Next CKS day 2026-08-22 can take Cluster Hardening Q04.

08-16, 08-13, 08-10, 08-07, 08-04, and 08-18 stay closed.

§VI — Drills

Related

• AKS scheduling (2026-08-16)
• GKE runtime isolation (2026-08-13)
• EKS EBS CSI (2026-08-10)
• Cross-References/domains/01-Earth-DevOps
• Poulton Ch.11 pp.148-150; Ch.15 p.204 · Rice Ch.12 pp.145-146

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura Filed 2026-08-19 · Fajr · K8s day 28 · tenth visit · trio #94