EKS control-plane audit logging enabled_cluster_log_types, CloudWatch, and a Rust census

An EKS cluster keeps no audit trail you can read until you ask for one.

An EKS cluster keeps no audit trail you can read until you ask for one. Ask for it in code, then prove every cluster asked.

§I. Frame

The last three K8s Ops ran AKS (namespace guardrails, 10-06), GKE (Binary Authorization, 10-03), and EKS (Fargate profiles, 09-30). Today returns to EKS, and to the question an incident responder asks first: who read that Secret?

On EKS the API server is AWS's. You cannot pass it flags. You get five log streams you can switch on, and the one that answers the responder is audit.

The problem for today: declare control-plane logging in Terraform, then build a read-only census that finds every cluster where audit or authenticator is not enabled.

§II. Off Until Asked

Off Until Asked (named technique). Treat every EKS control-plane log type as absent until a describe call shows it enabled.

1. Five types. api, audit, authenticator, controllerManager, scheduler. Each is enabled or disabled on its own.
2. Default off. A new cluster ships with all five disabled. Nothing in kubectl tells you.
3. One destination. Enabled logs land in CloudWatch Logs under /aws/eks/<cluster>/cluster, one stream per type and API server instance. You pay CloudWatch ingestion and storage.
4. The policy is AWS's. EKS chooses what the audit stream records. You choose only whether it ships. The paired Cert lesson writes the policy yourself on a cluster where you own the API server.

Two That Answer Who (named technique). audit records what was asked and the answer. authenticator maps an IAM principal to the Kubernetes username you see in audit. With one and not the other, the trail stops at a role name or at kubernetes-admin. The census requires both.

§III. The declaration

eks-logging.tf (terraform 1.16.5, hashicorp/aws v6.68.0, validate clean, not applied):

resource "aws_cloudwatch_log_group" "eks" {
  # EKS writes to exactly this name; create it first to own retention.
  name              = "/aws/eks/prod-payments/cluster"
  retention_in_days = 90
}

resource "aws_eks_cluster" "prod_payments" {
  name     = "prod-payments"
  role_arn = var.cluster_role_arn
  vpc_config { subnet_ids = var.subnet_ids }

  enabled_cluster_log_types = ["api", "audit", "authenticator"]
  depends_on                = [aws_cloudwatch_log_group.eks]
}

The depends_on matters. If EKS creates the group first, it creates it with no retention, and Terraform later fails to create a group that already exists. Owning the group first sets retention to 90 days.

On an existing cluster the same change is aws eks update-cluster-config --name prod-payments --logging '{"clusterLogging":[{"types":["audit","authenticator"],"enabled":true}]}'. It is an in-place cluster update, not a replacement.

§IV. The Rust census

eks-audit-log-census.rs is a cargo script. It reads a JSON array of aws eks describe-cluster outputs, unions the types of every clusterLogging entry with enabled: true, and flags:

• all_logging_off: no enabled type at all.
• missing=audit / missing=authenticator / missing=audit+authenticator: Two That Answer Who broken.

The API returns enabled and disabled types as separate entries in the same list, so a type is on only if it appears under an enabled entry. Reading only the first entry is the classic bug.

Checked on the box. No AWS credentials, so the input is fixture/clusters.json, a synthetic file in the documented DescribeCluster shape. This is the script's real output over that fixture (0 warnings, exit 2):

prod-payments	ok	enabled=5
prod-web	ATTENTION	missing=audit
staging	ATTENTION	all_logging_off
sandbox	ATTENTION	missing=authenticator
summary clusters=4 attention=3

staging lists all five types, under enabled: false. A grep for the string audit would pass it.

§V. How to run

cargo +nightly -Zscript ./eks-audit-log-census.rs fixture/clusters.json
nix-build eks-audit-log-census.nix -I nixpkgs=flake:nixpkgs && ./result/bin/eks-audit-log-census

The Nix wrapper lists clusters in the current region, describes each, and pipes the array into the script. It needs read-only eks:ListClusters and eks:DescribeCluster. It never updates a cluster. Exit 2 means at least one cluster needs attention, so CI can gate on it.

§VI. Close

Off Until Asked: EKS control-plane logs exist only when enabled, per type, and nothing in-cluster warns you. Two That Answer Who: ship audit and authenticator together. Terraform declares it, the log group is created first to own retention, and the census proves it across the fleet.

Paired Dev parses those audit events into Rust types and flags Secret reads by humans. Paired Cert writes the audit Policy that decides what the events contain.

Related

• Tome: Poulton, The Kubernetes Book ch15 pp.205-206 (threat model) (referenced)
• Prior K8s Ops: AKS guardrails 10-06 · GKE Binary Authorization 10-03 · EKS Fargate 09-30
• Web: EKS control plane logs · aws_eks_cluster · DescribeCluster API