CKS kube-apiserver audit Policy levels, omitStages, and first match wins

The first rule that matches decides. Never log a Secret's body.

The first rule that matches decides. Put the narrow rules on top, and never log a Secret's body.

§I. Frame

On EKS (paired Ops) AWS owns the audit policy. On the CKS exam you own the API server, and a typical task reads: write a policy, wire it in, prove it logs. The paired Dev lesson reads the result.

The task for today: on a kubeadm control plane, log Secrets and ConfigMaps at Metadata only, log Pod changes in prod with full request and response, drop noisy reads, and catch everything else at Metadata.

§II. Four Levels, One Winner

Four Levels (named technique). Say what each level writes before choosing one.

First Match Wins (named technique). Rules are checked top to bottom. The first rule that matches an event sets its level. Later rules never see it. A catch-all at the top swallows the file.

Never the Body. A Secret at Request or RequestResponse writes the base64 value into the log. Secrets and ConfigMaps go at Metadata, and that rule sits above any broader rule.

omitStages drops records at named stages. RequestReceived is the usual one: it doubles the volume and ResponseComplete carries the answer. Set at the top level it applies to every rule.

§III. The Policy

/etc/kubernetes/audit/policy.yaml (from the kubernetes.io reference, not applied this fire):

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
  # 1. Noise first, so nothing below logs it.
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
  - level: None
    nonResourceURLs: ["/healthz*", "/livez*", "/readyz*"]
  # 2. Secrets and ConfigMaps: who and what, never the value.
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
  # 3. Pod writes in prod: full detail.
  - level: RequestResponse
    namespaces: ["prod"]
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: ""
        resources: ["pods"]
  # 4. Everything else.
  - level: Metadata

Swap rules 2 and 3 and nothing changes for Secrets, because rule 3 names only pods. Swap rules 2 and 4 and Secrets fall to rule 4, which is also Metadata, so it still looks safe. Change rule 4 to Request later and Secret values start landing in the log. Keep the Secret rule above the catch-all.

§IV. Wire it

The API server is a static pod. It reads the policy and writes the log inside its container, so both paths need hostPath mounts. Edit /etc/kubernetes/manifests/kube-apiserver.yaml:

    - --audit-policy-file=/etc/kubernetes/audit/policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=7
    - --audit-log-maxbackup=3
    - --audit-log-maxsize=100
  volumeMounts:
    - {name: audit-policy, mountPath: /etc/kubernetes/audit/policy.yaml, readOnly: true}
    - {name: audit-log, mountPath: /var/log/kubernetes/audit/}
volumes:
  - {name: audit-policy, hostPath: {path: /etc/kubernetes/audit/policy.yaml, type: File}}
  - {name: audit-log, hostPath: {path: /var/log/kubernetes/audit/, type: DirectoryOrCreate}}

Only --audit-policy-file and a backend together turn auditing on. Without a policy file the API server logs nothing. A missing mount or a YAML error makes the kubelet's restart of the API server fail, and kubectl stops answering. Check with crictl ps -a | grep kube-apiserver and crictl logs <id>.

§V. Verify

kubectl -n prod get secret db-creds
tail -n 50 /var/log/kubernetes/audit/audit.log | grep '"resource":"secrets"'

Illustrative expected line (not captured this fire), trimmed:

{"level":"Metadata","stage":"ResponseComplete","verb":"get","user":{"username":"kubernetes-admin"},"objectRef":{"resource":"secrets","namespace":"prod","name":"db-creds"},"responseStatus":{"code":200}}

Three checks: level is Metadata, there is no requestObject or responseObject, and there is no RequestReceived line for the same auditID.

§VI. Drills

1. A Secret read shows "level":"RequestResponse". Name the two possible causes. (A broader rule sits above the Secret rule, or the Secret rule is missing.)
2. The API server never returns after your edit. First command? (crictl ps -a, then the container logs.)
3. Logs show every request twice. Fix? (omitStages: ["RequestReceived"].)
4. Log only who deleted Pods in prod, nothing more. Level? (Metadata, with verbs: ["delete"].)

Close

Four Levels, First Match Wins, Never the Body. Narrow rules on top, Secrets at Metadata, a catch-all last, two hostPath mounts, then read the log. The Dev fold counts exactly these lines.

Related

• Tome: Poulton ch15 (threat model) (referenced)
• Prior CKS: attestation admission 10-03
• Web: Kubernetes Auditing · audit.k8s.io/v1 · kube-apiserver flags · CKS curriculum