Hedronite · Ops Lesson · GKE · Sun 2026-10-18

Pods that still borrow the node's identity a Rust census of GKE Workload Identity, guarded by a flake check

The cluster switch is on. Check every node pool anyway, and make the checker fail its own build when it drifts.

Lesson Class: Ops (GKE security · Rust automation · Nix packaging)
Tested with: cargo 1.101.0-nightly script · nixpkgs rustc 1.98.1 · Nix 2.35.2
Lab: census/ crate + flake · synthetic fixture · real logs in out/
Source: Google Cloud GKE docs and API reference · Nixpkgs manual
Two Switches
The cluster pool, then GKE_METADATA on every node pool.
Unset Is Not Safe
A missing mode is VERIFY, never OK.
The Checker Is Checked
nix flake check runs the census over a fixture.
The cluster said enabled. The batch pool still handed out the node's token.

The problem

A platform team runs several Standard Google Kubernetes Engine (GKE) clusters. Last quarter someone enabled Workload Identity Federation for GKE on one of them and assumed every pod in that cluster now has its own identity. Three things make that assumption wrong. Enabling the feature at the cluster level leaves existing node pools unaffected, so pools created earlier keep the old behavior. A pool can also be set explicitly to GCE_METADATA, which opts that pool out even though the cluster is enabled. And an older cluster may never have been enabled at all.

On any of those nodes, a pod can call the Compute Engine metadata server at http://metadata.google.internal (169.254.169.254) and receive a token for the node's service account. If nobody chose a service account when the pool was created, that is the project's Compute Engine default service account, shared by every workload on the node. Google's own docs call this over-provisioning that violates least privilege and is inappropriate for multi-tenant clusters. One compromised pod then holds whatever the node holds.

Because the cluster-level setting looks enabled, the gap is easy to miss, and nobody notices until an audit or an incident finds it.

The idea, in plain words

Workload Identity is two switches: the cluster gets a workload pool first, and then each node pool runs the GKE metadata server. The census reads both settings from the JSON that gcloud container clusters list --format=json prints, and it flags any pool that is not on GKE_METADATA, because a pool in any other state may still hand out the node's credentials. The census then lives in a Nix flake whose check builds the tool, runs its unit test, and runs the binary over a known fixture. A code change that breaks the audit therefore fails the build instead of quietly reporting everything as fine.

Tested with: cargo 1.101.0-nightly (2026-09-29) for the script; inside the Nix build, nixpkgs rustc 1.98.1; Determinate Nix 3.22.3 (nix 2.35.2) on aarch64-darwin; serde 1 and serde_json 1.

The two switches and what the API returns

Workload Identity Federation for GKE lets workloads in pods authenticate to Google Cloud APIs with an identity of their own. Two kinds of service account are involved, and they are easy to confuse. A Kubernetes ServiceAccount is a Kubernetes resource that provides an identity for processes running in your pods. An IAM service account (IAM is Identity and Access Management) is a Google Cloud resource that lets applications make authorized calls to Google Cloud APIs. The node service account is the IAM service account attached to the node virtual machines, and it is the one this audit worries about.

Enabling the feature creates a fixed workload identity pool for the project, in the format PROJECT_ID.svc.id.goog. It also deploys the GKE metadata server on every node. It intercepts credential requests: when code asks for a token through Application Default Credentials (ADC), the GKE metadata server fetches a Kubernetes ServiceAccount token (a JSON Web Token signed by the API server) and trades it at Security Token Service for a short-lived federated access token. Code built on the Google Cloud client libraries needs no changes, because it still talks to the same metadata address.

Once a node pool runs the GKE metadata server, its pods can no longer reach the Compute Engine metadata server (pods on the host network are the exception). A pool without it still serves node credentials, so the per-pool switch is the one worth checking.

The two switches are separate, and the order is fixed. Workload Identity Federation for GKE must be enabled at the cluster level before you can enable it on node pools. The cluster switch is workloadIdentityConfig.workloadPool, whose documented meaning is "The workload pool to attach all Kubernetes service accounts to." The node pool switch is nodePools[].config.workloadMetadataConfig.mode, an enum with these values:

Two more fields matter. nodePools[].config.serviceAccount is the account the node VMs use, and if none is specified the "default" service account is used. autopilot.enabled marks an Autopilot cluster, where Workload Identity Federation for GKE is always enabled.

The fixture below is synthetic: the project example-proj does not exist, and the data was written to exercise every branch of the audit. Real gcloud output has many more fields, and this fixture keeps only the ones the census reads.

[
  {
    "name": "payments-prod",
    "location": "us-central1",
    "workloadIdentityConfig": { "workloadPool": "example-proj.svc.id.goog" },
    "nodePools": [
      {
        "name": "default-pool",
        "config": {
          "serviceAccount": "[email protected]",
          "workloadMetadataConfig": { "mode": "GKE_METADATA" }
        }
      },
      {
        "name": "batch",
        "config": {
          "serviceAccount": "default",
          "workloadMetadataConfig": { "mode": "GCE_METADATA" }
        }
      },
      {
        "name": "gpu",
        "config": {
          "serviceAccount": "[email protected]"
        }
      }
    ]
  },
  {
    "name": "legacy-tools",
    "location": "us-east1-b",
    "nodePools": [
      { "name": "default-pool", "config": { "serviceAccount": "default" } }
    ]
  },
  {
    "name": "web",
    "location": "us-central1",
    "autopilot": { "enabled": true },
    "workloadIdentityConfig": { "workloadPool": "example-proj.svc.id.goog" },
    "nodePools": []
  }
]

Each cluster covers one case. payments-prod has its workload pool set, but its pools differ: default-pool is on GKE_METADATA, batch is explicitly GCE_METADATA and runs as the shared default account, and gpu reports no mode at all. legacy-tools has no workloadIdentityConfig, the shape of a cluster that never enabled the feature. web is Autopilot and clean by construction.

The commands that move a cluster from bad to good come from the GKE documentation:

gcloud container clusters update CLUSTER_NAME --location=LOCATION --workload-pool=PROJECT_ID.svc.id.goog
gcloud container node-pools create NODEPOOL_NAME --cluster=CLUSTER_NAME --location=CONTROL_PLANE_LOCATION --workload-metadata=GKE_METADATA
gcloud container node-pools update NODEPOOL_NAME --cluster=CLUSTER_NAME --location=CONTROL_PLANE_LOCATION --workload-metadata=GKE_METADATA

The first command sets the cluster switch on an existing Standard cluster. The second creates a pool with the GKE metadata server; new pools get it by default once the cluster is enabled, but the explicit flag makes creation fail on a cluster that is not, which beats a pool that silently lacks it. The third flips an existing pool. Passing --workload-metadata=GCE_METADATA instead opts one pool out, which is how pools like batch come to exist.

Mistake: enabling at the cluster level and assuming the pools followed.

The census as a cargo script

A cargo script is a single .rs file that carries its own package description in a ---cargo frontmatter block. You run it with cargo +nightly -Zscript file.rs; script support is an unstable Cargo feature, so it needs the nightly toolchain (-Z enables unstable flags). The format suits a first audit because the whole tool is one file you can paste into a ticket and run with one command.

The file starts with its dependency block, which pulls in serde for deriving JSON parsing and serde_json for reading the input:

---cargo
[package]
edition = "2024"

[dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
---

The structs mirror only the fields the audit reads. Every optional field carries #[serde(default)] so that a cluster missing a whole section, like legacy-tools with no workloadIdentityConfig, still parses; otherwise the census would crash on exactly the clusters it exists to report.

use serde::Deserialize;
use std::io::Read;

#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
struct Cluster {
    name: String,
    location: String,
    #[serde(default)]
    autopilot: Option<Autopilot>,
    #[serde(default)]
    workload_identity_config: Option<WorkloadIdentityConfig>,
    #[serde(default)]
    node_pools: Vec<NodePool>,
}

#[derive(Deserialize)]
struct Autopilot {
    #[serde(default)]
    enabled: bool,
}

#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
struct WorkloadIdentityConfig {
    #[serde(default)]
    workload_pool: String,
}

#[derive(Deserialize)]
struct NodePool {
    name: String,
    #[serde(default)]
    config: Option<NodeConfig>,
}

#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
struct NodeConfig {
    #[serde(default)]
    service_account: Option<String>,
    #[serde(default)]
    workload_metadata_config: Option<WorkloadMetadataConfig>,
}

#[derive(Deserialize)]
struct WorkloadMetadataConfig {
    #[serde(default)]
    mode: Option<String>,
}

#[derive(Debug, PartialEq)]
enum Finding {
    /// Cluster has no workload pool: every pod can reach node credentials.
    ClusterOff { cluster: String },
    /// Node pool exposes Compute Engine metadata (GCE_METADATA) to its pods.
    PoolExposed { cluster: String, pool: String, node_sa: String },
    /// Node pool reports no mode, or one this census does not know: verify by hand.
    PoolUnverified { cluster: String, pool: String, mode: String },
}

rename_all = "camelCase" maps Rust's workload_pool to the API's workloadPool. The Finding enum is the census's vocabulary of problems, one variant per kind; keeping findings as data instead of printing as it goes lets a unit test assert on them, which the Nix section depends on. The rules live in audit:

fn audit(clusters: &[Cluster]) -> (Vec<String>, Vec<Finding>) {
    let mut ok = Vec::new();
    let mut findings = Vec::new();
    for c in clusters {
        let id = format!("{}/{}", c.location, c.name);
        if c.autopilot.as_ref().is_some_and(|a| a.enabled) {
            ok.push(format!("{id}: Autopilot (always on)"));
            continue;
        }
        let pool = c
            .workload_identity_config
            .as_ref()
            .map(|w| w.workload_pool.as_str())
            .unwrap_or("");
        if pool.is_empty() {
            findings.push(Finding::ClusterOff { cluster: id });
            continue;
        }
        let mut exposed = false;
        for np in &c.node_pools {
            let cfg = np.config.as_ref();
            let mode = cfg
                .and_then(|n| n.workload_metadata_config.as_ref())
                .and_then(|w| w.mode.clone())
                .unwrap_or_else(|| "UNSET".to_string());
            if mode == "GKE_METADATA" {
                continue;
            }
            exposed = true;
            if mode == "GCE_METADATA" {
                let node_sa = cfg
                    .and_then(|n| n.service_account.clone())
                    .unwrap_or_else(|| "default".to_string());
                findings.push(Finding::PoolExposed {
                    cluster: id.clone(),
                    pool: np.name.clone(),
                    node_sa,
                });
            } else {
                findings.push(Finding::PoolUnverified {
                    cluster: id.clone(),
                    pool: np.name.clone(),
                    mode,
                });
            }
        }
        if !exposed {
            ok.push(format!("{id}: pool {pool}, all node pools GKE_METADATA"));
        }
    }
    (ok, findings)
}

The main function, which reads standard input, calls audit, prints the lines, and exits, is omitted here. The loop applies four rules in order:

  1. Autopilot is skipped, because Autopilot always has the feature on.
  2. **An empty or missing workloadPool is ATTENTION** (ClusterOff). No pool can run the GKE metadata server without the cluster switch, so the pools are not examined one by one.
  3. **GCE_METADATA is ATTENTION** (PoolExposed), and the finding names the node service account, falling back to default as the API reference describes when the field is absent.
  4. **Anything else is VERIFY** (PoolUnverified). A missing mode becomes UNSET, and the census refuses to call it safe, because a clean report built on a guess is worse than a loud one.

Running it over the fixture with the nightly toolchain gives:

$ cargo +nightly -Zscript gke-wi-census.rs < census/fixtures/clusters.json
OK        us-central1/web: Autopilot (always on)
ATTENTION us-central1/payments-prod: node pool batch is GCE_METADATA, pods get node SA default
VERIFY    us-central1/payments-prod: node pool gpu mode=UNSET, not GKE_METADATA
ATTENTION us-east1-b/legacy-tools: workload identity OFF (no workloadPool)
3 cluster(s), 3 finding(s)
exit=2

Every fixture case appears once, and web is the only clean line. The program exits 0 when there are no findings and 2 when there is at least one, so a continuous integration (CI) job can gate on the code without parsing text. A broken input produces a third, different code, because expect panics and a Rust panic exits with 101:

$ echo "{}" | cargo +nightly -Zscript gke-wi-census.rs
thread 'main' (18801220) panicked at gke-wi-census.rs:129:61:
parse cluster list: Error("invalid type: map, expected a sequence", line: 1, column: 0)
exit=101

A CI step can therefore tell "clean" (0), "findings" (2) and "the census itself could not read its input" (101) apart. That last case matters: a pipeline that treats every non-zero code as "findings" would hide a gcloud call that printed an error page instead of JSON.

Mistake: treating a missing mode field as safe.

Promote the script to a flake with checks

A Nix flake is a directory with a flake.nix file that declares its inputs (such as nixpkgs, the Nix package collection) and its outputs (such as packages and checks). nix flake check evaluates the flake and builds every derivation listed under its checks output. A check passes when its derivation builds, so any assertion you want enforced has to be able to fail the build. flake-parts is a library that organizes flake outputs per system. You write a perSystem module once, and it is instantiated for each system in your systems list. Inside it, pkgs is nixpkgs for the current system, and self' is the flake's own outputs with the system already selected, so self'.packages.default means "this flake's default package for this machine".

The package comes from rustPlatform.buildRustPackage, nixpkgs's helper for Rust applications. Its cargoLock.lockFile argument vendors dependencies from Cargo.lock using fixed-output derivations, which pins them to the exact versions and hashes recorded in the lockfile. By default its check phase runs cargo test in release mode, so the unit test gates the build for free. The flake needs a normal Cargo project: the script's code lives in src/main.rs with a Cargo.toml, and cargo writes Cargo.lock. Because this is an ordinary package built with nixpkgs's rustc 1.98.1, no nightly toolchain is needed.

{
  description = "GKE Workload Identity census, built and tested as a flake check";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-parts.url = "github:hercules-ci/flake-parts";
  };

  outputs = inputs@{ flake-parts, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {
      systems = [ "aarch64-darwin" "x86_64-linux" "aarch64-linux" ];

      perSystem = { pkgs, self', ... }: {
        packages.default = pkgs.rustPlatform.buildRustPackage {
          pname = "gke-wi-census";
          version = "0.1.0";
          src = pkgs.lib.fileset.toSource {
            root = ./.;
            fileset = pkgs.lib.fileset.unions [
              ./Cargo.toml ./Cargo.lock ./src ./fixtures
            ];
          };
          cargoLock.lockFile = ./Cargo.lock;
          # checkPhase runs `cargo test`, so the fixture test gates the build.
        };

        # Run the built binary over the fixture; exit 2 means "findings", which is
        # what this fixture must produce. Any other code fails the check.
        checks.census-fixture = pkgs.runCommand "census-fixture" { } ''
          set +e
          ${self'.packages.default}/bin/gke-wi-census < ${./fixtures/clusters.json} > report.txt
          code=$?
          set -e
          cat report.txt
          test "$code" -eq 2
          grep -q '3 finding(s)' report.txt
          cp report.txt $out
        '';

        checks.package = self'.packages.default;
      };
    };
}

mkFlake takes the inputs and one module; systems lists the platforms that get outputs. The fileset copies only Cargo.toml, Cargo.lock, src and fixtures into the build, so unrelated edits do not trigger rebuilds. fixtures must be included because the unit test reads fixtures/clusters.json with include_str! at compile time; leave it out and the test fails to compile inside the sandbox.

census-fixture runs the built binary over the fixture. It suspends set -e around the run so the expected exit 2 does not abort the script, then asserts that the code is exactly 2 and that the report says 3 finding(s). The final cp gives the derivation its output. The unit test covers audit; this check covers the whole binary, including printing and the exit code. checks.package re-exports the package so its build, tests included, is part of the check run.

A green run on the lab machine looked like this, trimmed to the relevant lines:

$ nix flake check -L
✅ packages.aarch64-darwin.default (build skipped)
gke-wi-census> running 1 test
gke-wi-census> test tests::fixture_findings ... ok
gke-wi-census> test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
census-fixture> OK        us-central1/web: Autopilot (always on)
census-fixture> ATTENTION us-central1/payments-prod: node pool batch is GCE_METADATA, pods get node SA default
census-fixture> VERIFY    us-central1/payments-prod: node pool gpu mode=UNSET, not GKE_METADATA
census-fixture> ATTENTION us-east1-b/legacy-tools: workload identity OFF (no workloadPool)
census-fixture> 3 cluster(s), 3 finding(s)
✅ checks.aarch64-darwin.census-fixture
✅ checks.aarch64-darwin.package
warning: The check omitted these incompatible systems: aarch64-linux, x86_64-linux
exit=0

The gke-wi-census> lines are the unit test passing in the package's check phase; the census-fixture> lines are the binary's report, identical to the script run. Read the final warning: only aarch64-darwin was checked, and the two Linux systems in systems were skipped, so this green result says nothing about them until a Linux machine (or --all-systems with a builder for them) runs the check.

To prove that the check can fail, the fixture was edited so the batch pool reads GKE_METADATA. This leaves two findings instead of three:

# fixture edited: batch pool now GKE_METADATA
$ nix flake check -L
gke-wi-census> test tests::fixture_findings ... FAILED
gke-wi-census> thread 'tests::fixture_findings' (18794175) panicked at src/main.rs:153:9:
gke-wi-census>   left: 2
gke-wi-census>  right: 3
gke-wi-census> test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
error: Cannot build '/nix/store/a6c23kb6h180bvzrls4w1br52sl3c9di-gke-wi-census-0.1.0.drv'.
       Reason: builder failed with exit code 101.
       > test tests::fixture_findings ... FAILED
exit=1

In assert_eq!(findings.len(), 3) the left value is the real count (2) and the right is the expected one (3). The panic fails the check phase, the builder exits 101 (the same panic code as above), and nix exits 1. census-fixture never ran because it needs the package, so the failure stopped at the first and cheapest gate.

Mistake: a fixture or lockfile the flake cannot see.

Putting it together

The operator loop has four steps. Export the cluster list, run the census over it, fix what it flags, and run it again until it exits 0:

gcloud container clusters list --format=json > clusters.json
cargo +nightly -Zscript gke-wi-census.rs < clusters.json
echo "exit=$?"

Keeping the JSON in a file lets you attach the exact input to a ticket. For each ATTENTION or VERIFY line, apply the matching command from the first section: enable the workload pool on a cluster that has none, or move the flagged pool to GKE_METADATA.

Take care with the second fix. Modifying a pool "immediately enables Workload Identity Federation for GKE for any workloads running in the node pool", which "prevents the workloads from using the service account that your nodes use and might result in disruptions." Any workload that was quietly relying on the node's credentials stops working the moment you flip the switch. The documented best practice is to create new node pools if you also need to modify your applications to be compatible, move the workloads over, and retire the old pool.

Meanwhile CI runs nix flake check, so an edit that changes the rules, the output or the exit code fails against the fixture before anyone trusts a clean report from a broken auditor.

Some facts bound what a clean census proves:

Check yourself

  1. **A cluster has workloadPool set. Is every pod in it protected?** No. Existing node pools are unaffected by the cluster switch, and a pool can be set to GCE_METADATA explicitly, so each pool's workloadMetadataConfig.mode must be checked.
  1. **Why does the census report a pool with no mode field as VERIFY rather than OK?** A missing field proves nothing about whether the GKE metadata server runs there. Assuming safety would let an exposed pool pass silently.
  1. **The fixture edit left 2 findings, and nix flake check failed. Which component failed first, and why did census-fixture not run?** The unit test failed during the package's check phase (left: 2, right: 3, builder exit code 101). The census-fixture check uses the package, which never finished building.
  1. **You added a new fixtures/ file and nix flake check cannot find it. What do you do?** Run git add on the file. Inside a Git repository, a flake sees only files added to the repository, and uncommitted but added files give a dirty-tree warning rather than an error.

Sources