An ECR repository census in Rust fail-closed checks, an error that is an answer, and a one-file static image built with Nix

Pass only what is known to be safe, treat "no policy" as an answer, and ship one static binary with the certificates it needs.

The problem

A team pushes container images to a few dozen repositories in Amazon ECR (Elastic Container Registry), the AWS service that stores container images. Deployments name an image by tag, a human-readable label such as v1. If a repository lets tags be overwritten, the same tag can point at different images over time, so v1 on a running service is not a fixed thing. Repositories with no lifecycle policy, a set of rules that expire old images, keep every image forever.

Checking a few dozen repositories by hand in a console does not last. The team wants a census that answers with an exit code, so it can run in CI (continuous integration) and fail a job when something is wrong. They also want to ship it as a small image with nothing else inside to patch, because every extra file in an image is something that may need an update.

The lab behind this lesson ran the Rust program against a local AWS emulator (moto) on 127.0.0.1 with throwaway credentials. The Linux builds ran inside a Linux container with Nix, a build tool that compiles software from pinned inputs, on an arm64 Mac.

The idea, in plain words

The program pages through the repositories with the paginator from the AWS SDK (software development kit) for Rust, a helper that keeps requesting the next page until none is left. For each repository it reads the tag mutability setting and asks for the lifecycle policy. It passes only the settings known to be safe and reports everything else, including values the code has never seen, because a check that names only the bad values silently accepts whatever it did not anticipate. A missing policy is a finding, not a crash, while every other error means the run is broken, so that an unreachable registry never looks like a clean one. Then the same source is built twice with Nix, once for the normal package set and once for the static one, and only the static binary goes into an image.

Tag mutability, checked fail-closed

An image tag is a label attached to an image in a repository. Tag mutability is a repository setting that decides whether a push may reuse a tag that already exists. ECR documents four values: MUTABLE, IMMUTABLE, IMMUTABLE_WITH_EXCLUSION and MUTABLE_WITH_EXCLUSION. With immutability on, ECR documents that a push reusing a tag returns the ImageTagAlreadyExistsException error. Both settings accept exclusions, which are tag names or wildcards (*) that flip the rule for a few tags. A check is fail-closed when doubt counts as a finding, so that a setting nobody anticipated is reported instead of ignored.

The seed script creates five repositories in different states:

mk web-api   --image-tag-mutability IMMUTABLE
mk worker
mk batch     --image-tag-mutability IMMUTABLE
mk legacy    --image-tag-mutability MUTABLE
mk edge      --image-tag-mutability IMMUTABLE_WITH_EXCLUSION --image-tag-mutability-exclusion-filters filterType=WILDCARD,filter=latest || echo "edge: create failed (exit $?)"

Each mk call wraps aws ecr create-repository and prints the name and setting the service stored. worker gets no flag at all, so it shows the default. edge uses the exclusion form from the ECR documentation, with one wildcard filter that names latest as the tag allowed to move. After the five repositories exist, the script puts the same lifecycle policy, which expires untagged images after 14 days, on three of them. Its output was:

web-api	IMMUTABLE
worker	MUTABLE
batch	IMMUTABLE
legacy	MUTABLE
edge	IMMUTABLE_WITH_EXCLUSION
web-api
worker
edge

The first five lines are each repository and its setting; worker came back MUTABLE, which is the default. The last three lines are the repositories given a policy, so batch and legacy have none. The census should therefore find problems in worker, batch and legacy, and leave web-api and edge alone, because both are immutable and both have a policy.

The check itself is one function:

fn tags_can_move(m: Option<&ImageTagMutability>) -> bool {
    !matches!(
        m,
        Some(ImageTagMutability::Immutable | ImageTagMutability::ImmutableWithExclusion)
    )
}

The function lists the two settings that pass, so MUTABLE, MUTABLE_WITH_EXCLUSION, an absent value and Unknown(...) are all reported. The SDK enum is #[non_exhaustive] and has an Unknown variant, which is how a service value newer than the SDK build reaches the code. Writing the reverse, matching only Mutable as bad, would silently pass such a value, because a deny-list reports only what it names. The repository edge passes because its exclusion is an intentional list of movable tags, here latest. A team that wants stricter rules may choose to report it too.

Running the script gave this:

TAGS-CAN-MOVE  worker  (MUTABLE)
NO-LIFECYCLE   batch
TAGS-CAN-MOVE  legacy  (MUTABLE)
NO-LIFECYCLE   legacy
repositories=5 findings=4
exit 1

Four findings from five repositories, and exit code 1 so a CI job fails. web-api and edge are absent because they are immutable and have policies.

The lab also tried to show a tag moving, by pushing two different image descriptions under v1. In worker the emulator returned the same digest, an identifier that should be derived from the image content, for both pushes:

-- worker
sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
first push exit 0
sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
second push exit 0
sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855	v1

The two image descriptions differed, yet both pushes got one digest, and the final listing shows a single image tagged v1. A registry that computes digests from content would have shown two digests and the tag moving from the first to the second. So this run does not demonstrate a tag moving, and the emulator cannot be used to prove it. In the immutable web-api the second push failed with a different name than the documentation gives:

aws: [ERROR]: An error occurred (ImageAlreadyExistsException) when calling the PutImage operation: Image with digest 'sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' and tag 'v1' already exists in the repository with name 'web-api' in registry with id '123456789012'

The documented error is ImageTagAlreadyExistsException. The overwrite behaviour described here therefore rests on the ECR documentation, not on this lab.

Symptom: A repository switched to MUTABLE_WITH_EXCLUSION never appears in the census.

Cause: The check was written as "report if Mutable", a deny-list that passes every value it does not name.

Fix: Write the allow-list form, as tags_can_move does, so new or unknown values are reported.

An error that is an answer

In Rust, an operation that can fail returns a Result, and every SDK call returns one whose error type is SdkError. When that error is a ServiceError, the request reached the AWS service but could not be processed. GetLifecyclePolicy reports a missing policy exactly this way, as the error LifecyclePolicyNotFoundException with HTTP (HyperText Transfer Protocol) status 400. For a census that error is the answer to the question asked, so it must not stop the run.

match client.get_lifecycle_policy().repository_name(name).send().await {
    Ok(_) => {}
    Err(e) if e
        .as_service_error()
        .is_some_and(|s| s.is_lifecycle_policy_not_found_exception()) =>
    {
        println!("NO-LIFECYCLE   {name}");
        findings += 1;
    }
    Err(e) => {
        return Err(format!("GetLifecyclePolicy {name}: {}", DisplayErrorContext(&e)));
    }
}

The match guard turns exactly one error kind into a finding. as_service_error() yields the service-level error when there is one, and is_lifecycle_policy_not_found_exception() is true only for that variant. A timeout, a permission failure or a connection problem fails the guard, falls to the last arm and returns an Err, which main maps to exit code 2. DisplayErrorContext is used there because the default display of some errors says only "dispatch failure", which hides the cause.

The naive variant of the script differs only in how it handles that call:

client
    .get_lifecycle_policy()
    .repository_name(name)
    .send()
    .await
    .map_err(|e| format!("GetLifecyclePolicy {name}: {}", DisplayErrorContext(&e)))?;

The trailing ? returns on the first error of any kind, and map_err only turns the error into a message first. Nothing in this chain looks at which error came back, so the documented answer "this repository has no policy" is handled exactly like a dropped connection. The repositories are visited in order, so the first one without a policy ends the run. Against the seeded emulator it printed one finding and stopped at batch:

TAGS-CAN-MOVE  worker  (MUTABLE)
ecr-census: GetLifecyclePolicy batch: service error: LifecyclePolicyNotFoundException: Lifecycle policy does not exist for the repository with name 'batch' in the registry with id '123456789012'
exit 2

legacy was never examined, so its two findings were lost.

The opposite failure matters just as much. With the emulator stopped, a different program on the same port answered the census:

status: StatusCode(403), headers: {"content-length": "0", "server": "AirTunes/980.77.5"

The census exited 2 after 1 second. A census that treated any failure as "no findings" would have reported zero problems for a registry it never read, and a CI job would have gone green.

Symptom: The census aborts at the first repository without a lifecycle policy, so later repositories are never checked.

Cause: Every GetLifecyclePolicy error was treated as fatal, including the one that means "no policy".

Fix: Match that one error kind with as_service_error() and is_lifecycle_policy_not_found_exception(), report it, and keep all other errors fatal.

One source, two package sets, a static musl binary

Linking decides where a program finds the C standard library it needs. A dynamically linked binary loads shared libraries from the file system when it starts, while a statically linked one carries what it needs inside the file. musl is a small C standard library, and Nix's static package set, pkgsStatic, builds against it. flake-parts is a library for writing Nix flakes, pinned projects described by one flake.nix, so that outputs are declared once per system. Here is the crate's Cargo.toml:

[package]
name = "ecr-census"
version = "0.1.0"
edition = "2024"
rust-version = "1.98"

[dependencies]
aws-config = { version = "1", features = ["behavior-version-latest"] }
aws-sdk-ecr = "1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

The dependencies are the same three the single-file script declared in its ---cargo fence, so the crate is the script body moved into src/main.rs with nothing added. The behavior-version-latest feature lets aws_config::load_from_env() pick the current default behaviour without naming a version in code. rust-version = "1.98" matches the rustc 1.98.1 in the pinned package set that will compile it, so the crate does not claim a newer compiler than the build has. Next is the flake:

{
  description = "ecr-census: native and static musl builds, plus two minimal images";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/fe13242281a67641c0102cdb1087f0ebe586c7df";
    flake-parts.url = "github:hercules-ci/flake-parts";
  };

  outputs = inputs@{ flake-parts, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {
      systems = [ "aarch64-linux" "x86_64-linux" "aarch64-darwin" ];
      perSystem = { pkgs, lib, self', ... }:
        let
          # One recipe, applied to two package sets: the host's and the static musl one.
          census = p: p.rustPlatform.buildRustPackage {
            pname = "ecr-census";
            version = "0.1.0";
            src = lib.fileset.toSource {
              root = ./.;
              fileset = lib.fileset.unions [ ./Cargo.toml ./Cargo.lock ./src ];
            };
            cargoLock.lockFile = ./Cargo.lock;
          };
          image = name: contents: env: pkgs.dockerTools.buildLayeredImage {
            inherit name contents;
            tag = "lab";
            config.Entrypoint = [ "${self'.packages.static}/bin/ecr-census" ];
            config.Env = env;
          };
        in
        {
          packages = {
            default = census pkgs;
            static = census pkgs.pkgsStatic;
          } // lib.optionalAttrs pkgs.stdenv.hostPlatform.isLinux {
            # The binary alone: no shell, no libc files, no CA bundle.
            image-bare = image "ecr-census-bare" [ ] [ ];
            # The same binary plus Mozilla's CA bundle, and the variable that points at it.
            image-certs = image "ecr-census-certs" [ pkgs.cacert ]
              [ "SSL_CERT_FILE=${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt" ];
          };
        };
    };
}

perSystem is a function from a system name to that system's outputs, and flake-parts hands it pkgs, lib and self', the flake's own outputs for the current system. census is one function applied to pkgs and to pkgs.pkgsStatic, so the two builds cannot drift apart. lib.fileset.toSource copies only the union of Cargo.toml, Cargo.lock and src into the build input, so edits to other files in the directory do not trigger a rebuild. cargoLock.lockFile makes Nix read dependency hashes from Cargo.lock. The image outputs exist only on Linux, through lib.optionalAttrs.

The first question was whether pkgsStatic really changes the target:

this Mac:        pkgs.stdenv.hostPlatform.config = arm64-apple-darwin
this Mac:  pkgsStatic.stdenv.hostPlatform.config = arm64-apple-darwin
container:       pkgs.stdenv.hostPlatform.config = aarch64-unknown-linux-gnu
container: pkgsStatic.stdenv.hostPlatform.config = aarch64-unknown-linux-musl

On the Mac both sets target macOS. Only in the Linux container does the static set switch to musl, so a static Linux binary had to be built on Linux, and the lab used a container. Both builds then succeeded:

== 2.2 static build (from the build log)
exit 0 after 118 s
== 2.3 glibc build of the same source
exit 0 after 77 s

Timings come from one machine and are rough. Here is what each binary needs at run time:

result-default/bin/ecr-census: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/…-glibc-2.44-25/lib/ld-linux-aarch64.so.1, for GNU/Linux 3.10.0, not stripped
result-static/bin/ecr-census:  ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped
21276640 result-default/bin/ecr-census
20577088 result-static/bin/ecr-census

The file tool reports that the first binary names an interpreter, the glibc loader that must exist on the machine, while the second needs nothing. The files differ by under a megabyte. The closure, meaning the package plus everything it references at run time, shows the real difference. The command nix path-info -rsSh lists it, with own size and closure total in the last two columns:

-- result-default closure
/nix/store/…-glibc-2.44-25     	  44.8 MiB	  47.3 MiB
/nix/store/…-gcc-16.2.0-lib    	   9.7 MiB	  57.1 MiB
/nix/store/…-ecr-census-0.1.0  	  20.3 MiB	  77.4 MiB
-- result-static closure
/nix/store/…-ecr-census-static-aarch64-unknown-linux-musl-0.1.0	  19.6 MiB	  19.6 MiB

The glibc closure totals 77.4 MiB, most of it glibc and gcc libraries that any image would have to carry next to the binary (a few small libraries are trimmed from the listing). The static closure is the binary alone at 19.6 MiB. Nixpkgs lists aarch64-unknown-linux-musl as Tier 3, where "updates might break builds", so pin nixpkgs, as the flake does, and build the static package in CI. Rust also has a crt-static target feature controlling how the C runtime is linked. The lab did not set it, and the file output above is the only evidence of what this target did.

Symptom: nix build .#static on a Mac yields a Mac binary, and the image step is not even offered.

Cause: pkgsStatic keeps the host platform, so on macOS it stays arm64-apple-darwin, and the flake defines images only on Linux.

Fix: Build on a Linux machine, a Linux builder or a container, and check the result with file before packaging.

A one-file image, and the CA certificates it lacks

dockerTools.buildLayeredImage builds a compressed Docker-compatible tarball. Its contents argument is like ADD contents/ / in a Dockerfile, the file that describes an image build, and a null base image corresponds to FROM scratch, so with none given the result starts from nothing. The two image lines in the flake above use this: image-bare adds nothing, and image-certs adds pkgs.cacert. In both, the entrypoint is the static binary. Exporting the bare image listed this:

.dockerenv
dev/
dev/console
dev/pts/
dev/shm/
etc/
etc/hostname
etc/hosts
etc/mtab
etc/resolv.conf
nix/
nix/store/
nix/store/…-ecr-census-static-aarch64-unknown-linux-musl-0.1.0/
nix/store/…-ecr-census-static-aarch64-unknown-linux-musl-0.1.0/bin/
nix/store/…-ecr-census-static-aarch64-unknown-linux-musl-0.1.0/bin/ecr-census
proc/
sys/

The only real content is the static binary in its store path. There is no shell, no C library and no certificate directory. The dev/, proc/, sys/ and etc/ entries are empty mount points and a few files that docker export shows for the container; the lab did not inspect them. The compressed image archives were 6975039 bytes (bare) and 7247906 bytes (with certificates), while docker image ls showed 41.2MB and 42.5MB on a Docker using the containerd image store. These figures measure different things, and the archive size is the transfer size.

Against the emulator over plain HTTP, the bare image behaved like the script:

TAGS-CAN-MOVE  worker  (MUTABLE)
NO-LIFECYCLE   batch
TAGS-CAN-MOVE  legacy  (MUTABLE)
NO-LIFECYCLE   legacy
repositories=5 findings=4
exit 1

Same four findings, same exit code. The image was started on a Docker network shared with the emulator, with AWS_ENDPOINT_URL=http://moto-ecr:5000 and the throwaway credentials passed as environment variables, so the only thing that changed from the cargo script run is where the binary came from. Plain HTTP needs no certificates, which is why this test passes and proves nothing about TLS. Then the same bare image was pointed at the real ECR endpoint, over HTTPS (HTTP carried inside TLS, Transport Layer Security), and then the image with certificates:

== 2.7 bare image against the real ECR endpoint (HTTPS, fake credentials)
ecr-census: DescribeRepositories: dispatch failure: io error: client error (Connect): TLS handshake failed: invalid peer certificate: UnknownIssuer
exit 2

== 2.8 image with the CA bundle against the real ECR endpoint (HTTPS, fake credentials)
ecr-census: DescribeRepositories: service error: unhandled error (UnrecognizedClientException): Error { code: "UnrecognizedClientException", message: "The security token included in the request is invalid."
exit 2

TLS clients verify a server's certificate against CA (certificate authority) certificates, and the bare image contains none, so the handshake fails with UnknownIssuer. Cargo.lock lists rustls-native-certs 0.8.5, a crate that loads root certificates from the platform's store and, when SSL_CERT_FILE is set, only from the file that variable names. The second image adds pkgs.cacert and sets that variable. The handshake then succeeded and AWS answered with UnrecognizedClientException, which proves the connection worked. The credentials were the throwaway lab values on purpose, so no real account was touched.

Symptom: The image passes every test against a local emulator and fails against real AWS with "invalid peer certificate: UnknownIssuer".

Cause: The tests used plain HTTP, and the image has no CA certificates for HTTPS.

Fix: Add pkgs.cacert to contents and set SSL_CERT_FILE to its ca-bundle.crt, then test once against a real HTTPS endpoint, as step 2.8 did.

Putting it together

Run the census in CI with read-only credentials. Treat exit 1 as a ticket list: switch repositories to immutable tags where deployments depend on tags, and add a lifecycle policy after using the lifecycle policy preview that ECR recommends, because expiry can remove images, and ECR documents that images expire within 24 hours of meeting the criteria. Treat exit 2 as a broken job, never as a pass. Build the static image from the pinned flake on Linux and ship the variant with the certificates.

Verified in the lab: aws-sdk-ecr 1.137.0 and aws-config 1.13.0 against moto 5.2.4.dev0, with the cargo script on a nightly cargo; Nix builds in a nixos/nix container on aarch64-linux with the Nix sandbox off; Docker 29.8.2. Not verified: a real account, permissions and rate limits; multi-page results, since five repositories fit one page; the emulator's tag behaviour (see the v1 pushes above); x86_64 builds; and lifecycle rules actually expiring images.

Check yourself

Sources

1. Amazon ECR User Guide: Preventing image tags from being overwritten
2. Amazon ECR API Reference: Repository data type
3. Amazon ECR API Reference: GetLifecyclePolicy
4. Amazon ECR User Guide: Automate the cleanup of images by using lifecycle policies
5. docs.rs: aws-sdk-ecr, enum ImageTagMutability
6. docs.rs: aws-sdk-ecr, enum GetLifecyclePolicyError
7. AWS SDK for Rust Developer Guide: Handling errors
8. AWS SDK for Rust Developer Guide: Using paginated results
9. AWS SDKs and Tools Reference Guide: Service-specific endpoints
10. The Cargo Book: Unstable features, script
11. flake-parts: Options reference
12. Nixpkgs manual
13. docs.rs: rustls-native-certs, load_native_certs
14. The Rust Reference: Linkage, static and dynamic C runtimes