Hedronite · Ops Lesson · Nickel + Terraform · Tue 2026-10-20

Nickel to Terraform JSON environment overlays, merge priorities, and a Nix check that fails bad changes

One base, small overlays, a security floor no environment can quietly weaken, and a build that runs the export and terraform validate.

Lesson Class: Ops (configuration generation · Terraform · Nix gate)
Tested with: Nickel 1.18.0 · Terraform 1.16.5 + google provider 8.5.0 · Nix 2.35.2
Lab: base + overlays · six broken overlays · flake.nix · real transcripts in out/
Source: Nickel user manual · Terraform docs · Google Cloud Storage docs · Nix manuals · Brikman ch. 3
Default Is a Knob
Fields an environment may change are marked default; everything else is the base's.
Force Is Silent
force wins every merge without telling anyone; a contract on the field says no out loud.
Two Checkers
Nickel checks the policy you wrote; terraform validate checks the provider schema.
The overlay said false. The bucket said true. Nobody was told.

The problem

A team runs the same Google Cloud Storage (GCS) bucket for logs in two environments, dev and prod. Their Terraform lives in one folder per environment, the isolation layout Brikman's Terraform: Up & Running describes: "Put the Terraform configuration files for each environment into a separate folder." The same book names the cost: "Copy/paste: The file layout described in this section has a lot of duplication."

Duplication invites drift. Someone relaxes a setting in the dev copy while debugging, the diff looks harmless in review, and from then on the copies disagree without anyone deciding they should. Two settings matter most for a log bucket: uniform bucket-level access and public access prevention. Together they form the security floor, meaning the settings every environment must keep. Each is only as safe as the last copy-paste.

The team wants one base definition and small per-environment files, called overlays below, each merged over the base. They want a rule that no overlay can change the floor, errors that name the file that broke it, and a check that runs before terraform plan so a bad change fails early.

The idea, in plain words

Nickel is a configuration language. It evaluates a base record merged with an environment record and writes the result as plain JSON (JavaScript Object Notation). Terraform reads files ending in .tf.json as configuration, so the exported file needs no conversion step. Merge priorities decide which side wins when base and overlay set the same field, so file order never matters. Contracts, runtime checks attached to fields, reject bad values even when an overlay tries to force its way through. A Nix flake check runs the pipeline on every change: export both environments, assert the floor, run terraform validate, and confirm that known-bad overlays are rejected. Any failing step fails the build.

Tested with: Nickel 1.18.0, Terraform 1.16.5 with terraform-provider-google 8.5.0 (both from nixpkgs at commit 7fd4004, 2026-10-11), jq 1.8.2 and Determinate Nix 3.22.3 (nix 2.35.2) on aarch64-darwin. No Google Cloud credentials were used, and nothing was planned or applied.

Terraform's JSON syntax and the .tf.json file

Terraform "expects native syntax for files named with a .tf suffix, and JSON syntax for files named with a .tf.json suffix." A module is "a collection of .tf and/or .tf.json files kept together in a directory", and Terraform treats the whole module as a single document. The JSON form exists for generated configuration, because existing JSON libraries can prepare the files, so a generated main.tf.json can sit in envs/dev like any hand-written file.

Here is the dev file exactly as Nickel exported it:

{
  "resource": {
    "google_storage_bucket": {
      "logs": {
        "force_destroy": true,
        "labels": {
          "env": "dev",
          "managed_by": "terraform",
          "team": "platform"
        },
        "location": "US",
        "name": "acme-logs-dev",
        "public_access_prevention": "enforced",
        "storage_class": "STANDARD",
        "uniform_bucket_level_access": true,
        "versioning": {
          "enabled": false
        }
      }
    }
  },
  "terraform": {
    "required_providers": {
      "google": {
        "source": "hashicorp/google",
        "version": ">= 8.0, < 9.0"
      }
    }
  }
}

The root object's properties are Terraform's top-level block types. Native syntax would read resource "google_storage_bucket" "logs" { ... }; in JSON the two labels become two levels of nesting, type then name. versioning has no label, so it is a plain object. A nested block's value may be a JSON object or a JSON array, and prod uses an array for lifecycle_rule. The terraform block constrains the Google provider to the 8.x series.

Two arguments carry the floor. With uniform bucket-level access on, Access Control Lists (ACLs) are disabled, and "only bucket-level Identity and Access Management (IAM) permissions grant access to that bucket and the objects it contains". The provider default is false, and Google notes that uniform bucket-level access "cannot be disabled after it has been active on a bucket for 90 consecutive days". With public access prevention set to "enforced", "no one can make data in applicable buckets public through IAM policies or ACLs". The provider default is "inherited", which defers to an organization policy constraint. A bucket that never says "enforced" is only as safe as that outside policy.

Symptom: terraform fmt -check -recursive envs passes on the generated directory, which suggests the JSON was checked:

$ terraform fmt -check -recursive -no-color envs
[exit 0]

Cause: fmt formats native-syntax files to the canonical style. The run shows only that it reported nothing for the .tf.json files; it checked nothing about them.

Fix: Do not count a clean fmt as a check on generated JSON. Formatting is the generator's job (nickel export), and the content check is terraform validate, covered below.

Nickel records, merging, and priorities

Nickel is "a generic configuration language" for generating static configuration files, in essence "JSON with functions". Its data building block is the record, a set of named fields (an object in JSON). Records are combined with the merge operator &, which is symmetric, so left & right is the same as right & left. Fields present in both records are merged recursively, and two different plain values at the same priority cannot merge.

Priorities settle disagreements without relying on order. A field marked default has the lowest priority and force the highest, with values that carry no annotation at priority 0 in between. "If the priorities differ, the value with the highest priority simply erases the other", so {foo | default = 1} & {foo = 2} evaluates to {foo = 2}. The base marks the knobs an environment may turn as default; otherwise every overlay edit would be a merge conflict.

let S = import "schema.ncl" in
{
  # Inputs every environment sets. Used below, never written to the JSON.
  environment | S.Env | not_exported,
  owner_team | String | not_exported | default = "platform",

  terraform.required_providers.google = {
    source = "hashicorp/google",
    version = ">= 8.0, < 9.0",
  },

  resource.google_storage_bucket.logs = {
    name | S.BucketName | default = "acme-logs-%{environment}",
    location | default = "US",
    storage_class | default = "STANDARD",
    force_destroy | default = false,
    versioning.enabled | default = false,

    # Security floor, held two different ways (see the lesson).
    uniform_bucket_level_access | force = true,
    public_access_prevention | std.contract.Equal "enforced" = "enforced",

    labels = {
      env = environment,
      team = owner_team,
      managed_by = "terraform",
    },
  },
}

environment and owner_team are not_exported: they take part in evaluation but never appear in the output, so Terraform does not see them. environment has no default, so each overlay supplies it, and the S.Env contract accepts only "dev" or "prod". The bucket name is built by string interpolation, acme-logs-%{environment}, so it is never typed twice. Location, storage class, force_destroy and versioning are default knobs, and the labels reuse the two inputs, which is why the dev export shows "team": "platform". The two floor lines get their own section below.

(import "base.ncl")
& {
  environment = "dev",
  resource.google_storage_bucket.logs.force_destroy = true,
}

The dev overlay, dev.ncl, imports the base and merges one record over it. It sets environment and turns on force_destroy, which the provider documents as deleting all objects before deleting the bucket (otherwise a bucket with objects fails to delete), so disposable dev buckets can be torn down. The base value was a default, so the plain true wins with no annotation, giving the "force_destroy": true line in the JSON above.

(import "base.ncl")
& {
  environment = "prod",
  owner_team = "sre",
  resource.google_storage_bucket.logs = {
    location = "EU",
    versioning.enabled = true,
    lifecycle_rule = [
      {
        condition.num_newer_versions = 3,
        action.type = "Delete",
      },
    ],
  },
}

prod.ncl changes four things: the owner team, the location, versioning, and a new lifecycle_rule array holding one rule. The base never mentioned lifecycle_rule, so it merges as a new field with no priority involved. Everything prod does not mention, including both floor settings and force_destroy, keeps its base value. That is the point of an overlay: prod states only how it differs, and a reviewer reads four changes instead of a full copy of the bucket. Selecting a few fields from the export:

{"name":"acme-logs-prod","location":"EU","versioning":{"enabled":true},"labels":{"env":"prod","managed_by":"terraform","team":"sre"}}

Symptom: A second overlay is layered over prod and also sets location. Export fails:

error: non mergeable terms
  ┌─ /private/tmp/gcs-bucket-lab/prod.ncl:6:16
  │
6 │     location = "EU",
  │                ^^^^ cannot merge this expression
  │
  ┌─ /private/tmp/gcs-bucket-lab/broken/conflict.ncl:3:52
  │  
2 │ ╭ (import "../prod.ncl")
3 │ │ & { resource.google_storage_bucket.logs.location = "US" }
  │ │                                                    ^^^^ with this expression
  │ ╰─────────────────────────────────────────────────────────' originally merged here
  │  
  = Merge operands have the same merge priority but they can't be combined.
  = Both values are of type String but they aren't equal.
  = String values can only be merged if they are equal

Cause: Both values sit at priority 0, and two different strings cannot be combined. The error names both sides, prod.ncl line 6 ("EU") and broken/conflict.ncl line 3 ("US"), so the reviewer knows exactly which two files disagree.

Fix: Decide which layer owns the field. Keep the base value a default so a single overlay can set it, and remove one of the two definitions. Do not reach for force to settle a dispute between environments, because it erases the other side silently.

Symptom: The very first export failed before writing anything:

$ nickel export --format json --output envs/dev/main.tf.json dev.ncl
error: No such file or directory (os error 2)
[exit 1]

Cause: nickel export --output does not create parent directories, and envs/dev did not exist yet.

Fix: Run mkdir -p envs/dev envs/prod first. The example script and the flake check both do.

Symptom: Exporting the base by itself, to see what it produces, fails:

$ nickel export --format json base.ncl
error: missing definition for `environment`
   ┌─ /private/tmp/gcs-bucket-lab/base.ncl:4:3
   │  
 2 │ ╭ {
 3 │ │   # Inputs every environment sets. Used below, never written to the JSON.
 4 │ │   environment | S.Env | not_exported,
   │ │   ^^^^^^^^^^^ required here

Cause: environment is declared with a contract but no value, and the bucket name and the env label both read it. Nickel evaluates the whole record on export, so the missing input stops it at line 4 before any JSON is written.

Fix: Export an overlay, never the base. The base is a template with required inputs, and the error is the base refusing to be used without them.

Holding the floor: force versus a contract on the field

A contract is a runtime check attached to a field with |. std.contract.from_predicate turns a function returning a boolean into a contract, and std.contract.Equal is "a delayed contract enforcing equality to a given constant". What matters here is how contracts combine with merging: when a contract is attached to a field, "whatever this field is merged with, including being dropped in favor of another value, the final value for foo has to respect the contract as well". The contract therefore checks the winner of a merge, not only the base's own value.

The base holds the floor two ways so they can be compared. The first is uniform_bucket_level_access | force = true: force is the highest priority, so an overlay that sets false is erased. broken/weaken-force.ncl tries exactly that, and the export succeeds:

{"name":"acme-logs-dev","uniform_bucket_level_access":true,"public_access_prevention":"enforced","uniform_bucket_acess":null}
[exit 0]

(The last field is part of the jq filter and is null because that key is absent.) The value stayed true, which is safe, but nobody was told.

The second way is public_access_prevention | std.contract.Equal "enforced" = "enforced". An overlay that sets "inherited" collides with the base value at the same priority:

error: non mergeable terms
   = Both values are of type String but they aren't equal.

The full error points at base.ncl line 21 and broken/weaken-pap.ncl line 3, the rule and the offending line. An overlay that escalates with | force = "inherited" wins the priority contest and then meets the contract:

error: contract broken by the value of `public_access_prevention`
   ┌─ /private/tmp/gcs-bucket-lab/base.ncl:21:32
   │
21 │     public_access_prevention | std.contract.Equal "enforced" = "enforced",
   │                                ----------------------------- expected type
   │
   ┌─ /private/tmp/gcs-bucket-lab/broken/weaken-pap-force.ncl:3:76
   │
 3 │ & { resource.google_storage_bucket.logs.public_access_prevention | force = "inherited" }
   │                                                                            ^^^^^^^^^^^ applied to this expression
   │
   ┌─ <unknown> (generated by evaluation):1:1
   │
 1 │ "inherited"
   │ ----------- evaluated to this value

Read the report top to bottom: the first frame is the rule in the base ("expected type"), the second is the forced value in the overlay ("applied to this expression"), and the last is the value that reached the contract. The forced "inherited" won the merge, and then the contract that travels with the field rejected it, because "inherited" is not equal to "enforced". That is why a contract holds where force alone would only mask the attempt.

Contracts also guard values that no rule of the base fixes. The base imports them from schema.ncl:

# Contracts for the bucket fields this team cares about.
{
  BucketName
    | doc "GCS bucket name without dots: 3 to 63 characters of lowercase letters, digits, dashes and underscores, starting and ending with a letter or digit, no goog prefix, no google."
    = std.contract.from_predicate (fun name =>
      std.is_string name
      && std.string.is_match "^[a-z0-9][a-z0-9_-]{1,61}[a-z0-9]$" name
      && !(std.string.is_match "^goog" name)
      && !(std.string.contains "google" name)
    ),

  Env
    | doc "The environments this team deploys."
    = std.contract.from_predicate (fun e => e == "dev" || e == "prod"),
}

BucketName returns true only for a string that matches the pattern and avoids the goog prefix and the word google; Env accepts the two environment names and nothing else. The pattern mirrors Google's naming rules (lowercase letters, digits, dashes and underscores, a letter or digit at each end, 3 to 63 characters), except that it disallows dots, which Google permits, to keep the rule simple. Setting the name to "Acme-Logs-Dev" is rejected:

error: contract broken by the value of `name`
   ┌─ /private/tmp/gcs-bucket-lab/base.ncl:13:12
   │
13 │     name | S.BucketName | default = "acme-logs-%{environment}",
   │            ------------ expected type

The full output goes on to point at the overlay line that set the name. from_predicate allows no custom error message, so the contract's name is the message; name contracts so the next developer can tell what went wrong.

Symptom: An overlay sets uniform_bucket_level_access = false, the export succeeds, and the JSON still says true.

Cause: The base value is force, so the overlay's value is erased without a diagnostic, and its author may believe the change applied.

Fix: If a silent win is not what you want, hold the setting with a contract on the field, as public_access_prevention does, so that every attempt fails loudly with a file and line.

The Nix flake check

A Nix flake is a project with a flake.nix that declares inputs and outputs such as packages, dev shells and checks. nix flake check builds the checks for the current system by default, and --all-systems covers every system. A check is a build that must succeed, so any failing command inside it fails the change.

{
  description = "Nickel-generated Terraform JSON for a GCS bucket, gated by nix flake check";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/7fd4004d0511991fff3a0524856a9be09e28c5f0";

  outputs = { self, nixpkgs }:
    let
      systems = [ "aarch64-darwin" "x86_64-linux" ];
      forAllSystems = f: nixpkgs.lib.genAttrs systems (system:
        f (import nixpkgs {
          inherit system;
          # Terraform is BUSL-licensed, so nixpkgs marks it unfree. Allow only it.
          config.allowUnfreePredicate = pkg: nixpkgs.lib.getName pkg == "terraform";
        }));
      tools = pkgs: [
        pkgs.nickel
        pkgs.jq
        (pkgs.terraform.withPlugins (p: [ p.hashicorp_google ]))
      ];
    in
    {
      devShells = forAllSystems (pkgs: {
        default = pkgs.mkShell { packages = tools pkgs; };
      });

      checks = forAllSystems (pkgs: {
        bucket-config = pkgs.runCommand "bucket-config-check"
          { nativeBuildInputs = tools pkgs; src = self; }
          ''
            export HOME=$TMPDIR TF_IN_AUTOMATION=1
            cp -r $src/*.ncl $src/broken . && chmod -R u+w .

            for env in dev prod; do
              mkdir -p envs/$env
              nickel export --format json --output envs/$env/main.tf.json $env.ncl
              jq -e '.resource.google_storage_bucket.logs
                | .uniform_bucket_level_access == true
                  and .public_access_prevention == "enforced"' envs/$env/main.tf.json
              (cd envs/$env \
                && terraform init -backend=false -input=false -no-color > /dev/null \
                && terraform validate -no-color)
            done

            for bad in weaken-pap weaken-pap-force bad-name conflict; do
              if nickel export broken/$bad.ncl > /dev/null 2>&1; then
                echo "FAIL: broken/$bad.ncl should not export"; exit 1
              fi
              echo "ok: broken/$bad.ncl rejected"
            done

            mkdir -p $out
            cp envs/dev/main.tf.json $out/dev.tf.json
            cp envs/prod/main.tf.json $out/prod.tf.json
          '';
      });
    };
}

The nixpkgs input is pinned to one commit so that every machine gets the same Nickel, jq, Terraform and provider versions. Terraform is licensed BUSL-1.1, which nixpkgs marks unfree, so the flake sets allowUnfreePredicate, a function from a package to a boolean, to admit only terraform while every other unfree package stays blocked.

terraform.withPlugins (p: [ p.hashicorp_google ]) builds a wrapper that sets NIX_TERRAFORM_PLUGIN_DIR so Terraform finds providers in the Nix store. A comment in the nixpkgs source explains why it skips the wrapper when the plugin list is empty: the wrapper "will stop automatic downloading of plugins". With the Google provider in the list the wrapper is built, so terraform init -backend=false (the documented way to prepare a directory for validation without a backend) installs the store copy and needs no registry. The init output shows it, with a lock-file warning:

- Installing hashicorp/google v8.5.0...
- Installed hashicorp/google v8.5.0 (unauthenticated)
The current .terraform.lock.hcl file only includes checksums for
darwin_arm64, so Terraform running on another platform will fail to install

Inside runCommand, the first loop exports each environment, asserts the floor with jq -e, and validates. The second loop fails the build if any of four broken overlays exports. weaken-force and typo are left out because Nickel accepts both: the first is harmless, and terraform validate catches the second. The green run, trimmed:

bucket-config-check> true
bucket-config-check> Success! The configuration is valid.
bucket-config-check> 
bucket-config-check> true
bucket-config-check> Success! The configuration is valid.
bucket-config-check> 
bucket-config-check> ok: broken/weaken-pap.ncl rejected
bucket-config-check> ok: broken/weaken-pap-force.ncl rejected
bucket-config-check> ok: broken/bad-name.ncl rejected
bucket-config-check> ok: broken/conflict.ncl rejected
✅ checks.aarch64-darwin.bucket-config
warning: The check omitted these incompatible systems: x86_64-linux
Use '--all-systems' to check all.
[exit 0]

Each true is a floor assertion, followed by validate's Success!, and the four ok: lines prove the rejections. The warning appears because the flake lists two systems and the check ran only on the current one. With Terraform already built the run took 2.066 seconds, and a rerun with unchanged files printed (previously built) and ran nothing, because Nix reuses a check whose inputs have not changed.

For the red run, dev.ncl gained a misspelled argument, uniform_bucket_acess. Nickel exports it happily; validate does not:

bucket-config-check> Error: Extraneous JSON object property
bucket-config-check>   15:         "uniform_bucket_acess": true,
bucket-config-check> No argument or block type is named "uniform_bucket_acess".
❌ checks.aarch64-darwin.bucket-config
       Reason: builder failed with exit code 1.
[exit 1]

This is why both tools are needed. Nickel checks the values and policies you wrote contracts for. terraform validate checks names and types against the provider's schema, what its docs call "correctness of attribute names and value types". Neither covers the other's gap.

Symptom: The first nix flake check on a new machine spends minutes compiling Terraform.

Cause: Terraform is unfree in nixpkgs and the public binary cache had no build of it, so Terraform 1.16.5 compiled locally (real 2m31.990s on the test machine), while terraform-provider-google 8.5.0, which is not unfree, downloaded from cache.nixos.org.

Fix: Expect the cost once per machine, or share a cache between machines.

Two limits of the test run: the Nix sandbox was off on macOS (nix config show sandbox printed false), and --all-systems was not run, so x86_64-linux is untested.

Putting it together

The loop answers the problem from the start. Edit base.ncl or an overlay and run nix flake check; Nickel errors name the file, line and column, and Terraform names the line in main.tf.json. The floor cannot drift, because it lives in one file and every overlay is checked against it. Once green, export into envs/dev or envs/prod and run terraform plan there with real credentials. Plan and apply were not run here; no Google Cloud account was used.

Terraform's init message says to commit .terraform.lock.hcl so later runs make the same provider selections. The one written in envs/dev:

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/google" {
  version     = "8.5.0"
  constraints = ">= 8.0.0, < 9.0.0"
  hashes = [
    "h1:hTmeC+c2chMEB/R8U/TDhudW3FVOOGKIocKJG+BWC3c=",
  ]
}

It pins 8.5.0 and repeats the generated constraint, normalized to >= 8.0.0, < 9.0.0. It holds exactly one h1: hash, computed locally from the store copy for darwin_arm64, which is what the init warning meant. A teammate on Linux would fail to install the provider from this file, so run terraform providers lock -platform=linux_amd64 (or your platform) before committing it; the lab did not run that command.

Check yourself

Question 1
Why does a clean terraform fmt -check tell you nothing about a generated .tf.json file?
tap to reveal
fmt works on native-syntax files, and in the test it simply reported nothing for the JSON. Formatting generated JSON is the generator's job, and terraform validate is the check on its content.
Question 2
Two overlays both set location without any priority annotation. What happens, and what is the right fix?
tap to reveal
Nickel fails with "non mergeable terms", because both values have priority 0 and are different strings. Decide which layer owns the field and remove one definition, instead of using force.
Question 3
An overlay sets uniform_bucket_level_access = false and another sets public_access_prevention | force = "inherited". How do the outcomes differ?
tap to reveal
The first is silently ignored because the base value is force, and the export exits 0 with true. The second fails with "contract broken by the value of public_access_prevention", because the contract checks the merged result.
Question 4
A misspelled argument exports from Nickel without error. Which step catches it?
tap to reveal
terraform validate rejects it with "Extraneous JSON object property", because it checks names and types against the provider schema. Nickel only checks what the contracts cover.
Question 5
Why can terraform init -backend=false run inside the check without reaching a registry?
tap to reveal
terraform.withPlugins builds a wrapper that sets NIX_TERRAFORM_PLUGIN_DIR to providers in the Nix store, and that wrapper stops automatic plugin downloads; the init output shows the provider installed "(unauthenticated)" from that local copy.

Sources