A Terraform http state backend on deno serve scoped permissions, lock contention, the stale lock force-unlock cannot clear, and a Rust pre-run audit

A lock belongs to the process that took it. Knowing its ID is not enough to release it through Terraform.

The problem

A small team wants Terraform state to live behind a tiny HTTP service it controls instead of in a cloud bucket. The http backend needs only a REST (representational state transfer) endpoint, so this is possible [1]. Three things then go wrong.

First, a teammate's terraform plan fails with "Error acquiring the state lock" while someone else applies. Second, a CI (continuous integration) runner is killed in the middle of an apply, the lock stays behind, and terraform force-unlock <ID>, the documented repair [2][14], fails with "Unexpected HTTP response code 409". Third, someone pushes a state file from a different project over the shared one, and the service has to be the thing that refuses it.

Each failure is easy to explain once the protocol is visible. The lab behind this lesson ran entirely on one machine against 127.0.0.1, with Terraform's built-in terraform_data resource and no cloud provider or credentials.

The idea, in plain words

Terraform state is the JSON (JavaScript Object Notation) record of what Terraform manages, and a backend decides where it lives [12]. The http backend speaks plain HTTP verbs, plus two extra verbs named LOCK and UNLOCK, so a handler of under a hundred lines can serve it. Deno, a JavaScript and TypeScript runtime, gives a program no file, network or environment access unless granted, so the service can be limited to the one directory that holds state [6]. Terraform's lock is a JSON document with an ID. The point to carry out of this lesson: a lock belongs to the process that took it, and knowing its ID is not enough to release it through Terraform. A pre-run audit written in Rust reads the state and the lock before any job runs and answers with an exit code. Nix, a package manager that builds from pinned inputs, compiles the audit, wraps the service with its permissions fixed, and checks the lock contract on every change.

The backend as a Worker-style handler, and what Deno lets it touch

A backend is the part of Terraform that decides where state is stored. The http backend fetches state with GET, updates it with POST and purges it with DELETE. With locking on, it sends LOCK and UNLOCK requests with the lock info in the body. The endpoint answers 423 Locked or 409 Conflict, with the holding lock info, when the lock is taken, and 200 for success. "Any other status will be considered an error," and the holder's ID is added as a query parameter to state updates [1]. The lab's configuration names all three addresses:

terraform {
  backend "http" {
    address        = "http://127.0.0.1:8787/state/demo"
    lock_address   = "http://127.0.0.1:8787/lock/demo"
    unlock_address = "http://127.0.0.1:8787/lock/demo"
  }
}

The address serves state, while the two lock addresses serve LOCK and UNLOCK. Both lock options default to disabled, so naming them is what turns locking on [1]. Those two paths are the whole contract the service must honor.

deno serve runs a file as an HTTP server. The file exports a default object with a fetch handler, and Deno owns the listener [5]. That is the Worker-style shape. If the handler throws, the error is isolated to that request, and --host picks the interface instead of the default 0.0.0.0 [5]. Deno's permission model grants nothing by default, and flags can be scoped: --allow-read=./data limits reads to one directory, while a bare flag or -A grants far more [6]. A refused operation throws NotCapable, and Deno prompts only when stdout is a terminal; a service started by a script or a supervisor has no terminal, so it fails instead of asking, which is what every run below shows. Listening needs no --allow-net under deno serve, and the static module graph loads without --allow-read [5][6].

The core of edge/state.ts, trimmed:

const dir = Deno.env.get("TFSTATE_DIR") ?? "./data";
// ...
export default {
  async fetch(req: Request): Promise<Response> {
    const url = new URL(req.url);
    const [, kind, name] = url.pathname.split("/");
    // ...
    try {
      const held = await readJson(lockPath);
      if (kind === "lock") {
        if (req.method === "LOCK") {
          const info = await req.text();
          try {
            // createNew fails if the file exists, so two LOCKs cannot both win.
            await Deno.writeTextFile(lockPath, info, { createNew: true });
            return ok();
          } catch (e) {
            if (!(e instanceof Deno.errors.AlreadyExists)) throw e;
            return Response.json(await readJson(lockPath), { status: 423 });
          }
        }
        if (req.method === "UNLOCK") {
          if (!held) return ok();
          const body = await req.text();
          const id = body ? JSON.parse(body).ID : undefined;
          if (id !== held.ID) return conflict(held);
          await Deno.remove(lockPath);
          return ok();
        }
      }
      // state branch:
      if (req.method === "POST") {
        if (held && url.searchParams.get("ID") !== held.ID) return conflict(held);
        // ... lineage and serial checks, shown further down
      }
    } catch (e) {
      console.error(`${req.method} ${url.pathname}: ${e}`);
      return new Response("internal error\n", { status: 500 });
    }
  },
} satisfies Deno.ServeDefaultExport;

The routing line splits the path into a kind and a name. LOCK writes the lock file with createNew: true, so the filesystem decides the winner when two requests race, and the loser gets 423 with the holder's document. UNLOCK removes the file only when the ID in the request body equals the holder's, and otherwise answers 409. A state POST without the holder's ID is refused, so a writer that never took the lock cannot write. The final catch logs the error and returns 500, because anything unexpected, including a permission refusal, must reach the server log rather than vanish.

The permission experiment ran the same handler three ways. With no flags (step 1.1) the server died at startup, and curl got no answer:

[HTTP 000]
[curl exit 7]
error: Uncaught (in promise) NotCapable: Requires env access to "TFSTATE_DIR", run again with the --allow-env flag

The environment read sits at module top level, outside the handler. An error inside the handler would stay with one request, but this one happened while the module loaded, so the whole server exited before it ever listened. With env and read granted but not write (1.2), the server stayed up and LOCK returned 500 "internal error". Terraform's apply exited 1:

Error: Error acquiring the state lock

Error message: Failed to lock: LOCK http://127.0.0.1:8787/lock/demo giving up
after 3 attempt(s)

The server log named the exact path and flag each time:

LOCK /lock/demo: NotCapable: Requires write access to "~/lab/data/demo.lock.json", run again with the --allow-write flag

That line appeared four times, one from curl and three from Terraform, because retry_max defaults to 2, which means three attempts [1]. With --allow-env=TFSTATE_DIR, a scoped read and a scoped write (1.3), both requests returned 200.

Symptom: terraform apply says "Error acquiring the state lock ... giving up after 3 attempt(s)" while the state service is up and answering.

Cause: The service cannot write its lock file. Terraform treats the 500 as a failure to lock, not as "locked", because only 423 or 409 mean the lock is held [1].

Fix: Read the service log, since the NotCapable line names the path and the missing flag. Then grant --allow-write scoped to the state directory only, never a bare --allow-write or -A, which turns the sandbox off [6].

The lock lifecycle: contention, waiting, and a stale lock

State locking means Terraform takes a lock before any operation that could write state, automatically. If locking fails, Terraform does not continue, and -lock=false exists but "we do not recommend it" [2]. The reason is that without locking, two people running Terraform at once can cause race conditions, conflicts and corrupted state [13]. The lab did not test -lock=false.

The first apply (step 2.1) wrote serial 1 (serial is a counter that rises with every write) under a fresh lineage (a unique ID set when the state is first created), 345c0b5a-486f-746f-82c4-9ee5fe6e2570. For step 2.2 a second apply held the lock for eight seconds (its local-exec provisioner runs sleep 8) while the lab read the lock and ran a concurrent plan. The lock document served by GET /lock/demo was:

{
  "ID": "7fe27895-3a99-ea65-e407-c1e3422e1392",
  "Operation": "OperationTypeApply",
  "Info": "",
  "Who": "you@laptop",
  "Version": "1.14.3",
  "Created": "2026-10-23T17:41:55.56535Z",
  "Path": ""
}

The concurrent plan failed with "HTTP remote state already locked:" followed by ID=7fe27895-3a99-ea65-e407-c1e3422e1392. Below that came a "Lock Info:" block with Operation OperationTypePlan and a different ID, db80694d-b199-aac2-652e-09b82cadf55d. That block describes the requester, not the holder. Terraform's source builds the error as &statemgr.LockError{ Info: info, Err: fmt.Errorf("HTTP remote state already locked: ID=%s", existing.ID) }, where info is the plan's own lock info and existing.ID is the holder's [4]. So the holder's ID is the one after ID= on the error line.

With -lock-timeout, Terraform waits for a lock instead of failing [13]. Step 2.3 reran the plan with -lock-timeout=20s. It printed "Acquiring state lock. This may take a few moments...", waited 7s for the holding apply to finish, then printed "No changes. Your infrastructure matches the configuration."

Step 2.4 started an apply with hold=30 and killed it with SIGKILL (the signal kill -9 sends), which gives Terraform no chance to send UNLOCK. The lock file stayed, and the next plan failed with the same error, now naming the dead holder: ID=81098a05-3617-3e8a-f8c1-5452b9a34dd3. This is the failure the book describes when a CI server crashes mid-apply [14].

Symptom: Every plan fails with "already locked", and the Lock Info block shows an OperationTypePlan and an ID that changes on each run.

Cause: That block is your own request, because the source builds the LockError with Info: info, the requester's info [4]. The real holder is the ID= on the first line.

Fix: Copy the ID from the ID= line, then check the holder's Who and Created fields to confirm it is dead. Force-unlock "should only be used to unlock your own lock in the situation where automatic unlocking failed," because unlocking someone else's lock can cause multiple writers [2].

Why force-unlock cannot clear it here, and the lineage and serial guards

terraform force-unlock LOCK_ID removes the lock on the state, and "the behavior of this lock is dependent on the backend being used" [3]. The ID "acts as a nonce, ensuring that locks and unlocks target the correct lock" [2]. A nonce here is a value that proves the caller means this exact lock.

The http backend's source shows why that fails across processes. Unlock is declared as func (c *httpClient) Unlock(id string) error and sends resp, err := c.httpRequest(c.UnlockMethod, c.UnlockURL, &c.jsonLockInfo, "unlock") [4]. The id argument is not used in the request. The body is c.jsonLockInfo, which only this process's own successful Lock sets, through c.jsonLockInfo = jsonLockInfo in the http.StatusOK case [4]. A fresh terraform force-unlock process never locked anything, so it sends an empty body. The edge's UNLOCK branch compares the body's ID with the holder's, so an empty body does not match. Step 2.5 shows the result:

Failed to unlock state: Unexpected HTTP response code 409
[exit 1]
--- edge still holds: 81098a05-3617-3e8a-f8c1-5452b9a34dd3

The lock survived, so the way out is a route on the server. The lab's DELETE /lock/demo?ID=<holder ID> is this lab's own design, not part of the http backend protocol, which names only LOCK and UNLOCK for locks [1]. Step 2.6 shows DELETE ?ID=wrong [HTTP 409], then DELETE ?ID=<holder ID> [HTTP 200]. The next apply succeeded and the stored state read {"serial":3,"lineage":"345c0b5a-486f-746f-82c4-9ee5fe6e2570"}.

Lineage and serial also guard writes. Step 2.7 pulled the state, replaced its lineage with all zeros, raised the serial by one, and pushed it. Terraform itself refused: "cannot import state with lineage ... over unrelated state with lineage ...". With push -force that check was skipped, but the edge answered 409, and Terraform printed "Failed to persist state: HTTP error: 409". Its source returns HTTP error: %d for any status except 200, 201 and 204 [4]. The stored state stayed at serial 3 with the original lineage. The older-serial refusal was exercised only by the flake check below, not through Terraform.

Symptom: terraform force-unlock <ID> fails with "Unexpected HTTP response code 409", or whatever your server answers to an empty UNLOCK.

Cause: Through the http backend, force-unlock replays lock info held in memory, and a new process has none [4].

Fix: After confirming the holder is dead, break the lock on the server by ID. Give the server an authenticated break route like the lab's DELETE, and never use -lock=false as a workaround.

The pre-run audit as a cargo script, a Nix package, a pinned wrapper, and a flake check

A cargo script is a single .rs file that Cargo runs directly with cargo +nightly -Zscript file.rs. Its package metadata sits in a TOML fence at the top, labeled cargo [7]. It works on nightly only and is unstable. The audit uses ureq, an HTTP client library, and serde_json to parse JSON. Its header and core:

#!/usr/bin/env -S cargo +nightly -Zscript
---cargo
[package]
edition = "2024"

[dependencies]
ureq = { version = "3", default-features = false }
serde_json = "1"
---
// ...
let agent: ureq::Agent = ureq::Agent::config_builder()
    .http_status_as_error(false)
    .timeout_global(Some(Duration::from_secs(5)))
    .build()
    .into();
// ...
match status {
    200 => { /* print STATE; if EXPECT_LINEAGE differs: println!("DRIFT ..."); code = 2; */ }
    404 => { println!("MISSING no state stored under {name}"); code = 2; }
    other => { eprintln!("ERROR GET state returned HTTP {other}"); exit(3); }
}
// lock: 404 prints UNLOCKED; 200 prints LOCKED and sets code = 1 only if code == 0
exit(code);

By default ureq turns 4xx and 5xx responses into errors [8]. Setting http_status_as_error(false) lets the program read a 404 as "no state yet" instead of a failure, so the match can tell the cases apart. timeout_global bounds the whole call from DNS lookup to the end of the body [8], so a hung service cannot stall a pipeline. The lock match sets code 1 only if the code is still 0, so a wrong or missing state (2) wins over a held lock (1).

The lab's transcripts match the table:

## 3.1 state present, unlocked
STATE serial=3 lineage=345c0b5a-486f-746f-82c4-9ee5fe6e2570 terraform=1.14.3 resources=1
UNLOCKED
[audit exit 0]
## 3.2 EXPECT_LINEAGE set to the real lineage, then to another
STATE serial=3 lineage=345c0b5a-486f-746f-82c4-9ee5fe6e2570 terraform=1.14.3 resources=1
DRIFT lineage 345c0b5a-486f-746f-82c4-9ee5fe6e2570 is not the expected 11111111-2222-3333-4444-555555555555
UNLOCKED
[audit exit 2]
## 3.3 while an apply holds the lock
STATE serial=3 lineage=345c0b5a-486f-746f-82c4-9ee5fe6e2570 terraform=1.14.3 resources=1
LOCKED id=5d36fefc-2efc-2f7a-8f73-855dd62a6bcd op=OperationTypeApply who=you@laptop created=2026-10-23T17:42:11.93206Z
[audit exit 1]
## 3.4 a state name that was never written
MISSING no state stored under staging
UNLOCKED
[audit exit 2]
## 3.5 the edge is down
ERROR http://127.0.0.1:8787/state/demo: io: Connection refused (os error 61)
[audit exit 3]
usage: tfstate-audit <base-url> <state-name>
[audit exit 3]

Lines are trimmed (the run with the matching lineage exited 0 and is omitted). A CI job can branch on the code without parsing text. The DRIFT and MISSING runs also printed UNLOCKED, which shows both checks run before the exit. The order of the two checks matters when both fire: the code sets 1 for a lock only if nothing has already set 2, so a wrong state held by someone else still exits 2. Waiting out a lock fixes nothing when the address itself is wrong.

Packaging pins everything. The flake, verbatim:

{
  description = "state-edge (Terraform http backend on deno serve) and tfstate-audit (Rust pre-run check)";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";

  outputs = { self, nixpkgs }:
    let
      systems = [ "aarch64-darwin" "x86_64-linux" "aarch64-linux" ];
      forAll = f: nixpkgs.lib.genAttrs systems (system: f nixpkgs.legacyPackages.${system});
    in
    {
      packages = forAll (pkgs: rec {
        tfstate-audit = pkgs.rustPlatform.buildRustPackage {
          pname = "tfstate-audit";
          version = "0.1.0";
          src = pkgs.lib.fileset.toSource {
            root = ./audit;
            fileset = pkgs.lib.fileset.unions [ ./audit/Cargo.toml ./audit/Cargo.lock ./audit/src ];
          };
          cargoLock.lockFile = ./audit/Cargo.lock;
        };

        # The edge, with its permissions fixed in the wrapper instead of in someone's shell history.
        state-edge = pkgs.writeShellApplication {
          name = "state-edge";
          runtimeInputs = [ pkgs.deno ];
          text = ''
            dir=''${TFSTATE_DIR:?set TFSTATE_DIR to the state directory}
            mkdir -p "$dir"
            export TFSTATE_DIR="$dir"
            exec deno serve --host 127.0.0.1 --port "''${PORT:-8787}" \
              --allow-env=TFSTATE_DIR --allow-read="$dir" --allow-write="$dir" \
              ${./edge/state.ts}
          '';
        };

        default = state-edge;
      });

      # Prove the lock contract against the real edge: start it, drive it with curl, read it with the audit tool.
      checks = forAll (pkgs:
        let p = self.packages.${pkgs.stdenv.hostPlatform.system}; in {
          audit = p.tfstate-audit;
          lock-contract = pkgs.runCommand "lock-contract"
            { nativeBuildInputs = [ p.state-edge p.tfstate-audit pkgs.curl ]; }
            ''
              export HOME=$TMPDIR DENO_DIR=$TMPDIR/deno TFSTATE_DIR=$TMPDIR/state PORT=18787
              state-edge > $TMPDIR/edge.log 2>&1 &
              edge=$!
              for _ in $(seq 50); do curl -s -o /dev/null http://127.0.0.1:$PORT/lock/demo && break; sleep 0.2; done
              u=http://127.0.0.1:$PORT
              code() { curl -s -o /dev/null -w '%{http_code}' "$@"; }
              want() { echo "[$1: got $2, want $3]"; test "$2" = "$3"; }
              lock='{"ID":"aaaa","Operation":"OperationTypeApply","Who":"check@sandbox"}'
              state() { echo "{\"version\":4,\"serial\":$1,\"lineage\":\"$2\",\"terraform_version\":\"check\",\"resources\":[]}"; }
              want "LOCK free"          "$(code -X LOCK -d "$lock" $u/lock/demo)" 200
              want "LOCK taken"         "$(code -X LOCK -d "$lock" $u/lock/demo)" 423
              want "POST without ID"    "$(code -X POST -d "$(state 1 L1)" $u/state/demo)" 409
              want "POST with ID"       "$(code -X POST -d "$(state 1 L1)" "$u/state/demo?ID=aaaa")" 200
              set +e; tfstate-audit $u demo; a=$?; set -e; want "audit while locked" $a 1
              want "UNLOCK empty body"  "$(code -X UNLOCK $u/lock/demo)" 409
              want "UNLOCK with info"   "$(code -X UNLOCK -d "$lock" $u/lock/demo)" 200
              set +e; tfstate-audit $u demo; a=$?; set -e; want "audit unlocked" $a 0
              want "POST other lineage" "$(code -X POST -d "$(state 2 L2)" $u/state/demo)" 409
              want "POST older serial"  "$(code -X POST -d "$(state 0 L1)" $u/state/demo)" 409
              kill $edge
              touch $out
            '';
        });
    };
}

buildRustPackage with cargoLock.lockFile retrieves dependencies as fixed-output derivations from the lockfile [9], so the audit builds from exactly the crate versions recorded there (ureq 3.5.0 and serde_json 1.0.154 in the lab). Its check phase runs cargo test by default [9]. writeShellApplication writes the wrapper to the Nix store, checks it with shellcheck and bash -n, sets errexit, nounset and pipefail, and puts runtimeInputs on PATH [10]. The permission flags therefore live in one reviewed place instead of in shell history. ${TFSTATE_DIR:?...} makes the wrapper fail at once if the variable is unset, and --host 127.0.0.1 keeps the listener on loopback. The lock-contract check starts that packaged edge inside the build and drives it with curl. The recorded log shows every step:

[LOCK free: got 200, want 200]
[LOCK taken: got 423, want 423]
[POST without ID: got 409, want 409]
[POST with ID: got 200, want 200]
LOCKED id=aaaa op=OperationTypeApply who=check@sandbox created=?
[audit while locked: got 1, want 1]
[UNLOCK empty body: got 409, want 409]
[UNLOCK with info: got 200, want 200]
UNLOCKED
[audit unlocked: got 0, want 0]
[POST other lineage: got 409, want 409]
[POST older serial: got 409, want 409]

The want function exits nonzero on any mismatch, so a failed line fails the build. The "UNLOCK empty body" line is the force-unlock case, encoded as a test so a future edit cannot silently change it. The run printed "✅ checks.aarch64-darwin.lock-contract (previously built)" and "[nix exit 0]", plus "warning: The check omitted these incompatible systems: aarch64-linux, x86_64-linux". nix flake check checks whether the flake evaluates and runs its tests, and --all-systems would cover the other platforms [11]. Because the check was already built, the log above came from nix log. The lab machine runs Nix with sandbox = false, so the check was not tried with build sandboxing on.

Symptom: The audit prints MISSING no state stored under staging and exits 2, although the job "has run before".

Cause: The workspace name in the URL does not match where state was written. The service answered 404, and the audit reports that as missing instead of an error because of http_status_as_error(false).

Fix: Treat exit 2 as a hard stop and check the address before running init. Terraform's own client treats a 404 on GET as "no state yet" rather than an error [4], so without the audit a mistyped name looks like a fresh, empty state.

Putting it together

The job runs in this order. Start the edge only through the state-edge wrapper, so the permissions stay pinned to one directory and the host stays on loopback. Before a plan or apply, run tfstate-audit <url> <name> with EXPECT_LINEAGE set. On exit 0, run Terraform with a -lock-timeout long enough for a normal apply, so a short overlap waits instead of failing. On exit 1, wait, or break the lock on the server by ID once the holder is provably dead. On exit 2, stop, because the state is wrong or missing. On exit 3, fix the connection.

The lab verified a Deno 2.9.7 edge, the Terraform 1.14.3 http backend, the cargo script, and the Nix build plus flake check on aarch64-darwin. It did not verify TLS (transport layer security) or authentication. The backend offers username and password for basic auth and mTLS (mutual TLS) certificate options, and recommends supplying credentials through environment variables [1]. The lab used none of them and served plain HTTP on 127.0.0.1 only. It also did not test a hosted Workers runtime such as Cloudflare, more than two concurrent processes, Linux systems, or Terraform's state file format beyond the fields shown. The DELETE break route had no authentication, so a real deployment must add it.

Check yourself

Sources

1. HashiCorp, "Backend Type: http", Terraform documentation. https://developer.hashicorp.com/terraform/language/backend/http
2. HashiCorp, "State: Locking", Terraform documentation. https://developer.hashicorp.com/terraform/language/state/locking
3. HashiCorp, "terraform force-unlock command", Terraform documentation. https://developer.hashicorp.com/terraform/cli/commands/force-unlock
4. HashiCorp, Terraform source, repository hashicorp/terraform, tag v1.14.3, file internal/backend/remote-state/http/client.go. https://github.com/hashicorp/terraform/blob/v1.14.3/internal/backend/remote-state/http/client.go
5. Deno, "deno serve", Deno documentation. https://docs.deno.com/runtime/reference/cli/serve/
6. Deno, "Security and permissions", Deno documentation. https://docs.deno.com/runtime/fundamentals/security/
7. The Cargo Book, "Unstable features", section "script". https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#script
8. docs.rs, ureq 3.5.0, ConfigBuilder. https://docs.rs/ureq/3.5.0/ureq/config/struct.ConfigBuilder.html
9. Nixpkgs manual, "Compiling Rust applications with Cargo" (buildRustPackage). https://nixos.org/manual/nixpkgs/stable/#compiling-rust-applications-with-cargo
10. Nixpkgs manual, "writeShellApplication". https://nixos.org/manual/nixpkgs/stable/#trivial-builder-writeShellApplication
11. Nix manual, "nix flake check". https://nix.dev/manual/nix/latest/command-ref/new-cli/nix3-flake-check.html
12. Yevgeniy Brikman, Terraform: Up and Running, 3rd ed. (2022), chapter 3 "How to Manage Terraform State", section "What Is Terraform State?" (PDF p. 147).
13. Yevgeniy Brikman, Terraform: Up and Running, 3rd ed. (2022), chapter 3, sections "Shared Storage for State Files" and "Locking state files" (PDF pp. 148-150).
14. Yevgeniy Brikman, Terraform: Up and Running, 3rd ed. (2022), chapter 10 "How to Use Terraform as a Team", section "Deployment strategies", subsection "Errors releasing locks" (PDF p. 597).