Terraform S3 Backend and DynamoDB Lock — the lock you did not hold

Two applies, one lock. The second apply is a stranger until the first one lets go.

<!-- hal:authoritative:yaml -->

Two applies, one lock. The second apply is a stranger until the first one lets go.

§I — Frame

Saturday this arc bound a GKE ServiceAccount to a Google Service Account with a member Terraform computed and no author typed. 08-12 replaced an ASG in the right order. 08-09 hung policy on a plan. Today's overlay is AWS on purpose: S3 plus DynamoDB is the pair Brikman builds in Chapter 3, and it rebalances those two clouds.

Two operators, or an operator and a runner, call terraform apply against the same state. Without a lock both refresh, both write, and the second write can drop the first. Brikman names it in Shared Storage for State Files: race conditions, conflicts, data loss, state file corruption. Version control does not lock terraform apply.

The fix is a remote backend that stores the file and a lock table that stores the claim. On AWS the store is an S3 bucket. The claim is a DynamoDB table whose hash key is LockID, spelled and capitalized exactly that way. Terraform acquires the lock, applies, writes state, releases the lock. The other apply waits, or it fails, or (if someone passed -lock=false) it writes anyway.

That last path is today's name. You applied. You did not hold the lock. Or you force-unlocked a lock that belonged to a run still in progress. Coin it: the lock you did not hold.

This is not 07-31. That lesson isolated environments on a GCS prefix and asked when a workspace is a promotion boundary. Today the isolation that matters is a different key in the same bucket. One file for staging and production is the other half of the same accident.

§II — Foundations: four facts about the claim

Fact one. Shared storage is a bucket plus a lock table.

Brikman, Shared Storage for State Files, walks the bootstrap. An aws_s3_bucket holds the objects. Versioning is on so every write is a new version (Figure 3-2). Server-side encryption is on by default. Public access is blocked on all four switches. Then an aws_dynamodb_table whose hash_key is LockID, attribute type S, billing PAY_PER_REQUEST. DynamoDB is there for strongly consistent reads and conditional writes. That is a distributed lock.

The backend block that consumes both lives in a terraform block, not in a resource:

terraform {
  backend "s3" {
    bucket         = "atlas-tfstate-prod"
    key            = "network/prod/terraform.tfstate"
    region         = "us-east-2"
    dynamodb_table = "atlas-tf-locks"
    encrypt        = true
  }
}

bucket is the store. key is the object path, unique per root module. region is where the bucket lives. dynamodb_table is the lock. encrypt = true is a second encryption layer on the bucket default. After init migrates local state, every plan and apply pulls, locks, writes, unlocks. Brikman's apply transcript prints Acquiring state lock on the way in and Releasing state lock on the way out.

Fact two. The backend block does not evaluate variables.

Limitations with Terraform’s Backends is the section that fails Lab 04. The lab's broken main.tf writes key = local.network_state_key inside backend "s3". Terraform will not interpolate local, var, or a resource attribute there. The backend is configured at init, before the graph exists. Brikman's counter-example uses var.bucket and var.dynamodb_table and labels it "This will NOT work."

The chicken-and-egg is the first limitation in that section. Bootstrap with a local backend, apply the bucket and table, add the backend block, run init, copy state up. Destroy is the reverse. After the first account you share one bucket and one table.

Partial configuration cuts the copy-paste. Leave key in the terraform block. Pass the rest at init:

terraform {
  backend "s3" {
    key = "network/prod/terraform.tfstate"
  }
}

terraform init -backend-config=backend.hcl

backend.hcl carries bucket, region, dynamodb_table, encrypt. Lab 31 drills that merge. Reuse the file across modules; still set a unique key so one module does not overwrite another. The key is the isolation primitive.

Fact three. Isolation is a different key, not a bigger file.

State File Isolation opens with a ship. Bulkheads keep a leak in one compartment from flooding the rest. A single state file for every environment is a ship with no bulkheads. A mistake in staging, or a rare state-file bug, takes production with it.

Brikman offers two isolations. Workspaces (07-31 already spent this) are named slots under the same configuration, useful for a quick experiment, poor as a promotion boundary. File layout is the production answer: stage/vpc, prod/vpc, stage/data-stores/mysql, each a root module with its own key. The lock table can be shared. The object path cannot.

One lock serializes writers of one state. Two applies against network/prod/terraform.tfstate and app/prod/terraform.tfstate both succeed. They are different files. If you wanted them to wait on each other, you wanted one file, which is the accident isolation exists to prevent.

**Fact four. terraform_remote_state is a read-only consumer.**

The producer writes outputs into its own state. The consumer declares a data source that points at the producer's backend (bucket, key, region) and reads data.terraform_remote_state.network.outputs.vpc_id. Brikman is blunt: the result is read-only. Lab 04 names producer-versus-consumer as a first-class task. The broken lab invents a locals map of fake network outputs instead of reading state.

The consumer does not acquire the producer's lock to read. If the producer is mid-apply, the consumer sees the last written state. That is the same fact the Cert lesson uses when an HCP run holds a lock your laptop wants.

§III — Worked example: two applies, one LockID

A platform root module owns the store. After the local-backend bootstrap its live backend uses key = "global/s3/terraform.tfstate". A network root module points at the same bucket and table with a different key:

terraform {
  backend "s3" {
    bucket         = "atlas-tfstate-prod"
    key            = "network/prod/terraform.tfstate"
    region         = "us-east-2"
    dynamodb_table = "atlas-tf-locks"
    encrypt        = true
  }
}

resource "aws_vpc" "prod" {
  cidr_block = "10.20.0.0/16"
}

Operator A, in a pipeline runner, starts terraform apply. Terraform writes a lock item to atlas-tf-locks. The hash key value is a LockID Terraform minted for this operation. The apply is slow: VPC, then subnets, then a peering that waits on an accepter.

Operator B, on a laptop, in the same directory, against the same backend, starts terraform apply to add a tag. Terraform tries to write a lock item for the same state. DynamoDB refuses the conditional write. B sees:

Error: Error acquiring the state lock

Error message: ConditionalCheckFailedException: The conditional
request failed
Lock Info:
  ID:        9b2a1c4e-7d11-4f0a-a6e3-0f3c8d21ab77
  Path:      atlas-tfstate-prod/network/prod/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner@gha-prod-4
  Version:   1.9.8
  Created:   2026-08-18 10:12:03 UTC
  Info:

B did not hold that lock. The ID belongs to the runner. The correct move is wait, or terraform apply -lock-timeout=10m as Brikman shows under Shared Storage. The wrong move is -lock=false. The worse move is terraform force-unlock 9b2a1c4e-7d11-4f0a-a6e3-0f3c8d21ab77 while the runner is still applying. Force-unlock deletes the item. The runner still believes it holds the claim. B now writes the same object. Two writers, one file: the lock you did not hold, spent as a command.

When the runner finishes, it releases. B's waiting apply acquires and plans the tag on the VPC that now exists. B's LockID is a new string. Force-unlock of the old ID then says the lock was not found. That error means you were late, not that the table is broken.

§IV — Failure mode: the lock you did not hold

**-lock=false as a convenience.** The flag exists so a broken lock cannot freeze a destroy in a throwaway workspace. In a shared S3 backend it is a second writer. The apply succeeds locally. The state object in S3 is whichever write finished last. Resources exist that state does not name, or state names resources the account no longer has. The next plan looks like drift. It is not drift. It is a lost write.

Force-unlock as a reflex. The error printed a LockID, so it felt like a handle. Force-unlock is a last resort for a lock whose owner is dead: a runner killed mid-apply, a laptop that slept through a release, a process that lost its lease and will not come back. You confirm the owner is gone (no apply in the runner, no terraform process on the box named in Who) and then you unlock that ID. You do not unlock a live HCP run. You do not unlock a colleague who is three minutes into a VPC. The Cert lesson is this paragraph at exam altitude.

Variables in the backend block. Lab 04's key = local.network_state_key looks like good DRY. It is invalid. The operator "fixes" it by pasting a literal and then uses the same literal in staging and production. The backend now evaluates. Isolation is gone. Two environments share one object. The lock serializes them, which feels safe and is the opposite: the lock is now the only thing standing between a staging apply and a production write.

One file for all environments. The operator skipped file layout because "we have locking now." Brikman's bulkhead figure is the answer. A lock protects a file. It does not invent a bulkhead. Staging and production as two keys is the bulkhead. Workspaces under one configuration are a named slot, not a promotion path (07-31 already closed that claim).

A consumer that writes. Someone "shares" network state by pointing two root modules at network/prod/terraform.tfstate and applying both. The second is not a terraform_remote_state consumer. It is a second writer. The lock will serialize the writes. The state will still be whichever graph won. Read with the data source. Write with one root module.

§V — Pairing

Today's Dev lesson wraps terraform apply in subprocess.run. The wrapper must not pass -lock=false. Error acquiring the state lock on stderr is a hard fail, nonzero. try/finally does not release a Terraform lock; the CLI holds it. The coin is the same phrase.

Today's Cert lesson is Pro-depth remote operations. HCP holds the lock for the run. A laptop apply against that backend while the run is in progress is the exam trap. Force-unlock versus wait is the fork. Change sets are an AWS CloudFormation object from yesterday's Python-day Cert; they are not a Terraform lock.

08-15 stays closed (computed GSA/KSA). 08-12 stays closed (lifecycle order). 08-09 stays closed (Sentinel, plan-JSON, secrets in state).

§VI — Drills

Related

• Prior arc: workspaces and the promotion boundary (2026-07-31)
• Prior arc: GKE Workload Identity, computed not declared (2026-08-15)
• Domain hub: Cross-References/domains/01-Earth-DevOps
• Grounding tome: Terraform: Up and Running (Brikman Ch.3)

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura Filed 2026-08-18 · Fajr · sprint track TF day 27 · ninth TF visit · trio #93