Python Around Terraform — the lock the wrapper did not hold

The wrapper does not own the lock. The child does. Your finally is not a release.

<!-- hal:authoritative:yaml -->

*The wrapper does not own the lock. The child does. Your finally is not a release.*

§I — Frame

Today's Ops lesson puts state in an S3 bucket and the claim in a DynamoDB table whose hash key is LockID. Two applies, one lock. -lock=false is the footgun. Force-unlock is a last resort that needs the ID printed on stderr.

This slot is Python-around-TF. tf_day_dev_counter read 7, 7 mod 3 is 1. 07-28 wrote the HCP API with terrasnek. 08-09 read terraform show -json and walked resource_changes. Neither fire wrapped terraform apply as a child process and asked what the wrapper is allowed to do when the child cannot acquire the lock.

Gift already taught subprocess.run and check (Creating Command-Line Tools, Spawn Processes with the subprocess Module; 08-14 spent those pages as primary, on SIGTERM and a child that never saw the log context). Today those pages are referenced. The child is terraform. The question is the lock.

A wrapper that adds -lock=false so the pipeline is "never stuck" is applying without the claim. A wrapper that catches the lock error, prints "busy, continuing", and returns 0 has lied to the runner. A wrapper that try/finally closes a socket and believes it released the Terraform lock has confused two clocks. The CLI holds the lock. Python does not.

Coin the same name the Ops slot coined: the lock you did not hold.

This is not a plan-schema lesson. terraform show -json and state pull stay out of scope. If the wrapper starts parsing resource_changes, it has become 08-09 again.

§II — Foundations: four facts about the child

Fact one. The argv is the contract.

subprocess.run is the call. Gift's pages show run and check as the way a Python tool starts another program and notices whether it failed. The wrapper around Terraform looks like this and no cleverer:

import subprocess
import sys

def apply(workdir: str, extra: list[str] | None = None) -> subprocess.CompletedProcess:
    argv = ["terraform", "apply", "-auto-approve", "-input=false", "-no-color"]
    if extra:
        argv.extend(extra)
    if "-lock=false" in argv:
        raise ValueError("refusing -lock=false against a shared backend")
    return subprocess.run(
        argv,
        cwd=workdir,
        text=True,
        capture_output=True,
        check=False,
    )

-auto-approve is a pipeline choice, not a lock choice. -input=false keeps the child from blocking on a prompt the runner will never type. -no-color keeps stderr parseable. check=False because the wrapper, not CalledProcessError, decides how a lock error is reported. The refusal of -lock=false is the point of the function. A caller that wants to skip the lock has to do it somewhere this function can see, and then this function says no.

You can pass -lock-timeout=10m. That is Brikman's wait, expressed as argv. It is not a skip. The child still tries to acquire. It waits up to ten minutes. If the other apply is still running, the child still fails, and the wrapper still sees a nonzero returncode.

Fact two. The CLI holds the lock. Python does not.

Brikman, Shared Storage for State Files: Terraform acquires the lock, applies, writes state, releases the lock. Those four verbs are inside the terraform process. The DynamoDB item is written by the CLI using the backend configuration that init stored. Your Python process is a parent. It is not a party to the conditional write.

So this is not a lock:

proc = None
try:
    proc = apply("/srv/network/prod")
    return proc.returncode
finally:
    if proc is not None and proc.returncode != 0:
        print("child failed; lock should be clear", file=sys.stderr)

The finally runs when Python leaves the block. It does not talk to DynamoDB. If the child is still running (you should have waited on run, which does), the lock is the child's. If the child died mid-apply after acquiring, the lock item can remain. Your finally did not release it. Your finally printed a sentence. An operator who force-unlocks because the wrapper said "lock should be clear" is unlocking from a process that never held the claim.

08-14 taught a parent that dropped the log context on the way to a child. Today the parent drops nothing on the child and still does not own the child's lock. Same hop. Different object.

**Fact three. Error acquiring the state lock is a hard fail.**

When the child cannot write the DynamoDB item, it prints a block on stderr and exits nonzero. The first line the operator knows is:

Error: Error acquiring the state lock

Later lines carry Lock Info, including ID, Path, Operation, Who, Version, Created. The wrapper's job is to keep that nonzero. check=True would raise CalledProcessError and is legal. An explicit check is clearer in a runner that already treats return codes as the interface:

LOCK_ERR = "Error acquiring the state lock"

class LockHeld(RuntimeError):
    def __init__(self, lock_id: str | None, stderr: str):
        self.lock_id = lock_id
        super().__init__(stderr)

def lock_id_from(stderr: str) -> str | None:
    for line in stderr.splitlines():
        stripped = line.strip()
        if stripped.startswith("ID:"):
            return stripped.split(":", 1)[1].strip() or None
    return None

def apply_or_raise(workdir: str) -> None:
    proc = apply(workdir)
    if proc.returncode == 0:
        return
    if LOCK_ERR in (proc.stderr or ""):
        raise LockHeld(lock_id_from(proc.stderr or ""), proc.stderr or "")
    raise RuntimeError(proc.stderr or f"terraform exited {proc.returncode}")

LockHeld is how the runner distinguishes "someone else is applying" from "the plan was invalid." Both are nonzero. They are not the same page in the runbook. Parsing ID: is optional and for the operator. The wrapper must not call terraform force-unlock with that ID. Printing it is allowed. Unlocking a lock you did not hold is the Cert trap, reached from Python.

Swallowing stderr is the other lie. capture_output=True without writing proc.stderr on failure hides the LockID from the person who has to decide wait-versus-unlock. Capture so you can parse. Print so a human can read.

Fact four. This is the lock conversation, not the plan schema.

08-09 already taught terraform show -json and the resource_changes array. A wrapper that runs apply, then show -json, then walks actions, has left today's topic. state pull is the same gravity well: it is a read of the file the lock protects, and it becomes a secrets-in-state or plan-JSON lesson the moment you parse it.

The only Terraform subcommands this lesson is willing to spawn are apply (and, if you must preview without writing, plan). plan also acquires the lock on a remote backend unless you pass -lock=false, which this wrapper still refuses. A plan that skipped the lock can recommend a change the in-flight apply already made. The wrapper would then apply a stale plan. That is two writers by another name.

Bootcamp has no Python wrapper lab. The tfpro set is HCL correction labs. Empty-by-scope. No gap owed. The code above is taught from Brikman's lock cluster plus Gift's run/check posture.

§III — Worked example: a runner that refuses the skip

A small network pipeline has one job: apply /srv/network/prod after a human has approved the plan artifact in a prior step. The apply job should not invent a second plan. It should not skip the lock because "the plan job already locked." The plan job released. The apply job acquires.

import json
import os
import sys

def main() -> int:
    workdir = os.environ["TF_WORKDIR"]
    extra = []
    timeout = os.environ.get("TF_LOCK_TIMEOUT")
    if timeout:
        extra.append(f"-lock-timeout={timeout}")
    try:
        apply_or_raise(workdir)
    except LockHeld as exc:
        payload = {
            "event": "tf_lock_held",
            "lock_id": exc.lock_id,
            "workdir": workdir,
        }
        print(json.dumps(payload), file=sys.stderr)
        print(exc, file=sys.stderr)
        return 2
    except ValueError as exc:
        print(str(exc), file=sys.stderr)
        return 3
    except RuntimeError as exc:
        print(str(exc), file=sys.stderr)
        return 1
    return 0

if __name__ == "__main__":
    sys.exit(main())

Return code 2 is the lock. Return code 3 is the wrapper refusing -lock=false (if a future caller stuffs it into extra; today extra only grows a timeout). Return code 1 is every other Terraform failure. Return code 0 is an apply that held the lock, wrote state, and released.

A colleague will ask for an environment variable TF_SKIP_LOCK=1 that appends -lock=false. Do not add it. If a destroy in a throwaway local directory needs that flag, that destroy is not this runner. This runner's backend is the S3 bucket and the DynamoDB table from Ops. The variable would be a skip of the claim the table exists to store.

The JSON line on stderr is for the same heartbeat habit 08-02 taught: a machine-readable event, then the human text. It is not a release. It is a report that the child did not acquire.

Timeouts need the same honesty. subprocess.run(..., timeout=600) raises TimeoutExpired if the child is still applying at ten minutes. That exception is not a lock release. The child may still hold LockID. Do not translate the exception into force-unlock. Kill the child if the runner must stop, then inspect Who and the runner's job id, then wait for the process to be gone, then decide. A timeout that unlocks is Python setting a clock on a claim it never wrote.

§IV — Failure mode: the lock the wrapper pretended to hold

The helpful skip. The first lock error pages the on-call. Someone adds -lock=false to "unblock Friday." Friday's apply writes. The original apply, still running in a cancelled-but-not-dead runner, writes last. Monday's plan wants to destroy a subnet that Monday's account still has, or create a VPC that already exists. The wrapper's commit message says "make apply idempotent." It made apply unlocked.

**The finally that unlocks.** A well-meaning change calls terraform force-unlock in finally whenever returncode != 0. Most nonzero applies never acquired (validate error, provider error before lock in some paths) or already released (apply failed after the write). Some nonzero applies died after acquire. The finally cannot tell those apart from returncode alone. If it unlocks on every failure, it unlocks a live apply the moment a wrapper timeout fires. The timeout is Python's. The lock is Terraform's. Killing the parent does not release the child, and unlocking from the parent while the child still runs is the lock you did not hold.

The swallowed lock. capture_output=True, check=False, return 0 if the code path is "warnings only." A lock error is not a warning. Tests that assert returncode in (0, 2) are honest. Tests that accept 0 on a fixture whose stderr contains Error acquiring the state lock are documenting the lie.

The plan-JSON sneak. The wrapper grows a if os.environ.get("POLICY"): show_json_and_check() branch. That branch is 08-09. It will eat this lesson's word budget and re-teach resource_changes. Put policy in 08-09's tool. Keep this tool boring.

The terrasnek detour. 07-28 already starts runs through the HCP API. A wrapper that shells terraform apply against a local working directory whose backend is HCP, then also calls terrasnek to force-cancel the same run, is two clients on one lock. Pick one client. Today's client is the CLI, and the CLI's lock error is the event.

§V — Pairing

Ops is the table and the key. The wrapper is how a Python runner talks to that table without becoming a second locker. Cert is the same error when the holder is an HCP run rather than a colleague's laptop. Force-unlock from this wrapper against an HCP-held lock is the exam stem with a Python accent.

08-14 remains the SIGTERM lesson. Today the child saw the argv. The parent still does not hold the child's lock. 08-09 remains the plan-JSON lesson. 07-28 remains the HCP API lesson; terrasnek is a different client and is not a substitute for refusing -lock=false on the CLI.

§VI — Drills

Related

• Prior arc: plan-JSON checker (2026-08-09)
• Prior arc: subprocess and SIGTERM (2026-08-14)
• Domain hub: Cross-References/domains/Polyglot-Dev
• Grounding tome: Brikman Ch.3 Shared Storage (cluster the wrapper talks to); Gift subprocess referenced

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura Filed 2026-08-18 · Fajr · sprint track TF day 27 · ninth TF visit · trio #93