Hedronite · Ops Lesson · 01-Earth-DevOps / Terraform · Sat 2026-09-05

Terraform remote_state across AWS stacks — network producer, app consumer

A network root module writes VPC outputs into one S3 state object. An app root module reads those outputs.

Lesson Class: Ops (DevOps + Terraform + AWS S3 remote_state)
Cloud Referent: AWS S3 backend keys network/prod and app/prod via data.terraform_remote_state
Paired Dev: Python plan JSON resource_changes census
Paired Cert: Workspaces + terraform.workspace + prod guardrails
Paired Go: AWS SDK Go v2 ListObjects on backend bucket
Grounding: Brikman Ch.3 remote_state · Lab 12 · Lab 17
Producer
Own state object. Publish outputs as the contract.
Consumer
data.terraform_remote_state reads outputs only.
Backend
No expressions in backend blocks. Static keys or partial init.
Publish outputs. Consume with remote_state. Refresh after the producer writes.

<!-- hal:authoritative:yaml -->

A network root module writes VPC outputs into one S3 state object. An app root module reads those outputs. The read is a data source, not a copy-paste of IDs.

§I — Frame

Wednesday named a versionless Key Vault URI that leaves plan empty. Saturday this arc named a DynamoDB LockID that serializes writers of one state file. Neither shirt is today's job.

Today the track is Terraform on AWS. The last TF Ops visit was Azure App Service plus Key Vault. The visit before that sat on GCP. AWS returns as the concrete cloud so the S3 backend keys you already know from the lock lesson can host a different claim: cross-stack output consumption.

Two root modules share an account. network/ owns the VPC, public and private subnets, and a baseline security group. app/ owns the ASG, launch template, and ALB listener rules. The app must place instances into the private subnet and attach the baseline group. Copying vpc-0abc into a variable works until someone recreates the network stack. Then every consumer carries a stale string and plan still looks clean.

Brikman names the instrument in Chapter 3: The terraform_remote_state Data Source. The database writes its state to an S3 bucket. The web server cluster reads that state from the same bucket. The data returned is read-only. Lab 12 makes the same split a correction task: remove dynamic backend assumptions, consume upstream values with terraform_remote_state, surface them in outputs. Lab 17 asks for data-source lookups instead of hardcoded IDs and forces you to say what is known at plan time.

08-18 already spent Shared Storage for State Files, the LockID hash key, encrypt = true, and force-unlock. Do not reopen that lock narrative. Today's question is what a second root module does with the first root module's outputs once both backends are honest.

§II — Foundations: four facts about the cross-stack read

Fact one. Producer and consumer are two root modules with two state objects.

File layout, not a shared locals map, is the durable boundary. Brikman's State File Isolation section prefers a directory per environment and per component. On AWS that looks like:

# network/backend.tf
terraform {
  backend "s3" {
    bucket         = "acme-tf-state"
    key            = "network/prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "acme-tf-locks"
    encrypt        = true
  }
}

# network/outputs.tf
output "vpc_id" {
  value = aws_vpc.main.id
}

output "private_subnet_ids" {
  value = aws_subnet.private[*].id
}

output "baseline_sg_id" {
  value = aws_security_group.baseline.id
}

The producer applies. S3 receives one object at network/prod/terraform.tfstate. Outputs live inside that object. The consumer never opens the producer's .tf files. It opens the state.

Lab 12's broken starter invents the opposite. It keeps a locals.network_outputs map of fake VPC and subnet IDs and pretends that is the network. The task list says: use terraform_remote_state correctly. The fake map is the defect.

Fact two. The backend block cannot evaluate locals or variables.

Lab 12's broken main.tf writes:

locals {
  network_state_key = "network/${var.environment}.tfstate"
}

terraform {
  backend "s3" {
    bucket = "REPLACE-ME-WITH-A-REAL-BUCKET"
    key    = local.network_state_key
    region = "us-east-1"
  }
}

Terraform configures the backend at init, before the graph exists. Expressions that need var or local do not run there. Brikman's Limitations with Terraform's Backends section labels var.bucket inside backend "s3" as code that will not work. Partial configuration (terraform init -backend-config=...) is the supported escape, spent adjacent on Lab 31 and referenced on 08-18. Today's fix for Lab 12 is simpler: a static key string per root module, or a partial config file that CI injects at init. Do not bury environment selection inside an interpolating backend block.

**Fact three. terraform_remote_state is a data source that returns outputs only.**

The consumer declares:

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-tf-state"
    key    = "network/prod/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_instance" "app" {
  ami                    = var.ami_id
  instance_type          = var.instance_type
  subnet_id              = data.terraform_remote_state.network.outputs.private_subnet_ids[0]
  vpc_security_group_ids = [data.terraform_remote_state.network.outputs.baseline_sg_id]
}

Attribute path shape, per Brikman: data.terraform_remote_state.<NAME>.outputs.<ATTRIBUTE>. Nothing the consumer does can write the producer's state. That is the safety claim. The consumer also does not acquire the producer's lock to read. If the network stack is mid-apply, the app stack sees the last completed state object. Plan-time honesty requires the producer to have finished writing the outputs you intend to consume.

Lab 17's companion claim: AWS data sources such as aws_vpc and aws_subnet look up existing infrastructure by tag or filter when you do not own the producer state. Prefer terraform_remote_state when your team owns the upstream root module and published outputs. Prefer provider data sources when the object predates Terraform or lives outside your state graph. Mixing both without saying which is which is how teams accumulate two sources of truth for one VPC ID.

Fact four. Outputs are the contract. State is not a public API.

Remote state exposes every output the producer declared. Sensitive outputs still travel into the consumer's memory during plan. Do not publish database passwords as outputs "because the app stack needs them." Publish connection endpoints and ports. Hand secrets through a store the app already trusts (Secrets Manager, SSM Parameter Store) and let each stack read what it is allowed to read. Brikman's remote_state section is about address and port into User Data, not about smuggling credentials through S3 state objects.

If the producer renames vpc_id to vpc, every consumer plan breaks at refresh. Treat output names as a versioned interface. Add, do not rename, until consumers migrate. Document the contract in the network README the same way you document a module's input variables.

§III — Mechanism: init order, refresh, and the empty plan that lies

The consumer must terraform init with credentials that can s3:GetObject on the producer key. Missing that IAM right fails refresh, not apply. The error names the bucket and key. Fix the role, not the HCL.

Apply order is producer first, consumer second. A pipeline that fans both stacks out in parallel will race: the app plan may refresh against yesterday's network outputs, then apply into a VPC that no longer matches. Serialize with an explicit job dependency, or with a promotion gate that only starts app after network reports a green apply.

An empty consumer plan after a network change is not automatically success. If the network changed a tag the app does not read, empty is correct. If the network replaced the private subnet and the app still pins private_subnet_ids[0] from an old refresh cache you forgot to re-init, empty is a lie until you refresh. Habit: terraform plan in the consumer immediately after a producer apply that touched published outputs.

Workspaces can multiply state objects under one configuration. 07-31 already spent why production promotion prefers file layout over workspace names. Today keeps workspaces out of the producer/consumer contract. Two directories, two keys, one remote_state data source. Cert will spend workspace guardrails as its own Pro drill.

When the consumer declares remote_state, Terraform refreshes that data source on every plan. Network latency to S3 becomes part of plan time. Large state objects slow refresh. Keep producer state lean: fewer resources per root module, outputs that are IDs and lists rather than entire resource objects serialized by accident through careless output blocks.

Partial backend configuration remains legal for the consumer's own backend. It is not a way to parameterize the remote_state config map with forbidden expressions inside a backend block. The remote_state config argument is ordinary HCL. It may use variables. The backend block may not. That asymmetry is intentional: remote_state runs inside the graph; backend configuration does not.

§III.B — Failure modes you will meet on the first week

Missing output. The consumer references outputs.private_subnet_ids but the producer never declared that output. Refresh fails with an attribute error. Fix the producer. Do not invent a local default in the consumer to "keep moving." The local default is how Lab 12's fake map was born.

Wrong key. A typo in the remote_state key points at an empty object or a different stack's state. Symptoms range from "NoSuchKey" to silently reading staging outputs into a prod app plan. Prefix keys with environment and component (network/prod/..., app/prod/...) and review the key in code review the same way you review IAM ARNs.

Credential split. Backend credentials for the consumer's own state and credentials for reading the producer key can differ. Some teams use one role for both. Some use a read-only role for remote_state. Document which role the runner assumes. An engineer debugging on a laptop with AdminAccess will not reproduce the CI failure that lacks s3:GetObject on the producer prefix.

Circular remote_state. Stack A reads B while B reads A. Plan deadlocks conceptually even if Terraform eventually errors. Break the cycle: extract the shared facts into a third producer both consume, or pass the needed value as a variable at apply time for the one-off migration.

Sensitive bleed. Marking an output sensitive = true redacts it from CLI UI. It still enters the consumer's state when interpolated into a resource argument. Assume anything you output can land in another state file. Design outputs accordingly.

These five modes are operational, not theoretical. Lab 12's success criteria (validate passes, plan clean) will not catch wrong-key staging bleed by themselves. Add an output that echoes the consumed VPC ID and verify it against the producer’s last apply transcript before you merge.

§IV — Worked Example: repairing Lab 12's consumer

Start from Lab 12's defects.

  1. Delete key = local.network_state_key from the backend block. Put a static key, or move the key into a partial backend config file loaded at init.
  2. Delete locals.network_outputs and the fake VPC strings.
  3. Add data.terraform_remote_state.network pointing at the real producer bucket and key.
  4. Wire outputs and resources to data.terraform_remote_state.network.outputs.*.
  5. Run terraform init, terraform validate, terraform plan. Success mode for the lab is a clean plan.

A minimal repaired consumer shape:

terraform {
  backend "s3" {
    bucket         = "acme-tf-state"
    key            = "app/prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "acme-tf-locks"
    encrypt        = true
  }
}

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-tf-state"
    key    = "network/prod/terraform.tfstate"
    region = "us-east-1"
  }
}

output "consumed_vpc_id" {
  value = data.terraform_remote_state.network.outputs.vpc_id
}

The consumed output is visible. The producer lock table is untouched by this read. The app state object is a different S3 key, so two applies do not serialize against each other unless you intentionally shared one key (the accident isolation exists to prevent).

Pair the remote_state read with a small verification habit in CI. After the network apply job finishes, the app plan job should print consumed_vpc_id and compare it to the network job’s vpc_id output artifact. A mismatch means the remote_state key, the workspace (if someone sneaked one in), or the environment prefix is wrong. Catching that in plan is cheaper than catching it when instances launch into the wrong subnet.

Do not use terraform_remote_state to reach into another team’s state without an agreement. Outputs are a contract. Unannounced reads couple your release cadence to theirs. Prefer a published module data interface, a Parameter Store path, or an explicit API the producing team owns. Inside one platform team that owns both network and app, remote_state is the right sharp tool.

§V — Connection to Prior Lessons

08-18 taught the lock that serializes writers of one file. Today's read does not take that lock. Keep both facts: writers need LockID; readers of remote_state need GetObject and a finished producer apply.

07-31 taught workspace isolation on GCP and preferred file layout for durable environments. Today's producer/consumer split is that file-layout claim applied to two AWS components in one environment.

09-02 taught replace_triggered_by when plan cannot see a needed replace. Today's empty-plan hazard is different: stale remote outputs after a producer change you did not refresh.

08-27 taught STS assume-role for the AWS provider. The remote_state config block can use the same account credentials the provider uses, or a dedicated read role. Do not confuse provider assume-role with backend config. They are adjacent IAM stories, not the same block.

§VI — Connection to Today's Dev, Cert, and Go Lessons

Dev builds a Python census over terraform show -json resource_changes so a consumer plan can be summarized as create/update/delete/replace counts before anyone applies. That is the reporting half of today's refresh discipline.

Cert spends Terraform workspaces and terraform.workspace with Lab 13 prod guardrails. Keep that tool for disposable environments. Do not fold workspace names into the remote_state key as a substitute for separate root modules.

Go lists objects in the same S3 backend bucket with AWS SDK for Go v2's config chain. The ops tool that inventories network/prod/ and app/prod/ keys is the filesystem view of the contracts this lesson declares in HCL.

One more concrete referent for the AWS overlay: the producer key network/prod/terraform.tfstate and the consumer key app/prod/terraform.tfstate can share one bucket and one DynamoDB lock table. Sharing the table is fine. Sharing the key is not. Two root modules that accidentally use the same key will take turns destroying each other's resources because each apply reconciles the same state object toward a different configuration. The remote_state data source cannot save you from that mistake. Only distinct keys can.

When you migrate from hardcoded VPC IDs to remote_state, do it in two plans. First plan adds the data source and a temporary output that prints the looked-up ID next to the old variable. Second plan switches resources to the data source and removes the variable. A single giant cutover hides whether the looked-up ID matched the variable you trusted last week.

Name the habit once more. Producer apply completes. Consumer plan refreshes remote_state. Outputs match. Then apply the app. Skip the middle refresh and you are gambling that yesterday's network still describes today's VPC. Do not skip it.

§VII — Closing

Publish outputs from the network stack. Consume them with data.terraform_remote_state. Keep backend keys static. Refresh the consumer after the producer writes. Treat output names as an interface. Leave the lock table to the writers.

Related