Terraform AWS CloudWatch Log Groups retention_in_days and a Rust census

Omit retention and the group keeps every byte forever. Set a floor, then ask the API.

<!-- hal:authoritative:yaml -->

Omit retention and the group keeps every byte forever. Set a floor in days, then ask the Logs API what each group actually reports.

§I. Frame

Yesterday Maghrib closed Cloud #137. Tonight is TF day 72, trio #138. The cloud-native referent is AWS CloudWatch Logs: the log group that holds streams, the retention field that deletes aged events, and the metric filter that turns matching lines into a CloudWatch metric.

09-10 already counted CloudWatch alarms with boto3. That stack is retired. ECR 09-26 taught registry scanning on the same cloud. Artifact Registry 09-29 taught cleanup policies on GCP. Those shelves are adjacent and not this claim.

The problem for today: declare a CloudWatch log group the current way in Terraform (retention_in_days, optional KMS, a metric filter), then build a read-only Rust census that flags groups with Never Expire or retention below a floor you choose.

§II. Three owners on one log group

Retention Owner (named technique). Ask which field owns deletion before asking what the filter pattern looks like.

1. Retention belongs to retention_in_days on aws_cloudwatch_log_group. Valid values are a closed enum (1, 3, 5, 7, 14, 30, 60, 90, …, 3653). Omit the argument and AWS treats the group as Never Expire. That is not "infinite days"; it is a distinct API state.
2. Encryption belongs to optional kms_key_id. Null means the AWS-owned key. A customer CMK is a separate plan decision.
3. Signal extraction belongs to aws_cloudwatch_log_metric_filter. The filter attaches to the group by name. Creating the group does not create the filter.

If you mix up the owners, a green plan can leave Forever retention while dashboards pretend the group ages out.

§III. The Terraform

This module is the teaching referent. Validate it with terraform init -backend=false and terraform validate against hashicorp/aws ~> 5.0. Do not apply from this fire; no account mutations tonight.

resource "aws_cloudwatch_log_group" "app" {
  name              = "${var.name_prefix}/main"
  retention_in_days = var.retention_in_days
  kms_key_id        = var.kms_key_id

  tags = {
    Name = "${var.name_prefix}/main"
  }
}

resource "aws_cloudwatch_log_metric_filter" "error_rate" {
  name           = "${var.name_prefix}-error-count"
  log_group_name = aws_cloudwatch_log_group.app.name
  pattern        = "{ $.level = \"ERROR\" }"

  metric_transformation {
    name      = "AppErrorCount"
    namespace = "Hedronite/App"
    value     = "1"
  }
}

Fact one. retention_in_days missing is Never Expire. Reviewers who only read the resource type will miss it.

Fact two. The day enum is closed. retention_in_days = 45 fails validation. Pick a documented value or leave the field unset on purpose.

Fact three. The metric filter is its own plan row. Pattern syntax is CloudWatch filter grammar ({ $.level = "ERROR" } for JSON logs), not an arbitrary regex and not a heredoc story.

Fact four. Pair Dev tonight on jsonencode / templatefile when a filter pattern or IAM-ish document should be structured HCL instead of a raw string pasted into the attribute.

Bundle file: cloudwatch-log-groups.tf next to this lesson.

§III.b. Subscription filter edge (optional)

A subscription filter (aws_cloudwatch_log_subscription_filter) ships matching events to Kinesis, Firehose, or Lambda. It is a fourth owner: delivery, not retention. Do not confuse it with the metric filter. Retention still deletes aged events from the group even when a subscription is copying a subset elsewhere.

Tonight's referent stops at the metric filter. Name the subscription shelf so Maghrib can quiz the boundary later without inventing a fifth resource.

§IV. The Rust census

Automation is a cargo script, not a Python CLI. The AWS Rust client aws-sdk-cloudwatchlogs exposes DescribeLogGroups. Each LogGroup carries log_group_name, retention_in_days (absent when Never Expire), and stored_bytes.

let conf = aws_config::defaults(BehaviorVersion::latest()).load().await;
let client = Client::new(&conf);
let mut paginator = client.describe_log_groups().into_paginator().send();
while let Some(page) = paginator.next().await {
    let page = page?;
    for g in page.log_groups() {
        // classify: name, retention_in_days, stored_bytes
        // ATTENTION when Never Expire or days < floor
    }
}

Full script: cloudwatch-log-group-census.rs (nightly -Zscript). Nix wrapper: cloudwatch-log-group-census.nix (writeBashBin).

Illustrative output (no AWS credentials on the lab Mac as of this fire; the lab must export working credentials before a live census):

/hedronite/app/main retention_days=30 stored_bytes=1048576
/hedronite/legacy/api retention_days=NeverExpire stored_bytes=4194304 ATTENTION
/hedronite/scratch retention_days=7 stored_bytes=0 ATTENTION
summary floor_days=30 groups=3 never_expire=1 below_floor=2 attention=2

Flag ATTENTION when retention is Never Expire or when reported days sit below the floor argument (default 30). Those two states are the retention bugs the census exists to find.

§V. How to run

# Validate the referent (no apply)
terraform init -backend=false
terraform validate

# Census (needs working AWS credentials + region)
cargo +nightly -Zscript ./cloudwatch-log-group-census.rs 30 /hedronite
# or: nix-build -E 'with import <nixpkgs> {}; callPackage ./cloudwatch-log-group-census.nix {}'

If credential resolution fails, stop. Re-auth is a human step on the lab Mac. Do not paste keys into the lesson tree.

§VI. Close

CloudWatch Logs is the AWS event shelf for application stdout. Tonight's Ops claim is three owners on one group: retention (including Never Expire), optional KMS, and a separate metric filter. The Rust census reads what Logs reports, not what the HCL hoped.

Paired Dev: HCL jsonencode, templatefile, and when a filter pattern should stay structured. Paired Cert: Associate encoding functions and collection helpers on the same surface.

Related

• Tome: Brikman 3e p.89 (provider credentials via environment) — referenced
• Bootcamp: AWS DevOps Pro Notes (observability) — referenced; tfpro Lab 15 (retention adjacency) — referenced
• Prior Ops: Artifact Registry 09-29 · ECR scan 09-26 · CloudWatch alarms 09-10 (boto3, different claim)