AWS Backup coverage unprotected resources, copy rules, Vault Lock state, and a Rust census

A plan says what should happen. A recovery point says what did. Only a restorable point counts.

A backup plan says what should happen. A recovery point says what did. Only a restorable point counts.

§I. Frame

The last four Cloud Ops lessons were queues and topics: SNS subscriptions on 10-04, Service Bus on 10-01, Pub/Sub on 09-28, SQS on 09-22. Today leaves messaging for AWS Backup, which no lesson on SoT owns as its subject.

SRE ch26 states the rule this census enforces: no one wants backups, people want restores, and a backup schedule should follow how much recent data you can afford to lose (book p.343). That figure is the RPO. A plan with a daily rule promises a 24-hour RPO. The census checks the promise against the evidence.

The problem for today: for one account and Region, find resources with no recovery point, resources whose newest restorable point is older than the RPO, plans that never copy to another Region, and vaults whose lock can still be removed.

§II. Exists, Touched, Restorable

Exists, Touched, Restorable (named technique). Coverage is the comparison of three lists. Read each from a different API.

1. Exists. AWS Backup has no call that lists every resource in the account. The Resource Groups Tagging API (GetResources with type filters) gives the inventory. It returns only resources that have or once had tags, so an untagged volume is invisible to it. Say so in the report.
2. Touched. ListProtectedResources returns resources with recovery points created by AWS Backup, "regardless of the recovery point's status." Its LastBackupTime is a lead. A point still CREATING counts as touched.
3. Restorable. ListRecoveryPointsByResource, filtered to COMPLETED or AVAILABLE. The DescribeRecoveryPoint reference defines PARTIAL as a composite point missing nested points, EXPIRED as past retention but undeleted, and STOPPED as a continuous backup that was disabled. None of those is a point you want to restore from at 03:00.

Exists minus Touched is unprotected. Touched with an old Restorable is stale_recovery_point. Touched newer than Restorable is newest_not_restorable: the console looks fresh, the restore does not.

§III. Copy geography and the lock date

Copy rules. A backup rule may carry CopyActions. Each names a DestinationBackupVaultArn. Field 3 of that ARN is the Region and field 4 is the account. If no rule copies anywhere, the plan dies with the Region. If every copy lands in the plan's own Region, it still dies with the Region. The census flags no_copy_rule and copy_same_region_only, and notes cross_account_copy when field 4 differs from the plan's account.

The Lock Date Speaks (named technique). The vault API has no mode field. Locked: true appears in three different states, and LockDate tells them apart:

The Vault Lock guide gives the mechanics. PutBackupVaultLockConfiguration without ChangeableForDays creates governance mode. With it, compliance mode, and the value must fall between 3 and 36,500 days. After the grace time ends, the lock and the vault cannot be changed or deleted by any user or by AWS while recovery points remain. A logically air-gapped vault arrives locked in compliance mode.

Only compliance_locked passes. Governance is a door with a key in IAM. Grace is a door that closes on a date, and the census prints the days left.

§IV. The Rust census

Automation is a cargo script over aws-sdk-backup and aws-sdk-resourcegroupstagging. Four passes, each paginated by hand:

1. list_protected_resources into a BTreeMap<String, i64> of ARN to last_backup_time().secs().
2. get_resources over five types (ec2:volume, rds:db, dynamodb:table, elasticfilesystem:file-system, s3). For a touched ARN, list_recovery_points_by_resource finds the newest restorable point.
3. list_backup_plans then get_backup_plan, reading rules() and copy_actions().
4. list_backup_vaults, matching on (locked(), lock_date()). Restore access vaults are skipped: they are views of an air-gapped vault and hold no points.

The lock match carries the whole of §III:

let state = match (v.locked(), v.lock_date()) {
    (Some(true), None) => "governance_lock".to_string(),
    (Some(true), Some(d)) if d.secs() > now => format!("compliance_grace={}d", (d.secs() - now) / 86_400),
    (Some(true), Some(_)) => "compliance_locked".to_string(),
    _ => "no_vault_lock".to_string(),
};

Checked on the lab Mac. cargo check exit 0 with no warnings (aws-sdk-backup 1.127.0, rustc 1.99.0). Under cargo +nightly -Zscript with every credential source switched off, the script compiled and stopped at its first call:

Error: dispatch failure
    2: no credentials found in chain.
exit 1

Illustrative output (no AWS credentials in this lab; no AWS call was made, and every row below is invented in the documented shapes):

resource	arn:aws:rds:us-east-1:111122223333:db:orders	ATTENTION	stale_recovery_point=35h,newest_not_restorable
resource	arn:aws:ec2:us-east-1:111122223333:volume/vol-0a1b2c3d4e5f60718	ATTENTION	unprotected
resource	arn:aws:elasticfilesystem:us-east-1:111122223333:file-system/fs-0123456789abcdef0	ok
plan	prod-daily	ATTENTION	copy_same_region_only
plan	prod-12h	ok	cross_account_copy
vault	prod-daily	BACKUP_VAULT	ATTENTION	governance_lock	min=Some(7) max=Some(35)
vault	dr-airgap	LOGICALLY_AIR_GAPPED_BACKUP_VAULT	ok	compliance_locked	min=Some(7) max=Some(35)
vault	Default	BACKUP_VAULT	ATTENTION	no_vault_lock	min=None max=None
summary region=us-east-1 resources=3 plans=2 vaults=3 attention=5 rpo_hours=24

§V. How to run

AWS_REGION=us-east-1 cargo +nightly -Zscript ./aws-backup-coverage-census.rs --rpo-hours 24

IAM for a read-only run: backup:ListProtectedResources, backup:ListRecoveryPointsByResource, backup:ListBackupPlans, backup:GetBackupPlan, backup:ListBackupVaults, tag:GetResources. The script exits 2 when any row needs attention and never writes. Nix wrapper: aws-backup-coverage-census.nix (writeBashBin), built on the lab Mac this fire.

AWS Backup Audit Manager has managed controls for most of these flags: resources in at least one plan, cross-Region copy scheduled, resources in a plan with a Vault Lock, last recovery point created. Use them in production. The census shows what each control reads.

§VI. Close

Exists, Touched, Restorable: compare three lists, trust only the third. The Lock Date Speaks: read LockDate before you call a vault immutable. Run the census against your own account once the credential hold lifts, and write down the RPO each plan promises before you read the output.

Paired Dev writes the iterator that turns a resource's recovery points into gaps and finds the RPO breach. Paired Cert carries the same three ideas across accounts with Organizations backup policies.

Related

• Tome: SRE ch26 Backups Versus Archives, book pp.343-344 (grounded-in)
• Bootcamp: SAP dr.md, backup and restore tier (referenced)
• Prior Cloud Ops: SNS 10-04 · Service Bus 10-01 · StackSets 09-25 · Object Lock/WORM (Cert) 06-23
• Web: ListProtectedResources · BackupVaultListMember · Vault Lock · DescribeRecoveryPoint · Audit Manager controls