Hedronite · Ops Lesson · 01-Earth-DevOps / Python · Wed 2026-08-26

Python Route 53 Health Checks — the failover that is not a replica

The name moved. The data did not copy.

Lesson Class: Ops (DevOps + Python + Route 53)

Cloud Referent: AWS Route 53 — list_health_checks + get_health_check_status. Rebalances after 08-23 Azure Key Vault.

Paired Dev: try/else, for/else, the else that is not a handler

Paired Cert: SAP-C02 failover / health checks / alias vs CNAME

Grounding: Bootcamp route53.md · SAP notes 5.9 · aws-core-services Route 53

The probe

Health checks are separate objects. Records may point at them.

The pair

PRIMARY and SECONDARY are answers. Neither copies a volume.

The door

Alias is not CNAME. Checkers do not enter the VPC.

The name moved. The data did not copy.

The name moved. The data did not copy.

§I — Frame

Sunday's lesson named a chain. DefaultAzureCredential walked rungs. list_properties_of_secrets spoke the vault. No value sat in the process. Coin of that day: the credential that is not a key. The cloud seat was Azure because the Cert seat was AZ-900.

Today the Cert seat is AWS SAP-C02, the third SAP visit. 08-02 opened the landing zone. 08-14 opened the hub: Transit Gateway, RAM shares, the attachment that is not a peering. Those accounts still have to be named when a region goes dark. The name is Route 53. The concrete service an ops tool can ask without redrawing the 06-09 edge survey is the health check.

Name the duty. Coin it: the failover that is not a replica.

A replica copies data. RDS Multi-AZ copies storage to a standby you cannot read. A Route 53 failover record answers a different target when a probe fails. The primary's disk is not copied. The resolver receives a different IP or alias. That is a name change, not a clone.

The Python ops tool talks to boto3.client("route53"). It lists health checks. It asks each check for status. It lists record sets on one hosted zone and prints the pairs whose Failover field is PRIMARY or SECONDARY. It does not change_resource_record_sets. It does not flip a failover. If a primary has no HealthCheckId, the tool says so, because the Bootcamp rule is that the primary health check is mandatory (SAP notes, Route 53 Routing Policies / Failover).

08-20 already taught a census that refuses a leftover VM name. This lesson does not reopen that MIG. The leftover here is a memorized check id, or a CNAME on the apex.

§II — Foundations: four facts about the probe

Fact one. A health check is a separate object. A record may point at it.

The SAP clone says it twice: health checks are created separately and used by records (route53.md, Health Checks). The notes say the same in the failover bullet: Route 53 has checkers in locations around the world; they send requests to an endpoint you specify (SAP notes, 5.9.2.4 Failover). The cheatsheet row is one clause: "DNS, health checks, routing policies" (aws-core-services.md). That is the exam depth. The ops depth is two APIs.

list_health_checks returns HealthChecks[]. Each item has Id, HealthCheckConfig, and HealthCheckVersion. list_resource_record_sets returns ResourceRecordSets[]. A failover record may carry HealthCheckId. The join is yours. A check with no record pointing at it is still a check. A failover record with no HealthCheckId is a finding.

The CLI Evan types when the join is one check:

aws route53 get-health-check --health-check-id 01ff1c11-1111-2222-3333-abcdefabcdef
aws route53 get-health-check-status --health-check-id 01ff1c11-1111-2222-3333-abcdefabcdef
aws route53 get-health-check-last-failure-reason --health-check-id 01ff1c11-1111-2222-3333-abcdefabcdef

get-health-check is the config. get-health-check-status is the current vote. get-health-check-last-failure-reason is the last time a checker said no. Three calls. None of them copies a database.

Fact two. Failover routing is active-passive. The secondary is an address, not a copy.

Two records, same name, same type. One Failover=PRIMARY, one Failover=SECONDARY. While the primary's check is healthy, resolvers receive the primary target. When that check fails, they receive the secondary. The clone's sentence: "Failover routing should be used when we configure active-passive failover" (route53.md, Failover Routing). The notes add that the primary health check is mandatory and the secondary is optional, and they tag the pattern "Good for disaster recovery" (SAP notes, 5.9.2.4).

The secondary can be another region's ALB, a static S3 website that says "we are down", or a record in the same zone. Nothing in that pair replicates rows. If the stem says "the secondary kept serving yesterday's writes," the candidate has imported RDS vocabulary into a DNS question. Send them back to 08-14's coin: the attachment was not a peering. Today's coin is the cousin at the edge. The failover is not a replica.

Multi-value routing is the other trap. Many records, same name, each with a check. Route 53 returns up to eight healthy values. The client picks. The clone is blunt: "Multi value routing is not a substitute for an actual load balancer" (route53.md, Multi Value Routing). 08-20's MIG plus HTTP load balancer replaces an unhealthy VM. Multi-value omits an unhealthy answer. Different verb. Do not write a census that treats MULTIVALUE as FAILOVER. Print the policy. Leave the substitution to the on-call.

Simple routing cannot attach a health check at all (route53.md, Simple Routing; SAP notes, 5.9.2.4 Simple). A zone that looks like failover and is typed SIMPLE is a finding, even if someone created a check and never joined it.

Fact three. An alias is not a CNAME.

A CNAME maps a name to another name. It cannot sit on the apex. It cannot point at an IP. Many AWS fronts (ALB, CloudFront, S3 website) give you a DNS name, not a stable IPv4. The clone's problem statement is exactly that (route53.md, CNAME vs Alias Records). An alias maps a name to an AWS resource. It works on the apex. Alias queries that point at AWS resources are not billed as extra DNS queries. An alias is a subtype of an A or AAAA (or, in some cases, a CNAME alias). It is not a CNAME with a friendlier name.

list_resource_record_sets shows the difference on the wire. A CNAME record has ResourceRecords[].Value. An alias record has AliasTarget with DNSName, HostedZoneId, and EvaluateTargetHealth. If EvaluateTargetHealth is true, Route 53 asks the target (the ALB, the other record) whether it is healthy, and you may skip a separate endpoint check. If it is false, the alias is always a candidate. A census that prints Type and ignores AliasTarget will call an alias a CNAME and fail the next review.

Fact four. The checkers live outside your VPC.

The notes are the sentence that fails private-endpoint designs: "Route 53 health checkers are outside the VPC. They can't access private endpoints" (SAP notes, 5.9.3.3 Private Hosted Zones). The clone's health-check types are three: endpoint, CloudWatch alarm, calculated (route53.md, Health Checks). Endpoint checks need a public IP, or a public name that answers 2xx or 3xx within the timer. A TCP connect must complete in four seconds. The HTTP body, if you asked for a string match, must contain the text in the first 5120 characters (route53.md; SAP notes, 5.9.3.2 Setup).

A private ALB, an RDS hostname, an on-premises VIP reached only over Direct Connect: an endpoint check cannot see them. The notes give three exits. Give the resource a public IP (usually the wrong exit). Check something public the instance depends on. Or create a CloudWatch alarm and point a health check at the alarm (SAP notes, 5.9.3.3). Calculated checks are a parent over children: you say how many children must pass (SAP notes, 5.9.3.2; route53.md, Calculated Checks). The parent is still a health-check object with an Id. The census lists it like any other.

Default interval is 30 seconds. Ten seconds costs extra (route53.md). Eighteen percent or more of the checkers must report healthy for the check to be healthy (route53.md). get_health_check_status returns the per-checker votes. A tool that only prints HealthCheckConfig has not asked the fleet.

§III — Mechanism: the client, the join, the finding

The client you construct once

import boto3

r53 = boto3.client("route53")

Route 53 is a global service. There is no region_name="us-east-1" on this client the way there is on EC2. The signed endpoint is route53.amazonaws.com. A wrapper that passes region_name into boto3.client("route53") is carrying a habit from an RDS call. Drop the argument. Private-zone VPC attachments still use this same global client.

Construct the client once per process. list_health_checks paginates. get_paginator("list_health_checks") is the loop. A while that re-calls list_health_checks and forgets Marker / NextMarker will print twenty checks and leave the rest in the account.

The join you do yourself

Health checks do not know which records use them. Records know the check id. Walk checks first, then records.

For each check, print Id, Type, FullyQualifiedDomainName or IPAddress, ResourcePath, RequestInterval, FailureThreshold, AlarmIdentifier, ChildHealthChecks, and Disabled. Then call get_health_check_status. Print how many checkers reported Success and how many reported Failure. If the check is CALCULATED or CLOUDWATCH_METRIC, say so. An endpoint field on a calculated check is a misread of the config.

For each record in the named hosted zone, print Name, Type, SetIdentifier, Failover, Weight, Region, GeoLocation, HealthCheckId, and whether AliasTarget is present. A pair is two records with the same Name and Type and complementary Failover values. If you find a PRIMARY without a SECONDARY, that is a finding. If you find a PRIMARY without HealthCheckId and without AliasTarget.EvaluateTargetHealth, that is a finding. The Bootcamp rule is the authority, not a style preference.

The finding you refuse to "fix"

The tool does not call change_resource_record_sets. A census that "helpfully" attaches a check you just created is a change window, not a read. 08-17 taught a signal that must still fire: the CloudFormation ResponseURL POST. Today's cousin is the opposite duty. The signal that must still fire is the line that says the primary is unpaired. Silence that says "I repaired it" is a lie the next on-call cannot replay.

Exit 2 versus exit 1 follows 08-02. Exit 2 means the client could not run (no credentials, AccessDenied on ListHealthChecks). Exit 1 means the client ran and the zone is in a bad shape (unpaired primary, simple record pretending to fail over, endpoint check aimed at a 10.0.0.0/8 address). A supervisor that retries 2 and pages on 1 is reading the right number.

§IV — Worked example: the census

The script takes a hosted-zone id. It lists every health check in the account, then every record in that zone. It prints the join. It never writes.

import sys
import boto3
from botocore.exceptions import ClientError, NoCredentialsError


def status_votes(r53, health_check_id):
    try:
        body = r53.get_health_check_status(HealthCheckId=health_check_id)
    except ClientError as exc:
        code = exc.response.get("Error", {}).get("Code", "unknown")
        return f"status-error:{code}", 0, 0
    observations = body.get("HealthCheckObservations") or []
    ok = sum(1 for o in observations if (o.get("StatusReport") or {}).get("Status", "").startswith("Success"))
    bad = len(observations) - ok
    return "observed", ok, bad


def main(argv):
    if len(argv) < 2:
        print("usage: census.py HOSTED_ZONE_ID", file=sys.stderr)
        return 2
    zone_id = argv[1]
    try:
        r53 = boto3.client("route53")
    except NoCredentialsError:
        print("no credentials", file=sys.stderr)
        return 2

    worst = 0
    checks = {}
    try:
        pager = r53.get_paginator("list_health_checks")
        for page in pager.paginate():
            for hc in page.get("HealthChecks") or []:
                hid = hc["Id"]
                cfg = hc.get("HealthCheckConfig") or {}
                checks[hid] = cfg
                label, ok, bad = status_votes(r53, hid)
                target = cfg.get("FullyQualifiedDomainName") or cfg.get("IPAddress") or cfg.get("AlarmIdentifier") or ""
                print(
                    f"check id={hid} type={cfg.get('Type')} interval={cfg.get('RequestInterval')} "
                    f"disabled={cfg.get('Disabled')} target={target} votes={label} ok={ok} bad={bad}"
                )
                if cfg.get("Type") == "HTTP" and str(cfg.get("IPAddress") or "").startswith("10."):
                    print(f"finding private-endpoint-check id={hid}", file=sys.stderr)
                    worst = max(worst, 1)
    except ClientError as exc:
        code = exc.response.get("Error", {}).get("Code", "unknown")
        print(f"list-health-checks refused code={code}", file=sys.stderr)
        return 2

    try:
        records = []
        marker = None
        while True:
            kwargs = {"HostedZoneId": zone_id}
            if marker:
                kwargs["StartRecordName"] = marker["Name"]
                kwargs["StartRecordType"] = marker["Type"]
                if "SetIdentifier" in marker:
                    kwargs["StartRecordIdentifier"] = marker["SetIdentifier"]
            page = r53.list_resource_record_sets(**kwargs)
            records.extend(page.get("ResourceRecordSets") or [])
            if not page.get("IsTruncated"):
                break
            marker = {
                "Name": page["NextRecordName"],
                "Type": page["NextRecordType"],
            }
            if "NextRecordIdentifier" in page:
                marker["SetIdentifier"] = page["NextRecordIdentifier"]
    except ClientError as exc:
        code = exc.response.get("Error", {}).get("Code", "unknown")
        print(f"list-resource-record-sets refused code={code}", file=sys.stderr)
        return 2

    pairs = {}
    for rr in records:
        fail = rr.get("Failover")
        if not fail:
            continue
        key = (rr["Name"], rr["Type"])
        pairs.setdefault(key, {})[fail] = rr
        alias = rr.get("AliasTarget") or {}
        print(
            f"record name={rr['Name']} type={rr['Type']} failover={fail} "
            f"health={rr.get('HealthCheckId')} alias={bool(alias)} "
            f"evaluate={alias.get('EvaluateTargetHealth')} "
            f"cname_values={len(rr.get('ResourceRecords') or [])}"
        )
        hid = rr.get("HealthCheckId")
        if hid and hid not in checks:
            print(f"finding dangling-health-check-id name={rr['Name']} id={hid}", file=sys.stderr)
            worst = max(worst, 1)

    for key, sides in sorted(pairs.items()):
        if "PRIMARY" not in sides:
            print(f"finding secondary-without-primary name={key[0]} type={key[1]}", file=sys.stderr)
            worst = max(worst, 1)
            continue
        primary = sides["PRIMARY"]
        if "SECONDARY" not in sides:
            print(f"finding unpaired-primary name={key[0]} type={key[1]}", file=sys.stderr)
            worst = max(worst, 1)
        hid = primary.get("HealthCheckId")
        evaluate = (primary.get("AliasTarget") or {}).get("EvaluateTargetHealth")
        if not hid and not evaluate:
            print(f"finding primary-without-probe name={key[0]} type={key[1]}", file=sys.stderr)
            worst = max(worst, 1)
    return worst


if __name__ == "__main__":
    sys.exit(main(sys.argv))

Four disciplines sit in that script.

The pagination is written out so the next editor sees IsTruncated and the three start keys. A forgotten StartRecordIdentifier on a weighted or failover set drops the rest of the name.

status_votes treats a missing status call as a string, not as a raise that kills the rest of the census. One denied check must not hide the unpaired primary.

The private-endpoint finding is a prefix test on 10., which is a hint, not a proof. RFC1918 has three ranges. The hint is enough to send a human to SAP notes 5.9.3.3. Do not auto-rewrite the check as CloudWatch.

The script never prints a secret. Route 53 records are names and ids. Still refuse repr(exc) on ClientError: the response body can carry request XML. Print the error Code. 08-23 taught that cousin on HttpResponseError.status_code. Same door, different SDK.

§V — Connection to prior lessons

08-23 taught a census that refuses get_secret. list_health_checks is to Route 53 what list_properties_of_secrets was to the vault: the list that does not start from last night's handle. The leftover in DNS is a memorized check id, or a CNAME on the apex.

08-20 taught a census that refuses a leftover VM name. list_managed_instances spoke the MIG. Today's leftover is a record you treat as a replica because the word "secondary" sat next to "standby" in another chapter. The secondary is an answer. It is not a copy of the primary's volume.

08-17 taught a signal that must still fire. A Lambda that raised without a PUT left CloudFormation in CREATE_IN_PROGRESS. Today's cousin is a census that "repairs" a missing secondary and then exits 0. The signal that must still fire is the finding line. The write belongs to a different tool, on a different change ticket.

08-14's Cert lesson took the hub. This Ops lesson does not reopen Transit Gateway or RAM. The hub is how packets move. The name is how clients find the front door.

§VI — Connection to today's Dev lesson

Today's Dev lesson is try / except / else, and the same else on for and while. The else on try runs when no exception was raised. It is not a second handler. People treat it as a replica of except. It is the healthy-primary path.

The ops rule is: a failover record is a different answer, not a copy of the primary. The language rule is: else is a different clause, not a copy of except. One coin. The failover that is not a replica is also the else that is not a handler.

§VII — Closing

Construct the client once. List the checks. Ask each one for votes. List the records on the zone you were given. Join them yourself. Print unpaired primaries. Print a primary with no probe. Print an HTTP check aimed at a private address. Do not write a record.

The Bootcamp sentence is short: health checks are separate, failover is active-passive, alias is not CNAME. The ops sentence is shorter. The process that listed them flipped no name.

Examine the next failover pair in the zone. If the secondary exists because someone thought DNS would copy the database, the coin is already spent.

Prior arc: the credential that is not a key (2026-08-23)
Domain hub: Cross-References/domains/01-Earth-DevOps
Grounding: Route53 (Sovereign-Bootcamp SAP clone) · SAP notes — Route 53 Core Topic