Python TypedDict — total=False for partial API shapes

Raw AWS dicts are partial. Normalize into required census rows. Keep enrichments NotRequired.

<!-- hal:authoritative:yaml -->

*describe_vpc_endpoints returns dicts. Some keys are always present. Some appear only for Interface endpoints. TypedDict with total=False names that partial shape without lying to the type checker.*

§I — Frame

Last Python-day Dev lesson walked copy versus deepcopy. The visit before that weighed generator expressions against list comprehensions. Both were pure language depth with a cloud metaphor nearby.

Today stays in the type system. boto3 returns nested dicts. The Ops census for Interface VPC Endpoints reads VpcEndpointId, ServiceName, SubnetIds, and sometimes PolicyDocument or DnsEntries. If you annotate those rows as a closed dataclass with required fields, every .get() becomes a fight. If you leave them as dict[str, Any], the checker helps nobody.

typing.TypedDict describes a dictionary shape. total=False marks keys as optional. That is the idiom for partial API envelopes.

Fluent Python treats TypedDict as the tool for JSON-shaped and API-shaped mappings where keys are known strings and values have known types. This lesson applies that tool to the PrivateLink census shape without turning into an AWS networking lecture. Networking depth lives in Cert and Ops.

§II — Language idiom: required keys, optional keys

from typing import TypedDict, NotRequired


class VpcEndpointRow(TypedDict):
    id: str
    vpc: str
    service: str
    state: str
    private_dns: bool
    subnets: list[str]
    sgs: list[str]
    dns_count: int
    azs: NotRequired[list[str]]
    policy_chars: NotRequired[int]

On Python 3.11+, NotRequired is clearer than splitting into two TypedDict classes. On 3.8–3.10, the classic form is:

class VpcEndpointRowRequired(TypedDict):
    id: str
    vpc: str
    service: str
    state: str
    private_dns: bool
    subnets: list[str]
    sgs: list[str]
    dns_count: int


class VpcEndpointRow(VpcEndpointRowRequired, total=False):
    azs: list[str]
    policy_chars: int

Inheritance with total=False on the child adds optional keys while keeping the parent keys required. That is the pattern Ramalho-style typing chapters push: make the required surface small and honest.

§III — Worked example: normalize boto3 into TypedDict

from __future__ import annotations

from typing import TypedDict, NotRequired, Iterable


class RawEndpoint(TypedDict, total=False):
    VpcEndpointId: str
    VpcId: str
    ServiceName: str
    State: str
    PrivateDnsEnabled: bool
    SubnetIds: list[str]
    Groups: list[dict]
    DnsEntries: list[dict]
    PolicyDocument: str


class EndpointRow(TypedDict):
    id: str
    vpc: str
    service: str
    state: str
    private_dns: bool
    subnets: list[str]
    sgs: list[str]
    dns_count: int
    policy_chars: NotRequired[int]


def normalize(ep: RawEndpoint) -> EndpointRow:
    row: EndpointRow = {
        "id": ep["VpcEndpointId"],
        "vpc": ep["VpcId"],
        "service": ep["ServiceName"],
        "state": ep["State"],
        "private_dns": bool(ep.get("PrivateDnsEnabled")),
        "subnets": list(ep.get("SubnetIds") or []),
        "sgs": [g["GroupId"] for g in ep.get("Groups") or []],
        "dns_count": len(ep.get("DnsEntries") or []),
    }
    policy = ep.get("PolicyDocument")
    if policy is not None:
        row["policy_chars"] = len(policy)
    return row


def normalize_many(pages: Iterable[dict]) -> list[EndpointRow]:
    out: list[EndpointRow] = []
    for page in pages:
        for ep in page.get("VpcEndpoints") or []:
            out.append(normalize(ep))  # type: ignore[arg-type]
    return out

Three discipline points:

1. **Raw stays total=False.** AWS omits keys. Pretending every key exists is how you get KeyError in production and false confidence in CI.
2. Normalized rows require the census columns. After normalize, callers can rely on id and service without .get.
3. **Optional enrichment stays NotRequired.** policy_chars and later azs appear only when you paid for the extra API call.

§III.B — total=True traps

class Bad(TypedDict):  # total=True by default
    id: str
    policy: str


def oops(ep: dict) -> Bad:
    return {"id": ep["VpcEndpointId"], "policy": ep["PolicyDocument"]}

If PolicyDocument is missing, you crash at runtime. The type checker may still accept ep["PolicyDocument"] if ep is Any. TypedDict helps only when you type the input as partial and branch on .get.

Another trap: using Optional[str] inside a total=True TypedDict. That means the key is present and the value may be None. It does not mean the key may be absent. Absence is NotRequired or total=False. Presence-with-null is Optional.

§III.C — Structural checks with TypeGuard

from typing import TypeGuard


def is_endpoint_row(obj: object) -> TypeGuard[EndpointRow]:
    if not isinstance(obj, dict):
        return False
    required = ("id", "vpc", "service", "state", "private_dns", "subnets", "sgs", "dns_count")
    return all(k in obj for k in required)

Use TypeGuard at trust boundaries: after json.load from a cached census file, after a subprocess that printed JSON. Inside a live boto3 loop, normalize is enough.

§IV — Connection to today's Ops and Cert

Ops builds the census with plain dicts first. Promote those dicts to EndpointRow when the script grows a second consumer (Slack formatter, CSV writer, Go parity fixture). Cert explains why SubnetIds and Groups matter for Interface endpoints and why Gateway endpoints omit them. The type shape encodes the architecture: optional keys track optional AWS fields.

§V — Prior-lesson reach

09-04 taught that shallow copy shares nested mutables. A TypedDict row that stores subnets: list[str] still shares that list if you assign without copying. Normalize with list(...) as shown.

09-01 taught generator expressions for lazy scans. You can stream pages into normalize_many with a generator of pages and avoid materializing every raw endpoint when you only need WARN lines.

§VI — Closing

TypedDict names dictionary shapes. total=False / NotRequired names absence. Cloud APIs are absence-heavy. Type the raw envelope as partial, normalize into a required census row, and keep enrichments optional. The checker then guards the boundary that used to be a midnight KeyError.

Examine one live describe_vpc_endpoints payload and mark each key required or optional before you freeze the TypedDict.

Related

• Python ops VPC endpoint census
• copy versus deepcopy
• AWS SAP PrivateLink
• Cross-References/Archmagus-Stack — Polyglot-Dev hub
• Grounding: Fluent Python 2nd TypedDict; Bootcamp privatelink.md as motivating shape