Configuration and Secrets for Python Ops Tools — the pointer, never the value

A tool that reads its configuration in the middle of a run has already decided to fail in the middle of a run.

A tool that reads its configuration in the middle of a run has already decided to fail in the middle of a run.

§ IFrame: The Fifth Rung

Four rungs are behind us. The 07-24 lesson taught what a single step owes the operator when it breaks. The 07-27 lesson scaled that to a fleet and made the partial-failure ledger the deliverable. The 07-30 lesson turned the working script into an installable package with entry points. The 08-02 lesson made the unattended run leave three witnesses: log line, heartbeat, exit code.

Every one of those lessons assumed the tool already knew two things it was never told how to learn. Which environment am I running against. What credential am I allowed to use. Today those two questions get their own rung.

The naive answer is a constant at the top of the file. The slightly-less-naive answer is os.environ["DB_PASSWORD"] on line 400, deep inside the retry loop, where the KeyError arrives forty minutes into a sweep across two hundred hosts. Both answers share one defect, and naming it is most of the lesson. Call it the late read: the program discovers what it needs at the moment it needs it, rather than at the moment it starts. The late read converts a configuration error into a runtime failure, and a runtime failure into a half-finished job.

The discipline that cures it has three parts, and they run in a fixed order at process start. First, collect every source of settings into one merged view with declared precedence. Second, validate that view against a schema and refuse to continue if it does not hold. Third, resolve the secrets the validated view points at, once, through a cache, from a service built to hand them out. Collect, validate, resolve. Nothing after that touches os.environ again.

§ IIFoundations: The Four Layers and Their Order

Configuration arrives from four places, and an ops tool that runs on a laptop, in CI, and on a scheduled EC2 host will meet all four in the same week.

Defaults in code. Every setting the tool has, with the value that makes a first run work. This layer is documentation that executes. A reader who wants to know what the tool can be told opens one dataclass and reads it.

A config file. TOML for anything with structure, per the packaging conventions the 07-30 lesson established around pyproject.toml. The file is committed for non-secret values and holds the shape of an environment: region, endpoint, timeout, concurrency, the paths the tool writes to.

Environment variables. The layer the scheduler and the container runtime actually control. The twelve-factor argument for this layer is not that environment variables are elegant; it is that they are the one channel every orchestrator already speaks, from a systemd unit file to an ECS task definition.

Command-line flags. The operator standing at the terminal right now, overriding everything, usually to run a single host or to force a dry run.

The order above is also the precedence order, lowest to highest. Flags beat environment, environment beats file, file beats defaults. State the order in the tool's help text, because an operator debugging why a setting will not take is nearly always debugging a precedence surprise.

Notice which layer is missing from that list. Secrets do not appear, and the omission is deliberate. A secret in the config file is a secret in the repository. A secret in an environment variable is a secret in the process table, in the container inspect output, in the crash dump, and in every child process the tool ever spawns. Liz Rice makes the point plainly in the secrets chapter of Container Security (ch. 12, pp. 165-166): environment variables are visible to anyone who can inspect the container, they leak into logs by way of well-meaning dumps, and a file-based secret is at least revocable by unmounting the file. Her second observation is sharper still. A secret passed through a mounted file is readable by root on the host, so the file mechanism raises the bar without clearing it. The conclusion for a Python ops tool is not "use files instead of environment," but rather this: what the four layers carry is a pointer to a secret, never the secret itself.

The pointer is a name. /hedronite/prod/db/password. The tool reads that name from its merged configuration, then asks a service for the value behind it at startup, holds it in memory, and never writes it anywhere.

§ IIIMechanism: Collect, Validate, Resolve

Collect

Merging is a dictionary fold in precedence order. The subtlety is nesting. A flat dict.update() on a nested structure replaces whole sub-trees, so a file that sets three fields under [retry] will be erased entirely by an environment variable that sets one of them. Merge recursively, or flatten the namespace on the way in. Environment keys map to paths by a fixed convention, HEDRONITE_RETRY__MAX_ATTEMPTS to retry.max_attempts, with the double underscore as the separator so single underscores stay available inside field names.

Validate

The merged view is a dictionary of strings and nobody should trust it. Validation is where a dataclass or a Pydantic model earns its place: types coerced, ranges checked, mutually exclusive options caught, required fields proven present. This is the same boundary discipline the 08-02 paired Dev lesson drew around an ops library's public surface, applied one layer earlier, at the edge where the outside world hands the process untyped text.

Validation failure exits nonzero with a message naming the field, the offending value, and the layer it came from. That last part is what turns a support ticket into a ten-second fix. The 08-02 lesson's exit-code discipline applies without modification here: a configuration error is exit 2, distinct from a work failure at exit 1, because a supervisor should retry the second and never the first.

Resolve

Now the secrets. AWS Systems Manager Parameter Store is the plainest tool for the job: hierarchical names, optional KMS encryption on SecureString parameters, version tracking, IAM policy applied by path prefix, no per-secret charge on standard parameters. The DOP study corpus states the shape compactly under Policies and Standards Automation, and it names the property that matters for a tool holding many settings at once: parameters are addressable by path, so one call fetches a whole namespace.

That last property is the difference between one API call and forty. get_parameters_by_path with Recursive=True and WithDecryption=True returns every parameter under /hedronite/prod/, paginated, in one logical operation. Forty individual get_parameter calls will meet the account's throughput limit on a bad afternoon and turn a startup into a retry storm.

Secrets Manager is the right choice when rotation is the requirement. It runs a Lambda on a schedule, versions the secret with staging labels, and hands out the value under AWSCURRENT while the pending one is tested. A client that caches must therefore hold the value with a deadline, not forever, and must be able to re-resolve on an authentication failure rather than dying on it. The rule of thumb: Parameter Store for configuration that happens to be sensitive, Secrets Manager for credentials that rotate on a schedule.

Caching is a correctness concern before it is a cost concern. Google's SRE book makes the general form of the argument in its treatment of idempotent convergence (Part II, p. 104): an operation that re-runs must reach the same state as the first run rather than compound its effects. Applied here, a tool that re-resolves a secret mid-run and gets a newer version has silently split its own execution across two credentials. Resolve once at startup. Refresh only on an explicit expiry or an explicit authentication failure, and log the refresh as an event, because a rotation that lands mid-sweep is exactly the fact an investigator will want three days later.

§ IVWorked Example: A Startup That Refuses

The shape below is the whole discipline in one file. A frozen settings object, a merge in precedence order, a validation gate, and a single resolution pass that turns parameter names into values.

from dataclasses import dataclass, field
from pathlib import Path
import os, sys, tomllib
import boto3

@dataclass(frozen=True)
class Settings:
    region: str = "us-east-1"
    param_prefix: str = "/hedronite/dev/"
    max_attempts: int = 3
    concurrency: int = 8
    dry_run: bool = False
    secrets: dict = field(default_factory=dict)

def _from_file(path: Path) -> dict:
    if not path.exists():
        return {}
    return tomllib.loads(path.read_text())

def _from_env(prefix: str = "HEDRONITE_") -> dict:
    out = {}
    for key, value in os.environ.items():
        if key.startswith(prefix):
            out[key[len(prefix):].lower()] = value
    return out

The two readers above return plain dictionaries and nothing else. Neither one raises on a missing source, because absence of a layer is normal and only absence of a required field is an error. That distinction is the reason validation is a separate step rather than a side effect of reading.

The merge and the gate follow. Coercion lives here because this is the only place in the program that knows a string arrived from outside.

def build_settings(cli: dict, config_path: Path) -> Settings:
    merged = {}
    for layer in (_from_file(config_path), _from_env(), cli):
        merged.update({k: v for k, v in layer.items() if v is not None})

    try:
        settings = Settings(
            region=merged.get("region", Settings.region),
            param_prefix=merged.get("param_prefix", Settings.param_prefix),
            max_attempts=int(merged.get("max_attempts", Settings.max_attempts)),
            concurrency=int(merged.get("concurrency", Settings.concurrency)),
            dry_run=str(merged.get("dry_run", "false")).lower() == "true",
        )
    except (TypeError, ValueError) as exc:
        print(f"config: invalid value: {exc}", file=sys.stderr)
        raise SystemExit(2)

    if not settings.param_prefix.endswith("/"):
        print("config: param_prefix must end with '/'", file=sys.stderr)
        raise SystemExit(2)
    return settings

SystemExit(2) is the contract with the scheduler. The 08-02 lesson reserved exit 1 for work that failed and exit 0 for work that succeeded; exit 2 says the run never began, which tells a supervisor to page a human rather than retry a job that will fail identically forever.

Resolution comes last, and it collapses a namespace into a dictionary in one paginated sweep.

def resolve_secrets(settings: Settings) -> dict:
    ssm = boto3.client("ssm", region_name=settings.region)
    paginator = ssm.get_paginator("get_parameters_by_path")
    resolved = {}
    for page in paginator.paginate(
        Path=settings.param_prefix, Recursive=True, WithDecryption=True
    ):
        for param in page["Parameters"]:
            leaf = param["Name"].rsplit("/", 1)[-1]
            resolved[leaf] = param["Value"]
    return resolved

Three properties of that function are worth stating outright. It runs exactly once, at startup, before any work begins. It returns values that are never logged, never written to disk, and never placed back into os.environ where a child process would inherit them. And it fails loudly: an IAM denial or a missing path raises here, at second three of the process, rather than at minute forty.

The startup sequence then reads as four lines with no surprises left in them: build settings, resolve secrets, write the heartbeat that says the run began, do the work.

§ VConnection to Prior Lessons

The 07-30 lesson made the tool installable and gave it entry points. An entry point is a function that runs with no arguments, which means the entry point is precisely the place the four layers get collected. Configuration handling belongs to the console script, not to the library the console script imports, for the same reason logging configuration did: a library that reads the environment is a library that behaves differently depending on who imported it.

The 08-02 lesson made the run leave witnesses. Configuration deepens that record. The first structured log line of any run should name the environment, the parameter prefix, and the count of resolved secrets, with the values themselves absent. That line answers the first question of every incident investigation, which is not what broke but what was it pointed at.

The 07-24 lesson taught the fail-loud-or-degrade decision. Startup is the one phase where the answer is always fail loud. A run that degrades its way past a missing credential is a run that will produce a partial ledger with no explanation in it.

§ VIConnection to Today's Dev Lesson

Today's Dev lesson takes up Python performance: the GIL, the free-threaded build, and the choice between threads, processes, and dropping to C. Configuration is where that choice becomes a setting rather than an assumption. The concurrency field above is the number the Dev lesson teaches you to pick on evidence instead of instinct, and the dry_run flag is the switch that lets you profile the tool's own overhead without touching a production endpoint.

There is a sharper connection. Secrets resolved at startup live in the parent process's memory. A tool that later reaches for multiprocessing inherits that memory through fork on Linux and does not inherit it under spawn, which is the default on macOS and the direction CPython is moving. A configuration discipline that assumes fork semantics will break the day the tool runs on a different platform. Pass the resolved values explicitly to workers, or re-resolve inside each worker with its own cache. The Dev lesson's account of process boundaries is the mechanism behind that rule.

§ VIIClosing

Three sentences carry the discipline. Collect every layer in a declared order and print that order in the help text. Validate the merged view at startup and exit 2 when it does not hold. Resolve secrets once through a service built for the job, from a pointer the configuration carried, and never let the value itself enter a variable the process table can read.

The tool that follows all three has a property worth naming: it either runs correctly or refuses to run. There is no third outcome. The refusing startup is the cheapest failure an ops fleet can buy, because it happens in the first second, costs nothing, and names its own cause.

Examine well the first ten lines your tool executes. Everything after them depends on what those ten lines proved.

Related

• Prior arc: Observable Python Automation — Structured Logging, JSON Lines, the Heartbeat File, and Exit-Code Discipline
• Domain hub: 01-Earth-DevOps
• Grounding tome: Container Security — Liz Rice (Chapter 12, Passing Secrets to Containers, pp. 165-166)
• Paired Dev: Python Performance in Depth
• Paired Cert: AWS DOP-C02 Domain 1 — SDLC Automation