Bounded Concurrency for Python Ops Tools asyncio Semaphores, the Connection-Pool Ceiling, and Fan-Out Across Azure Resource Manager

One client's manners are a property of the client. A thousand coroutines' manners are a property of a number nobody set.

One client's manners are a property of the client. A thousand coroutines' manners are a property of a number nobody set.

§I — Frame

Saturday's lesson built a client that knows when to stop. Classify the error, jitter the backoff, cap the retries against a budget, break the circuit when the server is clearly sick. It closed on a distinction and then walked away from it: a client can be polite per-call while the fleet it belongs to is impolite in aggregate.

Today the fleet is not a fleet of machines. It is one process.

The tool is an inventory sweep. Four hundred subscriptions under a management group, and the operator wants a resource count and a tag audit for every resource group in every one of them. Written synchronously it takes forty minutes and nobody runs it. So it gets rewritten:

results = await asyncio.gather(*[fetch_rgs(sub) for sub in subscriptions])

Four hundred coroutines. Each of them well-mannered, each carrying Saturday's retry logic, each honoring its own budget. The sweep completes in nine seconds on the first run against a test tenant of six subscriptions, and then somebody points it at production, and Azure Resource Manager starts returning 429 on everything, and the retry logic each of the four hundred is carrying dutifully backs off and comes back, and the sweep neither finishes nor fails.

Name what went wrong. It is not the retries. Every retry was correct in isolation. What is missing is a number: the ceiling nobody set.

§II — Foundations: four facts about fan-out

**Fact one. gather is a starting gun, not a scheduler.**

This is the single most common misreading of asyncio among operators coming from thread pools. A ThreadPoolExecutor(max_workers=8) has a bound written into its constructor; hand it a thousand jobs and eight run. asyncio.gather has no such argument. Hand it a thousand coroutines and it wraps every one in a Task and schedules every one immediately. There is no queue. The concurrency is whatever you passed in.

Ramalho puts the mechanism plainly in the asyncio chapter: asyncio.gather accepts one or more awaitables and returns when all have completed, and every awaitable handed to it is scheduled on the loop. The bound has to come from somewhere else, because it does not come from gather.

Fact two. There are always three ceilings, and the lowest one wins.

Every fan-out has three limits on how many requests are in flight. The first is the one the author sets deliberately, a semaphore or a worker count. The second is the HTTP connector's connection pool, which in aiohttp defaults to one hundred total and, more consequentially, to a per-host cap. The third is the server's throttle, which the author does not control and often has not read.

If the author sets only the first, the other two still exist. They just get discovered by accident, in production, as latency that has no explanation in the code. A semaphore of two hundred sitting on top of a connector pool of one hundred is a semaphore of one hundred, and the second hundred coroutines are not idle-waiting politely; they are blocked inside the connector, holding their permits, invisible to every metric the tool emits.

Coin it: three ceilings, and the lowest one wins. Setting one and ignoring two is how a tool acquires a performance characteristic its author cannot explain.

Fact three. The server publishes its ceiling, and most tools discard it.

Azure Resource Manager throttles per subscription, per region, and per resource provider, and it does not make the operator guess. On a throttled request it returns 429 with a Retry-After header giving the wait in seconds. On successful requests it returns a remaining-budget header, x-ms-ratelimit-remaining-subscription-reads for reads, counting down toward the throttle.

That second header is the interesting one. Retry-After tells the tool what to do after it has already been punished. The remaining-reads header tells the tool it is about to be, while there is still time to slow down. A tool that reads only the first is reactive by construction. Saturday's exponential backoff is the correct response to a 429; the remaining-reads header is the chance to never see one.

Fact four. A per-attempt timeout is not a deadline.

aiohttp's ClientTimeout(total=10) bounds one request. It does not bound one logical operation, and once Saturday's retry logic sits underneath it, one logical operation is up to five requests with backoff between them. Five ten-second attempts with jittered sleeps is somewhere near a minute, and four hundred of those running under a semaphore is a sweep that can hang for an hour while every individual component behaves exactly as documented.

The operation needs its own clock. In modern Python that is asyncio.timeout, wrapped around the whole retry loop rather than around any single call inside it.

§III — Mechanism: where the permit goes

The semaphore is the ceiling the author sets, and the whole discipline is a question of scope. Ramalho's treatment builds a downloader whose Semaphore is created once and passed down to every worker, and the critical detail is where the async with sits: around the network call, not around the task.

Put the permit in the wrong place and it does nothing. Two wrong placements are common enough to name.

The first wrong placement wraps task creation:

async with sem:
    tasks = [asyncio.create_task(fetch(s)) for s in subs]
await asyncio.gather(*tasks)

The permit is held while the list comprehension runs, which takes microseconds and bounds nothing. All four hundred tasks are live the moment the block exits.

The second wrong placement wraps the retry loop from the outside while the sleep sits inside:

async with sem:
    for attempt in range(5):
        try:
            return await call()
        except Retryable:
            await asyncio.sleep(backoff(attempt))

This is correct, and it is also the trap Saturday's lesson set without disarming. A retry inside the permit holds the permit. A worker sleeping eight seconds between attempts is occupying one of eight concurrency slots while sending nothing at all. Under a broad throttle, all eight workers end up asleep simultaneously, the tool's effective concurrency drops to zero, and the throughput graph shows a flat line that no error log explains.

Two ways out, and the choice is a real one. Release the permit around the sleep and the tool keeps its slots busy, at the cost of letting more total requests through during a throttle, which is exactly when fewer are wanted. Hold the permit and the tool self-throttles hard, at the cost of stalling. For a control-plane sweep against a service that is already refusing, holding is right. Say so in a comment and move on, because the reader of this code will otherwise assume it was never considered.

The connector is set once, at session construction, and set to agree with the semaphore rather than to fight it:

connector = aiohttp.TCPConnector(limit=32, limit_per_host=16)
timeout = aiohttp.ClientTimeout(total=15, connect=5)
session = aiohttp.ClientSession(connector=connector, timeout=timeout)

The rule is one line: the connector's per-host limit is greater than or equal to the semaphore's permit count. Then the semaphore is the real ceiling and the pool is slack, which is the only arrangement in which the number in the config file means what it says.

§IV — Worked example: the ARM inventory sweep

The sweep runs against Azure Resource Manager. One bearer token from the credential chain, one session, one semaphore, a fan-out over subscriptions, and a throttle reader that slows the tool before Azure does.

The worker holds one permit for the duration of one subscription's read, including its retries, per the decision above:

async def read_subscription(sub_id, session, sem, gov, log):
    url = f"https://management.azure.com/subscriptions/{sub_id}/resourcegroups"
    params = {"api-version": "2021-04-01"}
    async with sem:
        for attempt in range(5):
            await gov.wait()
            async with session.get(url, params=params) as resp:
                gov.observe(resp.headers)
                if resp.status == 429:
                    delay = float(resp.headers.get("Retry-After", "30"))
                    log.warning("throttled", extra={"sub": sub_id, "retry_after": delay})
                    await asyncio.sleep(delay + random.uniform(0, 1))
                    continue
                if resp.status >= 500:
                    await asyncio.sleep(2 ** attempt + random.random())
                    continue
                resp.raise_for_status()
                body = await resp.json()
                return sub_id, body.get("value", [])
    raise RuntimeError(f"exhausted retries for {sub_id}")

Three details in that block are worth naming. The Retry-After value is honored as given rather than replaced by the tool's own backoff curve, because the server's number is information and the client's number is a guess. A small random addition rides on top of it, since four hundred workers told to wait thirty seconds will otherwise return in the same millisecond, which is the jitter argument from Saturday reappearing one layer up. And raise_for_status sits after the two retryable branches so that a 403 from a subscription the service principal cannot read fails immediately instead of consuming five attempts.

The governor is the piece that reads the ceiling Azure publishes:

class Governor:
    def __init__(self, floor=200, pause=20.0):
        self.floor = floor
        self.pause = pause
        self._gate = asyncio.Event()
        self._gate.set()

    def observe(self, headers):
        raw = headers.get("x-ms-ratelimit-remaining-subscription-reads")
        if raw is None:
            return
        if int(raw) < self.floor and self._gate.is_set():
            self._gate.clear()
            asyncio.get_running_loop().call_later(self.pause, self._gate.set)

    async def wait(self):
        await self._gate.wait()

An Event rather than a lock, because the intent is a gate that closes for everyone at once and reopens on a timer. When the remaining-reads count drops below the floor, the gate shuts and every worker parks at gov.wait() before its next attempt. Nothing is cancelled and nothing errors. The sweep simply exhales for twenty seconds. The header is absent on many responses, so the None branch returns rather than defaulting to zero, which would shut the gate permanently on the first uninstrumented reply.

The caller sets the ceiling from configuration rather than from a literal, per the 08-05 discipline:

async def sweep(subs, cfg, log):
    sem = asyncio.Semaphore(cfg.concurrency)
    gov = Governor(floor=cfg.rate_floor, pause=cfg.rate_pause)
    connector = aiohttp.TCPConnector(limit=cfg.concurrency * 2,
                                     limit_per_host=cfg.concurrency)
    timeout = aiohttp.ClientTimeout(total=cfg.request_timeout, connect=5)
    async with aiohttp.ClientSession(connector=connector, timeout=timeout,
                                     headers={"Authorization": f"Bearer {cfg.token}"}) as s:
        async with asyncio.timeout(cfg.sweep_deadline):
            return await asyncio.gather(
                *(read_subscription(x, s, sem, gov, log) for x in subs),
                return_exceptions=True)

return_exceptions=True is the choice that makes this a sweep rather than a transaction. One subscription the principal cannot read should not discard the three hundred and ninety-nine that succeeded. The caller then partitions the results and reports both halves, which is what the 08-02 exit-code discipline needs in order to distinguish a run that found problems from a run that could not execute.

Now the part that matters more than any of the code above.

Four hundred subscriptions and one resource-group listing each is four hundred control-plane reads for a question that Azure Resource Graph answers in one. Resource Graph indexes resources across subscriptions and answers a KQL query against the index, and for inventory and tag-audit work it is the intended surface. The sweep built here is the correct shape for operations that genuinely must touch each subscription's control plane. It is the wrong shape for asking what exists.

The call you do not make cannot be throttled. The best concurrency limit an ops author ever chooses is the one that turns out to be zero.

§V — Connection to prior lessons

Saturday's client that knows when to stop supplied the per-call discipline this lesson bounds in aggregate. The retry budget it introduced was described as the difference between per-call politeness and fleet politeness; inside a single event loop the semaphore is that budget's other half, and the retry-inside-the-permit trap is the seam where the two disciplines touch and can quietly cancel each other.

Wednesday's configuration lesson is why cfg.concurrency is a field and not an eight. A concurrency limit is the most environment-dependent number in an ops tool. It differs between a test tenant and a production tenant with the same code, which is the definition of configuration rather than constant, and it belongs in the frozen dataclass that gets validated at startup.

The 08-02 observability lesson is why the governor logs a throttled event with the subscription and the wait attached. Under fan-out, a log line without a correlating identifier is worse than no line, because four hundred interleaved workers produce a stream in which anything unkeyed is noise.

§VI — Connection to today's Dev lesson

The Dev slot takes gather's successor. asyncio.TaskGroup differs from gather in exactly the dimension this lesson has been circling: what happens to the other workers when one of them fails. gather with return_exceptions=True lets every task run to completion and hands back a mixed list. A TaskGroup cancels its siblings on the first unhandled exception and raises an ExceptionGroup carrying everything that went wrong.

For the inventory sweep, gather is right, and the Dev lesson says why in the language's own terms: the choice between the two is a choice about whether the work is a set of independent errands or one operation with many limbs. It also takes the cancellation semantics that make the async with sem block above safe, because a permit released by a cancelled task is only released if the cancellation propagates the way the language promises.

§VII — Closing

Three numbers govern a fan-out. The permit count you set, the pool size you probably did not, and the throttle the service publishes in a header your client is throwing away. Set the first, make the second agree with it, and read the third.

Then look at the workload once more and ask whether the fan-out is necessary at all. Four hundred well-behaved requests are still four hundred requests.

Open the ops tool you run most often and find where it creates its HTTP session. Read the connector arguments. If there are none, the tool has a concurrency limit of one hundred per host that nobody chose.

Related

• Prior arc: Resilient Python Ops Clients (2026-08-08)
• Domain hub: Cross-References/domains/01-Earth-DevOps
• Grounding tome: Fluent Python, 2nd ed (Asynchronous Programming, pp. 818-827)

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura Filed 2026-08-11 · Fajr anchor · sprint track Python day 20 · seventh Python visit · trio #86