Generator Expressions versus List Comprehensions — the scan that is not the table

Square brackets finish the work. Parentheses hand you a plan.

<!-- hal:authoritative:yaml -->

Square brackets finish the work. Parentheses hand you a plan.

§I — Frame

Saturday named two sleeps. asyncio.sleep yields the loop. time.sleep holds the thread. Coin: the schedule that is not a crontab. Wednesday named try/else and for/else. Coin: the failover that is not a replica. Sunday named repr and field(repr=False). Coin: the credential that is not a key. Those three leftovers still sit in the language. None of them is today's leftover.

Today the Ops lesson names a warehouse scan. A BigQuery table can be 12 TiB. A dry run can report 40 GiB. The difference is partitioning plus a predicate on the partition column. The language has the same split, and it does not need a billing account to show it. A list comprehension walks the iterable now and keeps every result. A generator expression walks when asked and keeps nothing extra.

Name the duty. Coin it: the scan that is not the table.

Ramalho opens Chapter 2 with the pair by name: list comprehensions if the target is a list, generator expressions for other sequences (Fluent Python 2e, Ch.2, pp. 55-57). He will also, later, show the pair as a timing fact rather than a taste fact. Lutz names the syntax: generator expressions are list comprehensions enclosed in parentheses instead of square brackets (Learning Python, Part IV, pp. 548-549). The coin is not the brackets. The coin is when the work runs.

05-18 already taught the iterator protocol as a streaming inference surface. This lesson does not reopen __iter__ / __next__ as the center. 08-11 already taught async generators and __aenter__. This lesson does not reopen cancellation. Today's object is the eager list versus the lazy expression, in sync Python, because that is the prune you can hold in one line.

§II — Language idiom: two comprehensions, one clock

A list comprehension is a loop that builds a list. The loop finishes before the statement does. Any side effect in the iterated object happens now. Any memory for the result is allocated now. If you do not use the list, Ramalho's verdict is blunt: you should not use that syntax. Side-effect listcomps are a tell. Keep them short if you keep them at all.

A generator expression is the same clause in parentheses. Building it does not run the clause. Iterating it does. Lutz: they combine iterators and list comprehensions. The parentheses drop when the expression is the sole argument of a call (sum(x ** 2 for x in range(4))). Extra parentheses return when the call already has them (sorted((x ** 2 for x in range(4)))). That is spelling. The clock is the point.

Lutz's generator function half sits next door. yield suspends. return exits. A generator function is a named cousin of the expression. This lesson uses the expression as the coin, and the function as the witness when a side effect must be visible. You do not need a class with __iter__ to see the prune. You need one function that prints as it yields, and two consumers.

Ramalho's gen_AB demo is that witness (Ch.17, pp. 641-643). A generator function prints start, yields 'A', prints continue, yields 'B', prints end. Feed it to a list comprehension: start, continue, end. print before you ever loop the result. Feed it to a generator expression: the object is a generator. The prints happen as you iterate, interleaved with the values. The listcomp scanned the table. The genexp scanned on demand.

The table, in the language, is the source iterable. The scan is the work done to produce values. They are allowed to differ. When they do not, you paid for every row before you needed the first.

One more spelling fact, because it bites in review. (x for x in items if pred(x)) is a generator expression. [x for x in items if pred(x)] is a list. {x for x in items if pred(x)} is a set, and it is eager. {k: v for k, v in pairs} is a dict, and it is eager. Only the parentheses form is lazy. Curly braces look like they might be. They are not. If the Ops dry run is the honest number, the language's honest number is "did I write square or curly braces." Those two finish the work.

§III — Code worked example

The warehouse analogue is a day-shaped filter over a sequence of events. The Python analogue must not pretend it can skip segments the way BigQuery can. A Python iterable does not have partitions unless you built them. What Python can refuse is the second copy, and the work-before-you-asked.

def gen_events():
    print("open")
    for day, country in (
        ("2025-03-14", "US"),
        ("2025-03-15", "US"),
        ("2025-03-15", "DE"),
        ("2025-03-16", "US"),
    ):
        print("row", day, country)
        yield {"day": day, "country": country}
    print("close")


eager = [row for row in gen_events() if row["day"] == "2025-03-15"]
print("eager built", len(eager))
for row in eager:
    print("use", row["country"])

lazy = (row for row in gen_events() if row["day"] == "2025-03-15")
print("lazy object", type(lazy).__name__)
for row in lazy:
    print("use", row["country"])

Run the eager half. open prints, then every row, then close, then eager built 2, then the two uses. The filter kept two dicts. The generator still walked four. The walk finished before anyone used a country.

Run the lazy half. lazy object generator prints first. Then open, the first row, the second row and a use US, the third row and a use DE, the fourth row, then close. Uses interleave with rows. The walk still visits four source rows, because this source is not partitioned. What you refused is the list. You also refused to run gen_events until the for asked.

That is the honest ceiling of the language prune. BigQuery can skip a segment it never opens. Python can skip building the list. Python cannot skip source rows it has not been taught to skip. If you need that skip, you partition the source yourself: a dict of day to list, a file per day, a BigQuery table. The genexp will not invent segments.

One more rule, because it is the one that makes the eager form look cheaper in a notebook. eager can be iterated twice. lazy cannot. The second for row in lazy is silent. The generator is spent. Lutz's state suspension is the reason: the function already ran to close. If the caller needs two passes, they asked for a table, not a scan. Materialize on purpose. Do not materialize by habit.

def take_us(rows):
    return (row for row in rows if row["country"] == "US")


once = take_us(gen_events())
print(sum(1 for _ in once))
print(sum(1 for _ in once))

The first sum walks and counts. The second sum counts zero. The function returned a scan. The caller treated it as a table. That is SQL B written as a helper: it looks selective, and the second consumer sees nothing.

§III.B — Four more rules the scan will break

Rule one. A genexp is one-shot. Store it, iterate it, and you have a husk. Listcomp survivors can be indexed, sliced, and passed to a second helper. If you needed that, you needed the list. Do not wrap the genexp in list() as a reflex. That is square brackets with extra ink.

Rule two. Side effects run on the consumer's clock. Logging, network, print, a counter increment: in a listcomp they all fire at construction. In a genexp they fire at iteration. If a test asserts the log before anyone iterates, the test was written against a table. Rewrite the test, or keep the list and admit it.

Rule three. Nested genexps do not make partitions. ((x for x in xs if p(x)) for xs in groups) is a scan of scans. It will not skip a group you never iterate. It will also surprise you with late binding if you write for i in range(3) inside and forget the default-arg trick. Keep nested genexps rare. Today's coin is the one-line prune, not a query planner.

**Rule four. any / all / sum / min / max are honest consumers.** They take the genexp as a sole argument and stop when they can. any(row["country"] == "US" for row in gen_events()) can halt after the first US row. A listcomp argument to any already finished the walk. If the Ops dry run is "how many bytes would this read," any on a genexp is the first time the language can read fewer source rows than the table, provided the source itself can stop. A generator function can stop. A materialized list cannot un-walk.

Ramalho's readability claim still holds: the clause is often clearer than map plus filter. The coin is not clarity. The coin is the clock. Write the clause. Then look at the brackets. Then decide whether you asked for a scan or a table.

§III.C — The partition you build yourself

BigQuery's prune is a property of the table. Python's prune is a property of the source you hand the clause. If the source is already a list of a million dicts, both the listcomp and the genexp will visit a million dicts. The genexp only refuses the second million. That is still a win in memory. It is not a win in CPU. Do not tell yourself you partitioned the warehouse by writing parentheses.

If you need a skip, group first, then scan the group.

from collections import defaultdict

def by_day(events):
    groups = defaultdict(list)
    for row in events:
        groups[row["day"]].append(row)
    return groups


groups = by_day(gen_events())
day = groups.get("2025-03-15", ())
us = (row for row in day if row["country"] == "US")

by_day is a real table: it walks once, it keeps the rows, it keys them. After that, us is a scan of one segment. You paid the first walk on purpose, the way a load job pays to land a partitioned table. You did not pay a second full walk to answer a day-shaped question. The helper is honest because it returns a dict, which is a table, and the caller can see the type.

The dishonest cousin is a helper that hides list(...) around a genexp "so it is easier to use." The type says scan. The body made a table. The second consumer works, the memory bill is the table's, and the next reader writes another genexp on top thinking they are still lazy. Print the type. If it is list, say list. If it is generator, say once. The receipt is the only belt Python will fasten for you.

Ramalho's Ch.2 warning about listcomps used for side effects is the same honesty in the other direction. If you are not doing something with the produced list, do not write square brackets. If you are doing something with the produced list twice, do not write parentheses. The coin is the clock. The type is the receipt.

§IV — Connection to today's Ops lesson

The Ops census dry-runs SQL and prints total_bytes_processed next to num_bytes. It never query()s for real. The Dev analogue is the genexp you pass to any or to a single for, and the listcomp you refuse to build "just in case."

SQL that filters country and never names event_date still scans every partition. A listcomp that filters country still walks every source row and keeps the matches. Both look selective. Both paid the full walk of the source they were given. The warehouse can still win, because partitions exist as segments the engine can skip. The language wins only if the source can stop, or if you never needed the kept list.

require_partition_filter is the table demanding the prune column. Python has no such belt on a listcomp. The belt is the review. Square brackets are the False flag. Parentheses are the belt you actually fastened.

Do not call BigQuery from this lesson's examples. The Dev file stays in the language. The pairing is the coin, not a second client.

§V — Prior-lesson reach

08-29 distinguished asyncio.sleep from time.sleep. One wait yields. One wait holds. Today's pair is the same shape at a different clock: one comprehension yields, one comprehension holds the whole result. Do not reopen the event loop. The leftover from 08-29 was a crontab on the instance. The leftover today is a listcomp that thought it was a filter.

08-26 distinguished try/else from a bare except. The else runs when no exception fired, which is the success path, not a second failure. A genexp's second iteration is also not a second success. It is an empty scan. Do not treat silence as a filter that found nothing in the source. It found nothing because the generator was spent.

08-23 hid a secret from repr. The object still held the secret. A listcomp that you print "for debugging" still holds every row you meant to hide from memory. field(repr=False) does not delete. Square brackets do not prune. They copy, then show less, or they copy, then show all.

08-20 taught is versus ==. Identity of the instance is not equality of the name. Identity of the scan is not the name of the table. Keep that split. Do not reopen hashing.

08-14 taught match on the subject, not on the keys. A genexp filters by predicate. It does not destructure. If you need the subject shape, match remains the 08-14 tool. Today's tool is the clock on the clause.

§VI — Closing

Write the clause. Look at the brackets. Square brackets finish the work and keep a table. Parentheses hand you a scan. Iterate a scan once. If you need two passes, you asked for a table. Say so with square brackets, on purpose.

Ramalho's gen_AB still prints end. before the listcomp's first -->. Lutz still suspends at yield. The language sentence is shorter still. The process that listed the events built no second copy.

Examine the next helper that returns a comprehension. If the caller iterates it twice and the second pass is empty, the coin is already spent.

Related

• Prior arc: the schedule that is not a crontab (2026-08-29)
• Language hub: Cross-References/dev-languages/Python
• Grounding: Ramalho — listcomps and genexps, gen_AB eager versus lazy · Lutz — Generator Expressions