Python Generators

A generator is a function that produces a sequence of values lazily, one at a time, using yield. Generators make iterators easy to write and let you process huge or endless data without loading it all into memory.

Your first generator

A function that contains yield is a generator function. Calling it does not run the body; it returns a generator object. Each next() runs the body until the next yield.

def countdown(n):    while n > 0:        yield n        n -= 1 gen = countdown(3)print(gen)print(next(gen), next(gen), next(gen))print(list(countdown(5)))

<generator object countdown at 0x000001D0A798A800>
3 2 1
[5, 4, 3, 2, 1]

Compare with the iterator class in the Iterators lesson: the same behaviour in four lines.

yield pauses, it does not end

return ends a function. yield hands out a value and pauses, keeping all local variables, until the next request.

def steps():    print("start")    yield 1    print("after first yield")    yield 2    print("finished") for value in steps():    print("got", value)

start
got 1
after first yield
got 2
finished

Why generators save memory

A list holds every item at once. A generator holds only its current state.

import sys numbers_list = [n * n for n in range(1_000_000)]numbers_gen = (n * n for n in range(1_000_000)) print(sys.getsizeof(numbers_list) > 1_000_000)print(sys.getsizeof(numbers_gen) < 1_000)print(sum(numbers_gen))

True
True
333332833333500000

Generator expressions

Round brackets around a comprehension make a generator expression. When it is the only argument to a function, you can drop the extra brackets:

words = ["python", "generators", "save", "memory"]print(max(len(w) for w in words))print(any(w.startswith("g") for w in words))lengths = (len(w) for w in words)print(next(lengths), list(lengths))

10
True
6 [10, 4, 6]

Infinite generators

Because values are produced on demand, a generator can run forever. Take only what you need:

def fibonacci():    a, b = 0, 1    while True:        yield a        a, b = b, a + b fib = fibonacci()print([next(fib) for _ in range(10)])

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Generator pipelines

Chain generators so each one processes the output of the previous one. Only one item flows through the pipeline at a time, which is how large log files are processed efficiently.

log_lines = [    "INFO start",    "ERROR db timeout",    "INFO request ok",    "ERROR disk full",] def errors(lines):    for line in lines:        if line.startswith("ERROR"):            yield line def messages(lines):    for line in lines:        yield line.split(" ", 1)[1] for msg in messages(errors(log_lines)):    print(msg)

db timeout
disk full

yield from

yield from hands out every item of another iterable, which is perfect for recursive generators:

def flatten(items):    for item in items:        if isinstance(item, list):            yield from flatten(item)        else:            yield item print(list(flatten([1, [2, [3, 4]], 5])))

[1, 2, 3, 4, 5]

Generators are single-use

gen = (x for x in range(3))print(list(gen))print(list(gen))

[0, 1, 2]
[]

Call the generator function again (or rebuild the expression) when you need the values a second time.

Check your understanding

def g():    for i in range(3):        yield i * 10    yield "done" it = g()print(next(it), next(it), list(it))

0 10 [20, 'done']

chunks(list(range(1, 11)), 4)

[1, 2, 3, 4]
[5, 6, 7, 8]
[9, 10]

def chunks(items, size):    for start in range(0, len(items), size):        yield items[start:start + size] for part in chunks(list(range(1, 11)), 4):    print(part)

[1, 2, 3, 4]
[5, 6, 7, 8]
[9, 10]

Program Practice

Python generators FAQ