Python Generators
A generator is a function that produces a sequence of values lazily, one at a time, using yield. Generators make iterators easy to write and let you process huge or endless data without loading it all into memory.
- Write generator functions with yield
- Understand how a generator pauses and resumes
- Use generator expressions
- Chain generators into pipelines
Your first generator
A function that contains yield is a generator function. Calling it does not run the body; it returns a generator object. Each next() runs the body until the next yield.
def countdown(n): while n > 0: yield n n -= 1 gen = countdown(3)print(gen)print(next(gen), next(gen), next(gen))print(list(countdown(5)))<generator object countdown at 0x000001D0A798A800> 3 2 1 [5, 4, 3, 2, 1]
Compare with the iterator class in the Iterators lesson: the same behaviour in four lines.
yield pauses, it does not end
return ends a function. yield hands out a value and pauses, keeping all local variables, until the next request.
def steps(): print("start") yield 1 print("after first yield") yield 2 print("finished") for value in steps(): print("got", value)start got 1 after first yield got 2 finished
Why generators save memory
A list holds every item at once. A generator holds only its current state.
import sys numbers_list = [n * n for n in range(1_000_000)]numbers_gen = (n * n for n in range(1_000_000)) print(sys.getsizeof(numbers_list) > 1_000_000)print(sys.getsizeof(numbers_gen) < 1_000)print(sum(numbers_gen))True True 333332833333500000
The exact byte sizes depend on the Python version, which is why the example prints comparisons. The list needs megabytes; the generator needs a few hundred bytes whatever the range.
Generator expressions
Round brackets around a comprehension make a generator expression. When it is the only argument to a function, you can drop the extra brackets:
words = ["python", "generators", "save", "memory"]print(max(len(w) for w in words))print(any(w.startswith("g") for w in words))lengths = (len(w) for w in words)print(next(lengths), list(lengths))10 True 6 [10, 4, 6]
Infinite generators
Because values are produced on demand, a generator can run forever. Take only what you need:
def fibonacci(): a, b = 0, 1 while True: yield a a, b = b, a + b fib = fibonacci()print([next(fib) for _ in range(10)])[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
Generator pipelines
Chain generators so each one processes the output of the previous one. Only one item flows through the pipeline at a time, which is how large log files are processed efficiently.
log_lines = [ "INFO start", "ERROR db timeout", "INFO request ok", "ERROR disk full",] def errors(lines): for line in lines: if line.startswith("ERROR"): yield line def messages(lines): for line in lines: yield line.split(" ", 1)[1] for msg in messages(errors(log_lines)): print(msg)db timeout disk full
yield from
yield from hands out every item of another iterable, which is perfect for recursive generators:
def flatten(items): for item in items: if isinstance(item, list): yield from flatten(item) else: yield item print(list(flatten([1, [2, [3, 4]], 5])))[1, 2, 3, 4, 5]
Generators are single-use
gen = (x for x in range(3))print(list(gen))print(list(gen))[0, 1, 2] []
Call the generator function again (or rebuild the expression) when you need the values a second time.
Check your understanding
Predict the output
def g(): for i in range(3): yield i * 10 yield "done" it = g()print(next(it), next(it), list(it))0 10 [20, 'done']
Two next() calls take 0 and 10. list() collects everything left: 20 from the loop and the final “done”.
Quick quiz
1. What does calling a generator function return?
The body starts running only when you ask for the first value.
2. What happens to local variables at a yield?
A generator pauses with its state intact.
3. Which creates a generator expression?
Round brackets make a lazy generator.
Exercise: read in chunks
Write a generator chunks(items, size) that yields lists of size items, with a shorter final chunk if needed.
chunks(list(range(1, 11)), 4)
[1, 2, 3, 4] [5, 6, 7, 8] [9, 10]
Hint 1
Loop with range(0, len(items), size) and yield a slice.
def chunks(items, size): for start in range(0, len(items), size): yield items[start:start + size] for part in chunks(list(range(1, 11)), 4): print(part)[1, 2, 3, 4] [5, 6, 7, 8] [9, 10]
range(0, len(items), size) gives the start of each chunk: 0, 4 and 8. Slicing past the end is safe, so the last chunk simply holds the remaining two items.
Program Practice
20 Python Iterator & Generator Programs
Every program has a step-by-step explanation, colour-coded code and its real output.
Python generators FAQ
What is the difference between yield and return?
return sends back one value and ends the function. yield sends back a value and pauses, so the function can continue and yield more values later.
When should I use a generator instead of a list?
When the data is large, endless, or only needed once in order, such as reading a big file line by line or streaming API results.
Can a generator be reused?
No. Once exhausted it stays empty. Call the generator function again to get a fresh one.