Generators in Python
Generators in Python
What is a Generator?
A generator is a special type of iterator that allows you to iterate over a sequence of values without storing them all in memory at once. Generators generate values on-the-fly (lazy evaluation) using the yield keyword.
Key Characteristics
- Memory efficient: Generate values one at a time
- Lazy evaluation: Values are computed only when needed
- Stateful: Remember their state between calls
- One-time use: Can only be iterated once
Basic Syntax
python
def generator_function():
yield value1
yield value2
yield value3
Simple Examples
Example 1: Basic Generator Function
python
def simple_generator():
yield "Hello"
yield "World"
yield "!"
# Create generator
gen = simple_generator()
# Iterate through values
print(next(gen)) # Output: Hello
print(next(gen)) # Output: World
print(next(gen)) # Output: !
# Next call would raise StopIteration
# print(next(gen))
Example 2: Using Generator in a Loop
python
def countdown(n):
while n > 0:
yield n
n -= 1
# Use generator
for number in countdown(5):
print(number) # Output: 5, 4, 3, 2, 1
Example 3: Number Sequence Generator
python
def number_sequence(limit):
num = 1
while num <= limit:
yield num
num += 1
# Generate numbers up to 5
for num in number_sequence(5):
print(num) # Output: 1, 2, 3, 4, 5
Practical Examples
Example 4: Fibonacci Sequence Generator
python
def fibonacci(limit):
a, b = 0, 1
count = 0
while count < limit:
yield a
a, b = b, a + b
count += 1
# Generate first 7 Fibonacci numbers
for num in fibonacci(7):
print(num) # Output: 0, 1, 1, 2, 3, 5, 8
Example 5: File Processing Generator
python
def read_large_file(filename):
with open(filename, 'r') as file:
for line in file:
yield line.strip()
# Process file line by line without loading entire file
# for line in read_large_file('big_file.txt'):
# process_line(line)
Example 6: Infinite Generator
python
def infinite_counter():
num = 0
while True:
yield num
num += 1
# Use with caution - infinite loop!
counter = infinite_counter()
print(next(counter)) # 0
print(next(counter)) # 1
print(next(counter)) # 2
# ... continues forever
Generator Expressions
Example 7: Generator Expression (like list comprehension)
python
# List comprehension (eager evaluation) squares_list = [x*x for x in range(5)] # [0, 1, 4, 9, 16] # Generator expression (lazy evaluation) squares_gen = (x*x for x in range(5)) # Generator object print(list(squares_gen)) # Convert to list: [0, 1, 4, 9, 16]
Example 8: Using Generator Expressions
python
# Memory efficient for large datasets
large_gen = (x * 2 for x in range(1000000)) # Doesn't create list
# Process one item at a time
for doubled in large_gen:
if doubled > 100:
break
print(doubled)
Advanced Generator Features
Example 9: Generator with send() method
python
def accumulator():
total = 0
while True:
value = yield total
if value is None:
break
total += value
acc = accumulator()
next(acc) # Start generator (returns 0)
print(acc.send(5)) # Output: 5
print(acc.send(10)) # Output: 15
print(acc.send(3)) # Output: 18
Example 10: Generator with Conditional Yielding
python
def even_numbers(limit):
for num in range(limit + 1):
if num % 2 == 0:
yield num
for even in even_numbers(10):
print(even) # Output: 0, 2, 4, 6, 8, 10
Real-World Use Cases
Example 11: Pagination Generator
python
def paginate(items, page_size):
for i in range(0, len(items), page_size):
yield items[i:i + page_size]
data = [1, 2, 3, 4, 5, 6, 7, 8, 9]
for page in paginate(data, 3):
print(page) # [1,2,3], [4,5,6], [7,8,9]
Example 12: Data Processing Pipeline
python
def read_data():
for i in range(10):
yield i
def filter_even(numbers):
for num in numbers:
if num % 2 == 0:
yield num
def square(numbers):
for num in numbers:
yield num * num
# Chain generators together
pipeline = square(filter_even(read_data()))
for result in pipeline:
print(result) # Output: 0, 4, 16, 36, 64
Key Benefits of Generators
- Memory Efficiency: Don’t store all values in memory
- Lazy Evaluation: Compute values only when needed
- Clean Code: Simple syntax for complex iterations
- Composability: Can chain generators together
- State Preservation: Remember state between calls
When to Use Generators
- Processing large files or datasets
- Generating infinite sequences
- Creating data pipelines
- When you need memory efficiency
- For lazy evaluation of expensive computations
Generator vs Normal Function
| Aspect | Normal Function | Generator Function |
|---|---|---|
| Return | return | yield |
| Execution | Runs to completion | Pauses and resumes |
| Memory | Stores all results | Generates on demand |
| State | No state between calls | Remembers state |
| Usage | Multiple calls independent | Single iteration |
Generators are powerful tools for efficient memory usage and creating complex data processing pipelines!
Generator in Python – Interview Q&A
Basic Concept Questions
1. What is a generator in Python?
Answer:
A generator is a special type of iterator that generates values on-the-fly using the yield keyword. It doesn’t store all values in memory at once, making it memory efficient for large datasets.
Example:
python
def simple_generator():
yield 1
yield 2
yield 3
gen = simple_generator()
print(next(gen)) # 1
print(next(gen)) # 2
2. What is the difference between yield and return?
Answer:
return: Terminates function execution and returns a valueyield: Pauses function execution, returns a value, and remembers state for next call
Example:
python
def normal_func():
return 1 # Function ends here
def generator_func():
yield 1 # Pauses here
yield 2 # Resumes here on next call
3. How do generators differ from lists?
Answer:
| Aspect | List | Generator |
|---|---|---|
| Memory | Stores all elements | Generates on demand |
| Speed | Faster access | Slower generation |
| Usage | Multiple iterations | Single iteration |
| Memory Use | High for large data | Low for large data |
Example:
python
# List - all 1M numbers in memory list_nums = [x for x in range(1000000)] # Generator - generates numbers on demand gen_nums = (x for x in range(1000000))
Technical Questions
4. What happens when a generator function is called?
Answer:
Calling a generator function returns a generator object without executing the function body. Execution only happens when next() is called.
Example:
python
def demo_generator():
print("Starting")
yield 1
print("Continuing")
yield 2
gen = demo_generator() # Nothing printed yet
print("Generator created")
print(next(gen)) # "Starting" then 1
print(next(gen)) # "Continuing" then 2
5. How can you create a generator without a function?
Answer:
Using generator expressions (similar to list comprehensions but with parentheses).
Example:
python
# List comprehension squares_list = [x*x for x in range(5)] # [0, 1, 4, 9, 16] # Generator expression squares_gen = (x*x for x in range(5)) # Generator object print(list(squares_gen)) # [0, 1, 4, 9, 16]
6. What is the StopIteration exception in generators?
Answer:StopIteration is automatically raised when a generator has no more values to yield. For-loops catch this exception to terminate.
Example:
python
def simple_gen():
yield 1
gen = simple_gen()
print(next(gen)) # 1
print(next(gen)) # Raises StopIteration
Implementation Questions
7. How would you create a Fibonacci generator?
Answer:
Example:
python
def fibonacci(limit):
a, b = 0, 1
count = 0
while count < limit:
yield a
a, b = b, a + b
count += 1
for num in fibonacci(8):
print(num) # 0, 1, 1, 2, 3, 5, 8, 13
8. How can you send data back to a generator?
Answer:
Using the send() method to pass values back into the generator.
Example:
python
def accumulator():
total = 0
while True:
value = yield total
if value is None:
break
total += value
acc = accumulator()
next(acc) # Initialize (returns 0)
print(acc.send(5)) # 5
print(acc.send(10)) # 15
Memory and Performance Questions
9. Why are generators memory efficient?
Answer:
Generators produce values one at a time on demand, unlike lists that store all values in memory simultaneously.
Example:
python
# This would use massive memory: # big_list = [x for x in range(1000000000)] # This uses minimal memory: big_gen = (x for x in range(1000000000))
10. When should you use generators over lists?
Answer:
Use generators when:
- Processing large datasets that don’t fit in memory
- You only need to iterate once
- You want lazy evaluation
- Working with data pipelines
Use lists when:
- You need random access to elements
- You need to iterate multiple times
- The dataset fits comfortably in memory
Advanced Questions
11. How do generators help with infinite sequences?
Answer:
Generators can produce infinite sequences without memory issues since they generate values on demand.
Example:
python
def infinite_counter():
num = 0
while True:
yield num
num += 1
counter = infinite_counter()
for i in range(5):
print(next(counter)) # 0, 1, 2, 3, 4
12. How can you chain generators together?
Answer:
Generators can be composed to create data processing pipelines.
Example:
python
def numbers():
for i in range(10):
yield i
def evens(seq):
for num in seq:
if num % 2 == 0:
yield num
def squares(seq):
for num in seq:
yield num * num
# Chain generators
pipeline = squares(evens(numbers()))
print(list(pipeline)) # [0, 4, 16, 36, 64]
13. What is yield from and how is it used?
Answer:yield from delegates generation to another generator or iterable.
Example:
python
def first_gen():
yield 1
yield 2
def second_gen():
yield from first_gen()
yield 3
yield 4
print(list(second_gen())) # [1, 2, 3, 4]
Real-World Scenario Questions
14. How would you use generators for large file processing?
Answer:
Example:
python
def process_large_file(filename):
with open(filename, 'r') as file:
for line in file:
# Process one line at a time
processed_line = line.strip().upper()
yield processed_line
# Memory efficient for huge files
for line in process_large_file('giant_file.txt'):
print(line)
15. How can generators be used for pagination?
Answer:
Example:
python
def paginate(items, page_size):
for i in range(0, len(items), page_size):
yield items[i:i + page_size]
data = list(range(20)) # [0, 1, 2, ..., 19]
for page in paginate(data, 5):
print(page) # [0,1,2,3,4], [5,6,7,8,9], etc.
Best Practices Questions
16. What are the limitations of generators?
Answer:
- Single use: Can only be iterated once
- No random access: Can’t access elements by index
- Stateful: Hard to debug due to paused state
- Not reusable: Need to recreate for second iteration
17. How can you convert a generator to a list?
Answer:
Use the list() constructor.
Example:
python
def number_gen():
yield from range(5)
gen = number_gen()
numbers_list = list(gen) # [0, 1, 2, 3, 4]
18. What’s the difference between a generator and an iterator?
Answer:
All generators are iterators, but not all iterators are generators. Generators are created using functions with yield, while iterators can be any object with __iter__() and __next__() methods.
Error Handling Questions
19. How do you handle errors in generators?
Answer:
Use try-except blocks within the generator function.
Example:
python
def safe_divide(numbers, divisor):
for num in numbers:
try:
yield num / divisor
except ZeroDivisionError:
yield "Error: Division by zero"
nums = [10, 20, 30, 40]
result = list(safe_divide(nums, 0))
print(result) # ['Error: Division by zero', ...]
20. How can you close a generator early?
Answer:
Use the close() method to terminate a generator.
Example:
python
def infinite_gen():
try:
num = 0
while True:
yield num
num += 1
except GeneratorExit:
print("Generator closed")
gen = infinite_gen()
print(next(gen)) # 0
print(next(gen)) # 1
gen.close() # "Generator closed"
