Demystifying Python Generators
In this article we will take a closer look at Python Generators. In my previous article I tried to experiment with Python Iterator object.
Please refer — https://medium.com/@basusaptarshi89/python-difference-between-iterable-and-iterator-0a519812951a
Encouraged by my dear colleague Vaibhav (Vaibhav Maurya), this is a humble attempt to demystify the Python Generator construct.
It looks like a normal function, but is it really?
Let’s start with a code example. In the following section, I have defined two functions — the only difference is that, in my_special_function I have used “yield” instead of “return”.
def my_simple_function():
return 1
def my_special_function():
yield 1The yield keyword makes my_special_function a little bit special.
The invocation of my_simple_function will simply return the integer 1, however, it is not the same for my_special_function.
return_value1 = my_simple_function()
return_value2 = my_special_func()
print(f"my_simple_function returns {return_value1}, which is of type {type(return_value1)}")
print(f"my_special_func returns {return_value2}, which is of type {type(return_value2)}")
::OUTPUT::
my_simple_function returns 1, which is of type <class 'int'>
my_special_func returns <generator object my_special_func at 0x0000020FF93D1C80>, which is of type <class 'generator'>The invocation of my_special_function doesn’t return 1, instead it returns something called a ‘generator’ object.
A closer look
Let’s take a closer look at this generator object, which was returned by invoking my_special_function
my_generator_object = my_special_func()
dir(my_generator_object)
::OUTPUT::
['__class__',
'__del__',
'__delattr__',
'__dir__',
'__doc__',
'__eq__',
'__format__',
'__ge__',
'__getattribute__',
'__gt__',
'__hash__',
'__init__',
'__init_subclass__',
'__iter__',
'__le__',
'__lt__',
'__name__',
'__ne__',
'__new__',
'__next__',
'__qualname__',
'__reduce__',
'__reduce_ex__',
'__repr__',
'__setattr__',
'__sizeof__',
'__str__',
'__subclasshook__',
'close',
'gi_code',
'gi_frame',
'gi_running',
'gi_yieldfrom',
'send',
'throw']Calling the built-in dir method on the generator object reveals that it contains the __next__ (dunder next) and __iter__ (dunder iter) methods. So technically I should be able to call “next” on this object and use it in Python ‘for loop’ construct — same as an Iterator.
print(next(my_generator_object))
print(next(my_generator_object))
::OUTPUT::
1
---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
<ipython-input-29-b0067deff613> in <module>
1 print(next(my_generator_object))
----> 2 print(next(my_generator_object))
StopIteration: As can be seen from above code example, calling the next method on the generator returns 1, and then it raises StopIteration exception.
Get Saptarshi Basu’s stories in your inbox
Join Medium for free to get updates from this writer.
The magic keywork yield
The yield statement suspends the execution flow of the function. Let’s make a small change to our functions and try to understand this concept.
def my_simple_function():
return 1
print("This line will never be printed")
def my_special_func():
yield 1
print("This line will be printed")The function my_simple_function, now has two statements.
As we can easily identify, the print statement inside this function is un-reachable. Python will not throw any error, however, the program execution will never reach the print statement inside the function.
However, the yield statement behaves differently. Upon reaching the yield statement in my_special_function, the program execution will be suspended. It can be proved by calling the next method twice on my_special_function
The first call to the next method returns 1, and the execution is suspended. The second call to next method executes the print statement and throws StopIteration exception.
Let’s modify my_special_function to understand this clearly.
We have three yield statements inside the function. Each time we call the next method, the value is returned and the execution gets suspended (or paused) at that point.
def my_special_func():
print("This line will be printed on first call of 'next'")
yield 1
print("This line will be printed on second call of 'next'")
yield 2
print("This line will be printed on third call of 'next'")
yield 3
# invoke the function to create generator object
my_generator_object = my_special_func()
# invoke 'next' on generator
print(next(my_generator_object))
print(f'Completed first call\n')
print(next(my_generator_object))
print(f'Completed second call\n')
print(next(my_generator_object))
print(f'Completed third call\n')
print(next(my_generator_object))
print(f'Completed fourth call\n')
::OUTPUT::
This line will be printed on first call of 'next'
1
Completed first call
This line will be printed on second call of 'next'
2
Completed second call
This line will be printed on third call of 'next'
3
Completed third call
---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
<ipython-input-46-163f50b09c27> in <module>
20 print(f'Completed third call\n')
21
---> 22 print(next(my_generator_object))
23 print(f'Completed fourth call\n')
StopIteration: A better generator
Let’s try to create some meaningful generator and use it in Python for loop.
In the following example we create arithmatic progression series generator.
def arithmatic_progression(first_term=1, common_difference=1):
n = 0
while True:
n += 1
nth_term = first_term + (n-1)*common_difference
yield nth_term
ap_series_generator = arithmatic_progression(10, 5)
for i in ap_series_generator:
print(i)
::OUTPUT::
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95
100
105
110
115
120
...
...
...
(continued infinitely)Why generators?
The Python generator syntax can be considers a syntactic sugar for defining Iterators. These objects are technically different, however, it is easier to define a generator with few lines of code as opposed to a full-fledged definition of Iterator class.
Use generators to save memory. The next value if calculated in-time, therefore you don’t need to store large sequences in memory.
