Sitemap

Python — Iterator & Iterable(Basic)

4 min readNov 21, 2022

--

Iterator和Iterable會分成兩篇來寫,這篇先寫Iterable和Iterator的基礎、規則等等,Python — Iterator & Iterable(Iteration tools)則是寫Iterator的延伸工具和套件。

文章簡介

  • Iterator Protocol
  • Iterator v.s Iterable
  • Reversed Iterable
  • Iterator的應用 — Cyclic Iterator Grouping
  • 補充 — Lazy Evaluation and Lazy Iterator

Iterator Protocol

進入iterator前,先知道三個implement在Iterator protocol的methods的作用:

  • .__iter__(): call the __iter__() function,return一個可供迭代的Iterator
  • .__next__(): call the __next() function直到StopIteration Error觸發
  • .__getitem__(): 若是Sequence type的物件,本身已有這個method的話,則可視為Iterator被迭代

Basic Iterator

先看程式碼實現一般Iterator:

class Square_Iterator:
def __init__(self, length):
self.length = length
self.i = 0
def __next__(self):
if self.i >= self.length:
raise StopIteration
else:
sqrt = self.i ** 2
self.i = self.i + 1
return sqrt
def __iter__(self):
return self

Square_iter_obj = Square_Iterator(5)
for square_result in Square_iter_obj:
print(square_result)

>> 0
>> 1
>> 4
>> 9
>> 16

Square_iter_obj是一個平方Iterator,隨著i的增加,return對應的平方結果,接著我們使用for迴圈來迭代,在Python for迴圈做的事情是:
1. call Square_iter_obj.__iter__() and get an Iterator(self)

2. call Square_iter_obj.__next__()直到StopIteration被觸發

Sequence Iteratation

實現Sequence type Iteration:

#Example: Here is a sequence witout iter function
class Square_Sequence:
def __init__(self, value):
self.value = value
def __len__(self):
return self.value
def __getitem__(self, i):
if isinstance(i, int):
if i < 0:
i = self.value + i #Handle [-1] index slicing
if i >= self.value:
raise IndexError
else:
return i**2

#We still be able to iter(the sequence since it has __getitem__)
Square_sequence_obj = Square_Sequence(5)
for square_result in Square_sequence_obj:
print(square_result)

>> 0
>> 1
>> 4
>> 9
>> 16

這是一個非常方便的built-in機制,Python在迭代Sequence時,會自動去增加__getitem__()的值,並且直到IndexError觸發為止,而實際上檯面下Python做的事情是:

#Underneath: Python did this for you
class Sequenceiterator:
def __init__(self, length):
self.i = 0
self.length = length
def __iter__(self):
return self
def __next__(self):
if self.i >= self.length:
raise IndexError
else:
result = self.i **2
self.i += 1
return result

Square_iter_obj = Sequenceiterator(5)
for square_result in Square_iter_obj:
print(square_result)

>> 0
>> 1
>> 4
>> 9
>> 16
>> IndexError

Iterator v.s Iterable

跑完了Iterator一遍會發現一個問題就是我們每個Iterator object是用attribute self.i去記目前迭代到哪,並檢查有沒有到達Stop Iteration condition。衍伸而來的問題就是Iterator在跑完一個for迴圈後,就無法被重複使用,因為self.i並沒有被重置:

#Iterator 1st time
for square_result in Square_iter_obj:
print(square_result)

>> 0
>> 1
>> 4
>> 9
>> 16

#Iterate 2nd time
for square_result in Square_iter_obj:
print(square_result)

>> no result

要解決這個問題,我們可以把Iterator implement成Iterable,這也是Iterator和Iterable的主要差異,Iterable能被重複迭代,Iterator被迭代一次後就會expire。
Implement Iterable程式碼:

class Square_Iterable:
def __init__(self, length):
self.length = length
self.i = 0
def __len__(self):
return len(self.length)
def __iter__(self):
return Square_Iterator(self) #Create a new iterator


class Square_Iterator:
def __init__(self, square_obj): #from Square_Iterable
self.square_obj = square_obj
self.i = 0
def __next__(self):
if self.i >= self.square_obj.length:
raise StopIteration
else:
sqrt = self.i ** 2
self.i = self.i + 1
return sqrt
def __iter__(self):
return self


square_iterable_obj = Square_Iterable()

#Iterate 1st time
for square_result in square_iterable_obj:
print(square_result)

#Iterate 2nd time
for sequence_result in square_iterable_obj:
print(square_result)

比較Iterator和Iterable我們會發現幾個重要的差異,並且各自的功用不一樣

  • Iterable每次都return一個新的Iterator。
  • Iterator每次return自己self,只能被迭代一次。
  • 只有Iterator需要implement __next__(),因為Iterator是實際被迭代的物件。

這就是為什麼在處理Python built-in的迭代物件時,有些可以被重複迭代、而有些只能被迭代一次,我們必須先去了解built-in function return的是Iterator還是Iterable,舉幾個例子:

zip(seq1, seq2) -> iterator
enumerate(seq1) - > iterator
open('test.csv') -> iterator
range(10) - > iterable
dictionary.keys() -> iterable
dictionary.values() -> iterable
dictionary.items() -> iterable

Reversed Iterable

這部分主要介紹.__reversed__() method,透過Iterable的概念來應用。

Get John’s stories in your inbox

Join Medium for free to get updates from this writer.

Target是希望能夠implement reversed() method在Iterable中,讓迭代時能夠雙向都進行迭代,程式碼的部分其實不會太難,只需要在.__reversed__被呼叫時去return一個Iterator,並且傳入reverse=True,接著我們在寫Iterator時,去根據reverse=True/False去做相對應的處理即可:

範例:融合Namedtuple,去做雙向的迭代

from collections import namedtuple
FULL_NAME = namedtuple("FULL_NAME", "first_name, second_name")
first_name = ['John', 'Mike', 'Ki']
second_name = ['Chen', 'Liu', 'Chi']

class NAME_Iterable:
def __init__(self, first_name, second_name):
self.first_name = first_name
self.second_name = second_name
self.length = len(first_name)
def __iter__(self):
return NAME_Iterator(self)
#Adding __reversed__ for reversed order: pass in reverse=True
def __reversed__(self):
return NAME_Iterator(self, reverse=True)


#Adding reverse parameter to handling reversed order, by default is False
class NAME_Iterator:
def __init__(self, NAME, reverse=False):
self.first_name = NAME.first_name
self.second_name = NAME.second_name
self.length = NAME.length
self.i = 0
self.reverse = reverse
def __iter__(self):
return self
def __next__(self):
if self.i >= self.length:
raise StopIteration
else:
if self.reverse == True:
if self.i == 0-self.length:
raise StopIteration
else:
self.i = self.i - 1
result = FULL_NAME(self.first_name[self.i], self.second_name[self.i])
else:
result = FULL_NAME(self.first_name[self.i], self.second_name[self.i])
self.i += 1

return result

name_iterable = NAME_Iterable(first_name, second_name)
reversed_name_iterable = reversed(name_iterable) #return reversed iterator

for name in name_iterable:
print(name)

>> FULL_NAME(first_name='John', second_name='Chen')
>> FULL_NAME(first_name='Mike', second_name='Liu')
>> FULL_NAME(first_name='Ki', second_name='Chi')

for name in reversed_name_iterable:
print(name)

>> FULL_NAME(first_name='Ki', second_name='Chi')
>> FULL_NAME(first_name='Mike', second_name='Liu')
>> FULL_NAME(first_name='John', second_name='Chen')

Reversed Sequence Iteration

若是要反向迭代的物件本身是Sequence type的話,則不需要額外再implement.__reversed__ method,Python會看Sequence中的.__len__去做運算:

from collections import namedtuple

FULL_NAME = namedtuple("FULL_NAME", "first_name, second_name")
first_name = ['John', 'Mike', 'Ki']
second_name = ['Chen', 'Liu', 'Chi']


class NAME_Sequence:
def __init__(self, first_name, second_name):
self.first_name = first_name
self.second_name = second_name
self.length = len(first_name)

def __len__(self):
return self.length

def __getitem__(self, i):
if i >= self.length:
raise IndexError
else:
result = FULL_NAME(self.first_name[i], self.second_name[i])
return result


name_iterable = NAME_Sequence(first_name, second_name)
reversed_name_iterable = reversed(name_iterable)

for name in reversed_name_iterable:
print(name)

>> FULL_NAME(first_name='Ki', second_name='Chi')
>> FULL_NAME(first_name='Mike', second_name='Liu')
>> FULL_NAME(first_name='John', second_name='Chen')

Iterator的應用 — Cyclic Iterator Grouping

Cyclic Iterator概念是創建一個循環的Iterator,在這個Iterator中,不特別去寫StopIteration condition,而持續地讓Iterator去迭代產出資料,假設有以下情境我們想要解決:

10位學生,依序把這10位學生分入GroupA, GroupB, GroupC, GroupD四個組,可以透過以下的Cyclic Iterator來完成:

#Cyclic iterators
#John, Jonathon, Charlie, Penny, Patrick, Scott, Pat, Sherry, Shane, Josh
#GroupA, GroupB, GroupC, GroupD
#-> outcome GroupA_John, GroupB_Jonathon, GroupC_Charlie, GroupD_Penny, GroupA_Patrick, GroupB_Scott, GroupC_Pat, GroupD_Sherry...
class CyclicIterator:
def __init__(self, groups): #groups = [GroupA, GroupB, GroupC, GroupD]
self.groups = groups
self.i = 0
def __iter__(self):
return self
def __next__(self):
group = self.groups[self.i%len(self.groups)]
self.i += 1
return group

members = ["John", "Jonathon", "Charlie", "Penny", "Patrick", "Scott", "Pat", "Sherry", "Shane", "Josh"]
groups_iter = CyclicIterator(["GroupA", "GroupB", "GroupC", "GroupD"])
result = list(zip(members, groups_iter)) #Would expired base on shorter iterable
final_result = [str(x[1]) +"_" + x[0] for x in result]
print(final_result)

>>['GroupA_John',
'GroupB_Jonathon',
'GroupC_Charlie',
'GroupD_Penny',
'GroupA_Patrick',
'GroupB_Scott',
'GroupC_Pat',
'GroupD_Sherry',
'GroupA_Shane',
'GroupB_Josh']

以上程式碼,我們定義CyclicIterator的class後,使用zip()去結合兩個Iterators,因為預設情況下,zip() function會根據評估,根據arguments中較短的iterator去結束迭代,並產生新的iteratable,在這個case,我們兩個傳入的arguments分別為10個人名以及一個infinite的Cyclic iterator,所以最終結果會是長度為10的iterable。

補充 — Lazy Evaluation and Lazy Iterator

之前的文章就有不斷使用到lazy iterator,而作為補充是因為lazy Iterator有個大概念來自lazy evaluation,剛好這邊使用lazy iterator的implement來做為一個範例闡述。

Lazy Evaluation

lazy evaluation定義:

  • properties only be populated when it is called(and only the first time), properties not being populated in __init__
  • properties been populated when change/request happened, otherwise just using the existing value/cache

範例:

class Actor:
def __init__(self, actor_id):
self.actor_id = actor_id
self.bio = lookup_function_actor_in_db(actor_id)
self.movies = None #Not populated in __init__
@property
def movies(self):
if self.movies == None: #Only evaluated in the first time
self.movies = lookup_function_movies_in_db(self.actor_id)
return self.movies

從程式碼明白兩件事:

  1. 當class Actor物件被創造時,並不會馬上運算查找並對self.movies賦值,而是我們call .movies(),才會進行lookup查找的運算
  2. 若之前已經call過 .movies,則不會再執行lookup,會使用之前的結果

lazy evaluation解決的問題是避免電腦做過多不必要的運算,只有在我們必要時才做運算。

Lazy Iterator

如其名,Lazy Iterator做的事情並不會在Iterator建立時就把所有的值計算出來,只有在迭代到這個值的時候才會去做計算,以Factorial Iterable為例:

import math
class Factorials:#iterable
def __init__(self, length):
self.length = length
def __iter__(self): #generate iterator
return FacIter(self.length)

class FacIter: #iterator for Factorials
def __init__(self, length):
self.length = length
self.i = 0
def __iter__(self):
return self
def __next__(self):
if self.i >= self.length:
raise StopIteration
else:
result = math.factorial(self.i)
self.i += 1
return result

factorial_example_iterable = Factorials(5)
print(factorial_example_iterable) #would not display full iterable cuz it use lazy evaluation so it would only show it when you iterate it
for f in factorial_example_iterable:
print(f)

>> 1 #for 0
>> 1
>> 2
>> 6
>> 24

--

--