Sitemap
John

John

NUS Singapore, from Taiwan, https://github.com/YCtw

Python — Iterator & Iterable(Iteration tools)

16 min readNov 25, 2022

--

接續Iterator & Iterable(Basic),這篇紀錄幾個Iteration tools套件有用的function,作為額外的學習。

文章簡介

  • Predicate Functions(filter, compress, takewhile, dropwhile)
  • Infinite Iteration Functions(count, cycle)
  • repeat
  • chain
  • tee
  • starmap
  • accumulate
  • zip longest
  • groupby
  • product

Predicate Functions(filterfalse, compress, takewhile, dropwhile)

Predicate中文是「斷言、判斷」,用在程式所指用來是判斷一個statement/condition的truth value,而Predicate functions可以讓我們傳入Iterable和一個function,將前者迭代的物件一個一個送入function中,最終return經過判斷後的結果。

filterfalse

filterfalse(predicate_func, iterable) -> return lazy iterator

關於一般filter function,在High Order Functions已經寫過了,filterfalse就是反向的把Iterable中truth value是false的物件留下,作為return的結果。

from itertools import filterfalse

iter_a = filterfalse(lambda x: x<4, [1,10,2,9,3,8])
print(list(iter_a)) #[10, 9 ,8]

compress

compress(data_iterable1, selector_iterable) -> return lazy iterator

compress function接收兩個arguments,並把兩個Iterable compress起來,用第二個Iterable的truth value值去對應第一個Iterable同index的資料,最後留下value=true的資料。

from itertools import compress

data = ['a', 'b', 'c', 'd', 'e']
selectors = [True, False, 1, 0] #e沒有配對,自動配對為None
compress_iter = compress(data, selectors) #compress(return truth value=true lazy iterator)
print(list(compress_iter)) #['a','c']

takewhile

takewhile(predicate_func, iterable) -> return lazy iterator

takewhile function做的事情是從傳入的Iterable的首項開始,把迭代物件一一丟入predicate_func中,直到predicate_func回傳的truth value為false。

from itertools import takewhile

list_a = [1,3,5,2,1]
iter_a = takewhile(lambda x: x<5, list_a) #return a lazy iterator yield the value till x<5 is falsy
print(list(iter_a)) #[3,5]

dropwhile

dropwhile(predicate_func, iterable) -> return lazy iterator

dropfile和takewhile一樣從Iterable的首項開始,但留下的Iterator 是takewhile的反面,會捨棄掉物件直到第一個falsy產生,才會return剩下的物件。

#dropwhile - return a lazy iterator after falsy value, it would drop previous truthy value
from itertools import dropwhile

list_a = [1,3,5,2,1]
iter_a = dropwhile(lambda x: x<5, d_list) #return a lazy iterator yield the value after falsy happened(5)
print(list(iter_a)) #[5,2,1]

Infinite Iteration Functions(count, cycle, repeat)

Infinite Iteration Functions顧名思義就是return回來的Iterator並沒有明確的StopIteration condition,因此可以無盡的迭代下去。

count

count(start, step) -> return an infinite lazy iterator

count相較一般的range(start, stop, step) Iterator裡arguments只能傳入整數,count function的arguments能傳入任何numeric type data(int, float, complex…)。

因為function出來的Iterator是無盡迭代下去,所以在使用時我們可以搭配前面說到的takewhile或是islice一起使用:

from itertools import count
from decimal import Decimal
from itertools import takewhile

iter_count = count(Decimal('10.5'), Decimal('0.1')) #10.5, 10.6, 10.7...
iter_takewhile = takewhile(lambda x: x<11.1, iter_count)
print(list(iter_takewhile)) #[Decimal('10.5'), Decimal('10.6'), Decimal('10.7'), Decimal('10.8'), Decimal('10.9'), Decimal('11.0')]

cycle

cycle(iterable) -> return infinite lazy iterator

cycle function會不斷循環傳入的iterable,無盡地迭代下去:

from itertools import cycle

cycle_iter = cycle("spam") #-> s, p, a, m
for _ in range(10):
print(next(cycle_iter)) #-> s, p, a, m, s, p, a, m, s, p

repeat

repeat(iterable, n) -> return lazy iterator

Get John’s stories in your inbox

Join Medium for free to get updates from this writer.

repeat Iterable n times

from itertools import repeat

repeat_iter = repeat([1,2,3,4,5], 3)
print(list(repeat_iter)) #[[1,2,3,4,5],[1,2,3,4,5],[1,2,3,4,5]]

Warning of repeat()

因為repeat() function是拿最一開始傳入的Iterable物件來使用,類似copy()中的shallow copy,所有的follow up repeat物件都是指向和同一個Iterable一樣的記憶體位置,所以若repeat的argument是mutable物件型別,在修改時我們必須特別注意,避免以下情況:

r1 = repeat([], 4)
list_r1 = list(r1)
list_r1[0].append(10) #其他三個repeat來的空list裡面也會有10,因為4個repeat empty lists都指向同個memory address
print(list_r1) [[10],[10],[10],[10]]

chain

chain(*args) -> return lazy iterator

把多個Iterable串起來,形成一個lazy iterator

from itertools import chain

iterable_1 = [1,2,3]
iterable_2 = [4,5,6]
iterable_3 = [7,8,9]
chain_iterator = chain(iterable_1, iterable_2, iterable_3)

print(list(chain_iterator)) #[1,2,3,4,5,6,7,8,9]

chain.from_iterable()

chain.from_iterable(iterable with multiple iterables inside) → return a lazy iterator

chain裡面中的function,若目標是iterable中含多個iterable,chain能夠同時unpack外部和內部的Iterable,並且把return含有所有內部Iterable elements的lazy iterator。基本上想成一樣是把多個Iterables串起來,只是多拆了一層外部的Iterable就好:

from itertools import chain

iterable_iterable = [[1,2,3,4,5], [6,7,8,9,10], [11,12,13,14,15]]
chain_from_iterable = chain.from_iterable(iterable_iterable)
print(list(chain_from_iterable))

tee

chain(iterable,n) → return a tuple of independent iterator

功能就是複製傳入的iterable n遍,回傳的物件是一個tuple,並且內涵n個Iterator,要特別注意的是即便傳入的參數是Iterable,最後轉換出來的都是Iterator。

from itertools import tee

iterable_1 = [1,2,3,4]
copy_of_iterable_1 = tee(iterable_1, 3) #-> even though iterable_1 is iterable, it becomes iterator in the copy_of_iterable tuple
print(type(copy_of_iterable)) #tuple

for copy_item in copy_of_iterable: #copy_item is independent iterator
for item in copy_item:
print(item)
>> 1
>> 2
>> 3
>> 4
>> 1
>> 2
>> 3
>> 4
>> 1
>> 2
>> 3
>> 4

starmap

starmap(func, iterable) -> return lazy iterator

map function的延伸,處理的狀況和tee一樣,針對內含Iterable的Iterable會直接unpack內部Iterable的元素,過去map function一次只能迭代一個element,starmap function則可以一次處理多個elements作為參數:

from itertools import starmap

iterable_1 = [[1,2,3], [3,4,5]]

#Define a function with three arguments since we unpack internal iterable with three arguments
def power(x,y,z):
return x**2 + y**2 + z**2

result_iterator = starmap(power, iterable_1) #unpack internal iterable of iterable_1並一次丟三個參數給power function
print(list(result_iterator)) #[14,50]

accumulate

accumulate(iterable, fn) → return lazy iterator

用一個Iterator去紀錄參數Iterable再經過傳入function運算後的每次結果,運算方式和reduce function一樣,只是記錄了每次運算後的結果,對於trace code很有幫助。

from itertools import accumulate

accu_list = [1,2,3,4]
accu_iter = accumulate(accu_list, lambda x, y: x*y)
print(list(accu_list)) #[1,2,6,24]

zip longest

zip_longest(iterableA, iterableB, fillvalue=NA) → return lazy iterator

預設的zip function會根據最短的Iterable決定要return的tuple組數量,避免有些tuple會有沒有匹配到值的狀況:

zip_iter = zip([1,2,3], [10,20], ['a','b','c','d'])
for iter in zip_iter:
print(iter)

#最短的Iterable是[10,20],其他長的Iterable去配合他
>> (1, 10, 'a')
>> (2, 20, 'b')

我們可以使用zip_longest function的參數fillvalue,讓zip function去根據最長的Iterable決定return tuple的組數,較短的Iterable就用fillvalue的值去補:

from itertools import zip_longest

zip_iter = zip_longest([1,2,3], [10,20], ['a','b','c','d'], fillvalue="NA")
for iter in zip_iter:
print(iter)

>> (1, 10, 'a')
>> (2, 20, 'b')
>> (3, 'NA', 'c')
>> ('NA', 'NA', 'd')

groupby

groupby(iterable, key=) -> return lazy iterator contains (key, subiterator)

groupby是將資料分類的function,但在使用這個function前必須確保資料的groupby key已經排序好了,有以下資料:

(1, 20, 100),
(1, 11, 101),
(1, 12, 102),
(2, 20, 200),
(2, 21, 201),
(3, 30, 300),
(3, 31, 301),
(3, 32, 302)

而我們想要用每筆資料的第一個element作為key來分類,因為資料都已經以key值排序好了,接著我們可以直接呼叫groupby function:

from itertools import groupby

iterable_to_sort = [(1,10,100), (1,11, 101), (1,12, 102), (2, 20, 200), (2, 21, 201), (3,31, 301), (3,32, 302)]
groupby_iter = groupby(iterable_to_sort, key=lambda x: x[0]) #-> return a lazy iterator contains(key, subiterator)

for item in groupby_iter:
print(item) #(key, sub interator)
for sub_item in item[1]: #sub interator
print(sub_item)

>> (1, <itertools._grouper object at 0x0000019C1C273280>)
>> (1, 10, 100)
>> (1, 11, 101)
>> (1, 12, 102)
>> (2, <itertools._grouper object at 0x0000019C1C2731F0>)
>> (2, 20, 200)
>> (2, 21, 201)
>> (3, <itertools._grouper object at 0x0000019C1C273280>)
>> (3, 31, 301)
>> (3, 32, 302)

return出來的Iterator可以迭代出tuple,tuple的第一個element是分類排序的依據key值,第二個element則是被排進這個key的物件。

product

product(*args) -> return lazy iterator

product function中可以傳入多個Iterable,並以tuple Iterator的形式回傳這些Iterable可能形成的組合。

from itertools import product

iterable_1 = [1,2,3]
iterable_2 = ['a', 'b', 'c', 'd']
iterable_3 = [100, 200]
product_iterator = product(iterable_1, iterable_2, iterable3) #-> return lazy iterator contains tuples(1,'a',100),(1,'a',200),(1,'b',100),(1,'b',200),(1,'c',100),(1,'c',200)...


for product_result in product_iterator:
print(produc_result)

>> (1, 'a', 100)
>> (1, 'a', 200)
>> (1, 'b', 100)
>> (1, 'b', 200)
>> (1, 'c', 100)
>> (1, 'c', 200)
>> (1, 'd', 100)
>> (1, 'd', 200)
>> (2, 'a', 100)
>> (2, 'a', 200)
>> (2, 'b', 100)
>> (2, 'b', 200)
>> (2, 'c', 100)
>> (2, 'c', 200)
>> (2, 'd', 100)
>> (2, 'd', 200)
>> (3, 'a', 100)
>> (3, 'a', 200)
>> (3, 'b', 100)
>> (3, 'b', 200)
>> (3, 'c', 100)
>> (3, 'c', 200)
>> (3, 'd', 100)
>> (3, 'd', 200)

這個function主要使用的是在機率運算時,可以幫助快速列舉出所有的可能性。

除了這邊紀錄的function外,itertools module還有非常多的實用的function,例如排列組合的.permutations()和.combinations()等等,可以再去官方的document參考,是一個非常棒的資源。

--

--