Python Iteration Protocol: Iterables vs Iterators

Python separates the `Iterable` and `Iterator` protocols to support multiple independent cursors and lazy streaming computation. It achieves O(1) memory usage at the cost of random access and replayability.

If you use a Python list to load a 100 GB log file all at once, the process will quickly run out of memory (OOM) and the operating system will terminate it. Switch the code to stream the file through an iterator, and memory usage immediately drops to a constant level. However, if you then call len() to get the total number of records, Python raises a TypeError. If you traverse the same stream a second time, downstream computations silently receive no data.

If you are new to Python — or simply approaching the code with conventional assumptions — these behaviors can be confusing. (AI-generated code may not expose these issues directly, but you still need to understand the underlying mechanics.)

Many engineers can comfortably write for x in data: to traverse a sequence. However, they often conflate the data owner with the progress tracker. This conceptual ambiguity can easily cause production defects—for example, nested loops interfering with each other’s traversal state, or one-shot streams being mistakenly treated as reusable collections.

The key to understanding Python’s iteration model is recognizing the separation of responsibilities between an Iterable and an Iterator.

The rest of this article works through three steps:

1. Clarify how Python’s iteration protocol moves between containers and cursors, and compare the different cursor implementations used by lists and dictionaries;
2. Write a custom cursor class and deconstruct generators to demonstrate that stream processing and lazy evaluation share the same underlying contract;
3. Quantify the time and space trade-offs of one-way consumption and non-sliceable data, then establish boundaries for defensive programming.

To develop a complete mental model for stream processing, we must first answer a fundamental architectural question: why does Python not allow a container to track its own read position directly? We will start with the decoupling contract between containers and cursors.

1. Separation of Responsibilities and Protocol Contracts: Why Containers and Cursors Must Be Separate

If a list tracked its own traversal progress, nested loops would fail immediately. Once the outer loop reached its second element, starting the inner loop would reset the container’s internal cursor or advance it to the end, directly overwriting the outer loop’s state.

To prevent this state conflict, Python separates data ownership from progress tracking into two distinct roles: iterables and iterators.

Why Containers Must Be Decoupled from Cursors

An iterable is the data owner. It is responsible for storing data and providing access to it. An iterator is a forward-only cursor that tracks only the current read position.

Data Owner and Cursors

Mental model:

An iterable is like a published book, while an iterator is a bookmark placed inside it by a reader. Multiple readers can use separate bookmarks to read the same book without affecting one another’s progress.

• Where the analogy holds: The book stores the content, while each bookmark records the current page.
• Where it breaks down: A physical bookmark can move backward and forward. A Python iterator cursor can only move forward and cannot rewind.

Common misconception: anything you can put in a for loop is an iterator. In fact, list, tuple, and dict are only iterables — they carry no cursor state of their own and cannot answer ‘where am I in the traversal?’.

This decoupling introduces some state-management overhead. Each traversal requires the runtime to instantiate a new iterator cursor, which means allocating a new object on the heap in interpreters such as CPython. While this creates a small allocation cost, the per-traversal allocation provides independent cursors that won’t interfere with each other’s iteration state — the same property that keeps nested loops predictable.

How Python’s for Loop Drives the Iteration Protocol

Python’s for loop is not fundamentally an index-based counting loop. It is syntactic sugar built on the iteration protocol. It completes traversal in three steps: acquire a cursor, advance it repeatedly, and handle the termination signal.

For Loop Driver

The following code reproduces the underlying logic of a for loop using the iteration protocol directly:

# Equivalent low-level implementation of: for x in [10, 20]: print(x)
iterator = iter([10, 20])        # 1. Acquire an independent cursor (calls __iter__)
while True:
    try:
        x = next(iterator)       # 2. Advance the cursor and retrieve a value (calls __next__)
        print(x)
    except StopIteration:        # 3. Catch the termination signal
        break                    # 4. Exit the loop and release the cursor

Mechanics and trade-offs:

1. A unified access abstraction: Code does not need to know whether a container supports indexed access. Lists with subscripting support, unordered collections such as set, database cursors, and file streams all expose data through the same __next__() interface.
2. Exception-driven control flow: Python follows the EAFP (Easier to Ask Forgiveness than Permission) philosophy and relies on the cursor raising StopIteration to signal the end of a stream. Because termination is signaled by an exception, tight loops over very small collections pay a small interpreter cost per iteration for exception handling and stack unwinding.

2. Implementing the Iterator Design Pattern and Behavioral Differences in Built-In Containers

A standard iteration protocol separates two distinct roles in code: a data holder and an independent cursor.

Hand-Written, Decoupled Data Holder and Cursor Classes

Under Python’s iteration protocol, the data holder stores the data and returns a fresh cursor for each iteration request. The cursor is responsible for maintaining its own forward-only traversal state.

Python’s iteration protocol uses duck typing and does not require explicit inheritance. A class satisfies the protocol contract as long as it implements the required magic methods. A cursor class must implement both __next__() and __iter__().

class NumberData:  # Data holder (Iterable)
    def __init__(self, items):
        self._items = list(items)

    def __iter__(self):
        return NumberCursor(self._items)  # Return an independent, fresh cursor for each traversal


class NumberCursor:  # Independent cursor (Iterator)
    def __init__(self, source):
        self._source = source
        self._index = 0

    def __iter__(self):
        return self  # Self-reference: the cursor itself is also a valid Iterable

    def __next__(self):
        try:
            val = self._source[self._index]
        except IndexError:
            raise StopIteration  # An out-of-range index terminates iteration without checking container length
        self._index += 1
        return val

The rationale for self-reference: NumberCursor.__iter__() returns self directly. As a result, NumberCursor can serve as both an Iterator and an Iterable.

When developers pass an already-created cursor directly to a for loop, zip(), or enumerate(), the interpreter calls iter(cursor) and correctly receives the cursor itself. The collections.abc.Iterator abstract base class provides this same __iter__() implementation, which returns self.

Hand-Written, Decoupled Data Holder and Cursor Classes

Common misconceptions and trade-offs: Developers often assume that an ordinary iterable container can also return self from __iter__(). In practice, once a container returns itself, it becomes a single-use consumable stream. Nested loops then share and overwrite the same traversal state.

The hand-written two-class pattern fully decouples data storage from traversal state, but it also introduces additional boilerplate. When the iteration logic does not require complex state transitions, manually defining classes is more verbose than using generator syntax.

How Built-In Container Iterators Differ (list / tuple / dict)

Python’s built-in container iterators differ fundamentally in their underlying memory layouts and their handling of concurrent structural modifications.

1. Sequence iterators (list / tuple):

In CPython, lists and tuples use sequence iterators (list_iterator and tuple_iterator). The cursor maintains an integer offset into a contiguous memory array. Each call to next() increments the index and yields an object reference. If external code inserts or removes list elements during iteration, the iterator does not raise an exception. However, the mutation can cause logical errors, such as skipped elements or repeated reads.

Skipped Element

2. Dictionary iterators (dict) and version tracking:

To guard against hash-table structural changes, CPython (3.6+) maintains an incrementing version counter (ma_version) on the dictionary struct. Every key insertion or deletion updates this version tag. When a dict_keyiterator is created, it captures the current version and checks it on every next() call. If the versions diverge, the interpreter immediately raises RuntimeError: dictionary changed size during iteration, providing fail-fast behavior.

Dictionary iterator

CPython’s dict iterator uses an internal version stamp (ma_version), not a size comparison, so it detects structural changes even when the new size happens to match the old one.

Mental model:

• A list iterator: Like walking down a corridor and visiting rooms by fixed address numbers.
• A dictionary iterator: Like reading a loose-leaf binder whose pages may be inserted or removed; before turning each page, you check whether the binder’s structure has changed.
• What they have in common: Both produce data in a forward-only sequence.
• How they differ: Room numbers in a corridor do not automatically move, while modifying the pages in a loose-leaf binder can invalidate the current reading position.

Boundaries and applicable conditions:

Dictionary iterator validation primarily detects changes to the number of entries. Updating the value of an existing key does not change the table structure, so the iterator keeps running.

Although hand-written cursor classes provide clear separation of responsibilities, each data stream requires its own class. The next chapter explains how generators simplify this process.

3. The Structural Equivalence of Generators and Iterators, and Lazy Execution

At the implementation level, a generator object fully satisfies the iterator protocol contract. It is not a separate technology or concept outside the iteration model; instead, it is Python’s higher-level abstraction for eliminating the boilerplate of hand-written cursor implementations.

Demonstrating That Generators Behave as Iterators

In my early development, whether I was working with TypeScript or Python, I kept mistaking generators for a standalone concept that exists outside the iteration model. In reality, in Python’s type system and runtime, a generator is fundamentally just a standard iterator.

Get Nan Zhou’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Calling a function that contains yield does not immediately execute its body. Instead, it returns a generator object.

from collections.abc import Iterator

def count_stream():
    yield 1
    yield 2

gen = count_stream()
# Verify that a generator fully satisfies the Iterator contract
assert isinstance(gen, Iterator)
assert hasattr(gen, "__iter__") and hasattr(gen, "__next__")
assert iter(gen) is gen  # Satisfies the cursor self-reference rule

Causal chain and mechanisms:

1. A function containing the yield keyword is marked as a generator factory at compile time.
2. When the function is called, the interpreter allocates a generator object and associates it with the code object and stack frame. At this point, it does not execute any actual instructions.
3. Inspecting the generator object shows that it natively implements the __iter__ and __next__ methods.
4. Only when next(gen) is called for the first time does execution actually advance to the first yield, produce a value, and suspend.

Common misconception:

Generators are sometimes treated as a separate mechanism outside the iteration model. In practice, their interface and runtime behavior match an iterator exactly.

A function containing yield is merely a factory for generator objects. The generator instance returned by calling that function is the actual forward-only iterator cursor.

Boundaries and trade-offs:

Generators eliminate the boilerplate required to implement cursor classes manually. However, the runtime stack frame fully encapsulates their internal cursor position. Generators hide their cursor position inside the runtime frame — there is no public cursor.index equivalent to peek at or reset.

Lazy Evaluation and the Suspension and Resumption of Execution Frames

The essence of stream processing is trading time for space. Generators produce one element on demand by suspending and resuming their execution stack frames at runtime.

Underlying principles and execution model (based on CPython 3.8+): When execution reaches a yield statement, the interpreter marks the generator’s stack frame as suspended. The runtime preserves the current frame’s local-variable table and the most recent instruction offset (f_lasti), then returns the yielded value to the caller. The function does not destroy its execution stack frame at this point.

When the caller next calls next(gen), the interpreter restores the generator’s stack frame and resumes execution from the instruction immediately following f_lasti, continuing until it reaches another yield or the function completes and returns.

Mental model:

• A fully materialized list: Pre-making 1,000 boxed meals and stacking them in the warehouse fills the room before the first customer arrives.
• A generator: Cooking each dish to order. A customer orders one serving (one call to next), the chef prepares one serving (executes until yield), and then waits in place.
• What they have in common: Both can ultimately provide the caller with the same number of data elements.
• How they differ: You can retrieve the fifth prepackaged meal at any time by its number (random indexed access). Made-to-order cooking must proceed in order, cannot skip ahead, and cannot reproduce elements that have already been consumed.

Trade-offs and costs:

Lazy evaluation drops working-space complexity from O(N) to O(1) — the iterator only holds the current frame, never the full collection.

However, this space optimization comes at a CPU-throughput cost. Each generated element requires the generator stack frame to be suspended and resumed. If a generator pipeline becomes deeply nested, frequent stack-frame context switches and function-dispatch overhead can make total CPU time noticeably higher than a one-shot batch computation over contiguous memory.

4. Engineering Trade-offs, Costs, and Pitfalls of Stream Processing

Stream-based iterators reduce memory complexity to O(1). The trade-off is that they give up random access, known length, and repeatable iteration. This design is highly efficient for processing unbounded data streams. However, in typical application code, re-consuming an exhausted iterator does not raise an error; it simply returns no elements. As a result, downstream computations can silently omit data.

Time-Space Costs and Lost Capabilities of Stream Processing

An iterator cursor stores only its current position in memory. It neither retains previously visited records nor prefetches elements that have not yet been traversed. This lightweight design directly eliminates the capabilities normally associated with sequences.

data_stream = (x * 2 for x in range(5))

# 1. Length is unavailable: raises TypeError: object of type 'generator' has no len()
# len(data_stream)

# 2. Random indexing and slicing are unavailable: raises TypeError: 'generator' object is not subscriptable
# first_item = data_stream[0]

# 3. One-way, irreversible consumption
list_a = list(data_stream)  # Produces [0, 2, 4, 6, 8]
list_b = list(data_stream)  # The cursor is exhausted, so this returns []

Causal chain and cost analysis:

1. Loss of length and indexing: Iterators do not scan the entire dataset internally. Calling len(iterator) raises a TypeError. Likewise, using iterator[i] for random access or slicing also raises a TypeError. Any algorithm that depends on indexed backtracking cannot operate directly on a streaming iterator.
2. One-way state exhaustion: After an iterator has been fully traversed, its cursor remains at the end. If business logic performs multiple aggregations over the same dataset — for example, calculating the total count before calculating the average — the second traversal does not raise an error. Instead, it produces an empty sequence. This can cause division-by-zero errors or silently skip subsequent statistical calculations.

What goes wrong with itertools.tee?:

Many developers mistakenly assume that itertools.tee can clone multiple independent streaming iterators at no cost.

itertools.tee shares the underlying data source across branch iterators. It uses a first-in, first-out (FIFO) queue to buffer elements that have not yet been consumed by every branch.

If one branch consumes quickly while another lags, the FIFO buffer balloons — space complexity effectively becomes O(N) for however long the lag persists.

Three Production Pitfalls and Defensive Patterns

Three failure modes show up repeatedly in production: state exhaustion, concurrency conflicts, and resource leaks. Each calls for a specific defensive pattern.

1. Failed Reuse Caused by Passing Iterators Across Functions

When a generator is passed as an argument across multiple business functions, an earlier function may fully traverse it or invoke consuming built-in functions such as any() or max(), thereby exhausting its cursor. When a later function attempts to read from it again, it receives empty data and produces incorrect results.

Defensive pattern: If the dataset is known to be small and requires repeated access, explicitly convert it to a static list at the entry point with list(stream). If the dataset is too large to load entirely into memory, refactor the functions to accept a generator factory—one that returns a fresh iterator each time—instead of accepting an iterator instance directly.

2. Race Conditions When Multiple Threads Advance an Iterator

Advancing a Python iterator is not an atomic operation. In a multithreaded environment, if multiple worker threads concurrently call next(iter) on the same iterator, the internal cursor state can experience a race condition. This can cause some elements to be processed more than once or skipped entirely.

Defensive pattern: Never share a bare iterator across threads. When distributing tasks across multiple threads, push data into a thread-safe queue.Queue in advance, or explicitly synchronize calls to next() with a mutex lock (threading.Lock).

3. External Resource Leaks Caused by Early Termination

Generators often encapsulate external I/O resources, such as file descriptors or database cursors. If the caller exits iteration early with break, the generator function remains suspended. System resources held internally may not be released promptly.

def read_records(filepath):
    f = open(filepath, "r", encoding="utf-8")
    try:
        for line in f:
            yield line.strip()
    finally:
        # Ensure that the file is closed whether iteration completes,
        # exits early with break, or raises an exception
        f.close()

Defensive pattern: Protect any underlying system resource acquired inside a generator with a try...finally block or a with context manager. When the caller exits early, the generator receives a GeneratorExit (from close() or eventual GC). The finally block then runs and releases the handle.

Summary

Python separates the Iterable and Iterator protocols to support multiple independent cursors and lazy streaming computation. It achieves O(1) memory usage at the cost of random access and replayability.

Key Advantages

• Containers and cursors are decoupled: Multiple cursors can traverse the same data independently, so nested loops do not interfere with each other.
• Lazy execution: Streaming iterators produce values on demand with O(1) memory complexity, making them suitable for extremely large or unbounded data sources.
• Unified protocol: Custom cursors, generators, and built-in iterators (list_iterator, dict_keyiterator, and so on) all follow the same interface.

Limitations and Trade-Offs

• Iterators do not support len() or indexed access such as iter[i]. You can advance them only once, in one direction, through next().
• Once a streaming iterator is exhausted, it cannot be replayed. To consume it again, you must recreate it or materialize it at the entry point.
• Generators encapsulate stack frames, so you cannot directly inspect their internal local variables.
• The internal FIFO buffer used by itertools.tee can grow from O(1) to O(N) when branches consume values at different speeds, creating an OOM risk.
• Adding or removing dictionary keys during iteration raises RuntimeError: dictionary changed size during iteration. CPython implements this fail-fast behavior through ma_version.
• Sharing next() across threads is not atomic and can introduce race conditions. Do not use an iterator as a cross-thread queue.

Appropriate Use Cases

• Process items one at a time when the data source is unknown or extremely large, such as logs, database cursors, file lines, or network streams.
• Traverse the same collection concurrently with multiple independent cursors, such as in board traversal, two-pointer algorithms, or nested grouping.
• Replace manual state machines with generators to reduce boilerplate and memory usage.

Pitfalls and How to Avoid Them

• Multiple consumers: Materialize the stream at the entry point with list(stream), or use a factory function that returns a new generator each time.
• Multithreaded distribution: Use queue.Queue to decouple producers and consumers instead of sharing a raw iterator. When necessary, protect next() with threading.Lock.
• Early exit with break: Wrap resources such as file handles, connections, and locks inside the generator with try ... finally or with.
• Do not return self from a container's __iter__(): Doing so shares consumption state and breaks nested iteration.
• Debugging generator exhaustion: Catch the exception rather than checking for None, consistent with the EAFP style.