Boost Python Performance with yield Generators

Learn how yield simplifies memory-efficient code with generators in Python.

Image generated by me, CyCoderX

Python is known for its simplicity and powerful features that make coding more efficient. One such feature is the yield keyword, which allows you to write functions that return generators, making it easier to work with large datasets and complex operations. While often overlooked by beginners, yield plays a significant role in creating more memory-efficient, lazy-evaluated code.

In contrast to the typical return statement, which immediately stops a function and returns a value, yield pauses the function, returning a value and resuming exactly where it left off when the next value is requested. This makes it perfect for building iterators and working with streams of data without loading everything into memory at once.

Hi all, my name is CyCoderX, and in this article, we’ll explore how the yield keyword works, its benefits, and some practical examples of how you can use it in your Python programs.

Let’s dive in!

Please consider helping me out by clapping. And if you’re interested in more Python, Data Engineering and Data Science content, please consider following me.😊

What is yield?

The yield keyword in Python is used in functions that generate an iterator—referred to as a generator function. Unlike a normal function that stops execution and returns a value when return is called, a generator function with yield suspends its state after each yield statement. It can resume later on, continuing where it left off.

This suspension and resumption of function execution make yield a powerful tool for working with large or potentially infinite data sets, as it generates values one at a time and only when requested. This is different from a typical function that returns all values at once, which can be memory-intensive.

Difference Between yield and return:

• return: Immediately exits the function and returns a value.
• yield: Pauses the function and saves its state, returning a value. The function can be resumed where it left off, allowing it to produce multiple values over time.

Example of a Simple Generator Function:

def simple_generator():
    yield 1
    yield 2
    yield 3

gen = simple_generator()
print(next(gen))  # Output: 1
print(next(gen))  # Output: 2
print(next(gen))  # Output: 3

In this example, the simple_generator function uses yield to return values one at a time. Instead of producing all three values at once, the generator returns each value when next() is called, and it keeps its state between calls.

Generators created by yield are particularly useful when you need to deal with large amounts of data, as they provide a way to iterate over items lazily, generating them on demand rather than all at once.

Understanding Unstructured and Semi-Structured Data: YAML, JSON, and XML

How yield Works

The yield keyword allows a function to return values one at a time, suspending its execution between each call. When the generator is called for the first time, it starts executing the function from the top. Each time the yield statement is encountered, the function outputs the value specified by yield and pauses its execution. When the generator is called again (using next() or in a loop), it resumes from where it left off.

This behavior makes yield ideal for scenarios where you need to produce a sequence of values but don’t want to store them all in memory at once. Instead of generating the entire sequence at once, the generator produces each value only when it's needed, reducing memory consumption.

Step-by-Step Explanation of How yield Works:

1. First Call: The generator function starts from the beginning and runs until it reaches the first yield statement. It returns the value and suspends execution, saving the function's current state.
2. Subsequent Calls: On subsequent calls (using next() or iterating over the generator), the function resumes from where it left off, executes until the next yield statement, returns the next value, and suspends again.
3. Exhausting the Generator: When the function runs out of yield statements, it raises a StopIteration exception, signaling that the generator is exhausted.

Example of How yield Works:

def count_up_to(limit):
    count = 1
    while count <= limit:
        yield count
        count += 1

counter = count_up_to(3)
print(next(counter))  # Output: 1
print(next(counter))  # Output: 2
print(next(counter))  # Output: 3

In this example:

• The generator count_up_to counts up to a specified limit. It yields each value one at a time.
• The next() function is used to retrieve the next value in the sequence, and the function resumes from the point after the last yield.

Once all the values are generated, the function is finished, and calling next() again will raise a StopIteration exception.

Using yield in Loops:
When working with generators, Python’s for loop is often used because it automatically handles StopIteration for you.

for number in count_up_to(3):
    print(number)

This will output:

1
2
3

The loop stops when the generator is exhausted, without needing to call next() manually.

Data Science Fundamentals: Techniques and Tools

You’re helping a stuck developer out there when you create developer content. — William Imoh

Choosing the Right Azure Storage Solutions for Online Retail

Advantages of Using yield

The yield keyword offers several advantages, particularly when working with large datasets or streams of data that are impractical to store in memory all at once.

1. Memory Efficiency

One of the key advantages of yield is memory efficiency. When dealing with large datasets or infinite sequences, generating all values at once can consume a large amount of memory. With yield, values are generated lazily—only when needed—significantly reducing memory usage.

Example: Imagine needing to process a file line by line. Instead of loading the entire file into memory, you can use yield to read and process each line one at a time:

def read_large_file(file_name):
    with open(file_name) as file:
        for line in file:
            yield line

for line in read_large_file("big_file.txt"):
    process(line)

This approach is far more memory-efficient because the file’s contents aren’t loaded into memory all at once.

2. Lazy Evaluation

yield enables lazy evaluation, meaning values are only produced when explicitly requested. This can improve performance when working with large or expensive-to-compute sequences of data. If not all values are needed at once, yield helps avoid unnecessary computations.

3. Infinite Sequences

With yield, it’s possible to generate infinite sequences, as the function doesn’t try to return all values at once. Instead, it produces values on demand and pauses between each one.

Example: Infinite Fibonacci sequence generator:

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib_gen = fibonacci()
for _ in range(5):
    print(next(fib_gen))

Output:

0
1
1
2
3

This generator can produce Fibonacci numbers endlessly, without needing to compute or store the entire sequence in memory.

4. Simplifying Iterators

Without yield, creating an iterator requires writing a class that defines __iter__() and __next__() methods. With yield, you can easily convert any function into an iterator, simplifying your code significantly.

Python zip() and When to Use It

Practical Use Cases of yield

The yield keyword shines in real-world applications where memory efficiency, lazy evaluation, and handling large datasets are critical. Here are some practical use cases where yield proves invaluable:

1. Iterating Over Large Files

When working with large files, loading the entire file into memory can be inefficient or even impossible. Using yield allows you to process each line or chunk one at a time without overwhelming your system's memory.

Example: Reading a large file line by line:

def read_large_file(file_path):
    with open(file_path, 'r') as file:
        for line in file:
            yield line

# Process each line one at a time
for line in read_large_file('large_text_file.txt'):
    process(line)

This approach is memory-efficient because each line is processed one at a time rather than reading the entire file into memory.

2. Streaming Data

In scenarios where data is continuously generated or retrieved (such as a data stream from a sensor or a server), using yield to handle the stream one piece at a time ensures efficient resource use.

Example: Streaming sensor data:

def sensor_data_stream():
    while True:
        data = get_sensor_data()  # Hypothetical function
        yield data

for data in sensor_data_stream():
    process(data)

This infinite generator simulates a data stream where new data is continuously generated, and yield ensures that only one piece of data is handled at a time.

3. Infinite Sequences

With yield, you can create infinite sequences, generating each value as it’s needed. This is useful for algorithms or simulations that need a potentially unlimited set of values, such as generating Fibonacci numbers or simulating clock ticks.

Get CyCoderX’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Example: Fibonacci sequence generator:

def fibonacci_sequence():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# Generate the first 10 Fibonacci numbers
fib_gen = fibonacci_sequence()
for _ in range(10):
    print(next(fib_gen))

In this case, the Fibonacci sequence is generated lazily, ensuring that each number is computed only when requested.

4. Paging Through Database Results

When querying large datasets from a database, fetching all results at once can be resource-intensive. Using yield, you can fetch results in smaller batches, processing them without consuming too much memory.

Example: Fetching results in chunks:

def fetch_data_in_batches(batch_size):
    offset = 0
    while True:
        results = query_database(offset, batch_size)  # Hypothetical function
        if not results:
            break
        yield results
        offset += batch_size

# Process batches of data
for batch in fetch_data_in_batches(100):
    process(batch)

Here, data is fetched and processed in small batches, avoiding the overhead of loading the entire dataset into memory.

NumPy Array Copy vs. View

Photo by Francesco Maria Achille on Unsplash

Sorted vs. Sort in Python

Handling State in Generators

One of the unique features of yield is that it allows you to maintain the state of a function between calls. Unlike regular functions that start fresh with each invocation, a generator preserves its local variables and execution state across successive calls, making it perfect for scenarios where state management is required.

Example: A Stateful Counter

Let’s look at a simple example where the generator maintains state across iterations:

def counter():
    count = 0
    while True:
        yield count
        count += 1

# Create a generator that counts up
counter_gen = counter()
print(next(counter_gen))  # Output: 0
print(next(counter_gen))  # Output: 1
print(next(counter_gen))  # Output: 2

In this case, the counter function preserves the value of count between iterations, so each time next() is called, the generator resumes from where it left off, incrementing the count.

Example: Handling External Input

Generators can also interact with the outside world by receiving values via the send() method, which allows external input to modify the generator’s state while it is running.

Example of send():

def accumulate():
    total = 0
    while True:
        value = yield total
        if value is not None:
            total += value

# Create the accumulator generator
acc = accumulate()
print(next(acc))  # Output: 0 (initial total)

# Send values to the generator and accumulate the total
print(acc.send(10))  # Output: 10
print(acc.send(5))   # Output: 15

Here, the accumulate generator starts with a total of 0. Each time a value is sent using send(), it adds that value to the total, preserving the state between calls.

Why State Management is Useful

• Stateful computations: Generators can easily keep track of accumulated values, counters, or other evolving states over multiple calls, which is useful in tasks like data aggregation or event handling.
• Efficient execution: By maintaining state internally, generators avoid the overhead of re-initializing variables or recalculating intermediate results on every call.

Azure Data Factory: The Ultimate Cloud Data Integration Tool

Using yield from for Nested Generators

In Python, the yield from expression is a convenient way to delegate part of a generator’s operations to another generator. This is particularly useful when you need to yield values from multiple sources or when a generator wants to yield all values produced by another generator without having to explicitly iterate over it. The yield from statement allows a generator to yield every value from a sub-generator.

Using yield from simplifies code by avoiding nested loops and making the delegation of generator functions more readable and concise.

How yield from Works

The yield from expression connects a generator to a sub-generator, automatically yielding values from the sub-generator as if they were part of the original generator.

Example: Basic yield from Usage:

def generator_a():
    yield 'A1'
    yield 'A2'

def generator_b():
    yield 'B1'
    yield from generator_a()  # Delegate to generator_a
    yield 'B2'

# Using the combined generator
for value in generator_b():
    print(value)

Output:

B1
A1
A2
B2

In this example, generator_b uses yield from generator_a() to delegate part of its operation to generator_a. The values A1 and A2 are yielded by generator_a, but they appear as if they are being yielded by generator_b.

Advantages of Using yield from

1. Simplifies Code: Without yield from, you would need to manually loop through the sub-generator and yield each value, which can make your code more complex. yield from abstracts this process and makes delegation easy.
2. Composability: It encourages composable generator functions, allowing you to break down complex logic into smaller, reusable generators that can be easily combined.

Example: Summing Nested Generators:

Here’s an example of using yield from to combine data from nested generators:

def numbers():
    yield from range(1, 4)  # Yields 1, 2, 3

def letters():
    yield from ['A', 'B', 'C']  # Yields 'A', 'B', 'C'

def combined():
    yield from numbers()
    yield from letters()

# Using the combined generator
for value in combined():
    print(value)

Output:

1
2
3
A
B
C

In this case, combined() uses yield from to pull values from both numbers() and letters() generators, making the code simple and elegant.

Handling Delegation with yield from:

• Delegation of Iterables: You can also use yield from with any iterable, not just generators. This makes it versatile for working with lists, tuples, and other iterable objects.

def generator():
    yield from [1, 2, 3]  # Delegates to a list

• Passing Values Back: In advanced use cases, yield from can also pass values back to the delegating generator or handle exceptions. This makes it a powerful tool for writing robust and reusable generator-based code.

Python type() vs isinstance()

Conclusion

The yield keyword is a powerful feature in Python, enabling developers to create memory-efficient, lazy-evaluated functions known as generators. By allowing functions to produce values on demand, yield reduces memory usage and improves performance, especially when working with large datasets, infinite sequences, or streams of data.

We’ve explored how yield differs from return, how it works in practice, and the many benefits it offers in real-world use cases. From reading large files line by line to generating Fibonacci numbers or working with nested generators using yield from, yield provides a flexible and efficient way to manage state and handle sequences of data.

Mastering yield and understanding how to use it effectively can enhance your ability to write clean, efficient Python code, especially when dealing with complex data workflows or performance-critical applications.

Happy Coding!

Always remember to conduct your own research and verify the information you encounter. Relying solely on others can result in outdated or incorrect practices. Take the initiative to implement and refine the code you find, ensuring it meets your specific needs and standards.

What is PEP 8 and Why is it Important in Python?

Text Manipulation with ord() and chr() in Python

Photo by Call Me Fred on Unsplash

Final Words:

Thank you for taking the time to read my article.

This article was first published on Medium by CyCoderX.

Hey There! I’m CyCoderX, a data engineer who loves crafting end-to-end solutions. I write articles about Python, SQL, AI, Data Engineering, lifestyle and more!

If you want to explore similar articles and updates, feel free to explore my Medium profile:

Join me as we explore the exciting world of tech, data and beyond!

What did you think about this article? Let me know in the comments below … or above, depending on your device! 🙃

Please consider supporting me by:

1. Clapping 50 times for this story
2. Leaving a comment telling me your thoughts
3. Highlighting your favorite part of the story

In Plain English 🚀

Thank you for being a part of the In Plain English community! Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Discord | Newsletter | Podcast
• Create a free AI-powered blog on Differ.
• More content at PlainEnglish.io