DS in Python 1.1: Use of Iterators in Real Life

Let us first understand with an example how Iterators actually help to save memory

Example: List vs Iterator

Suppose we want to process numbers from 1 to 1,000,000.

1. Using a List

numbers = [i for i in range(1, 1_000_001)]

for number in numbers:
    print(number)

Let’s look at the Space complexity for the above code step by step:

Step 1:
range(1, 1_000_001)
Space: O(1)

range() itself is memory efficient. It represents the sequence without creating one million integers in a list.

Step 2:
[i for i in range(1, 1_000_001)]
Space: O(n)

Now we create a list containing 1,000,000 values.

If n is the number of values: Space = O(n)

The list has to keep all those elements in memory.

Step 3:
for number in numbers:

Space: O(n)

The list still exists while we’re processing it, so its memory requirement remains O(n).

2. Using an Iterator

Now let’s use an iterator:

numbers = iter(range(1, 1_000_001))
for number in numbers:
    print(number)

Let’s see the space complexity in this case:

Step 1:
range(1, 1_000_001)
Space: O(1)

range() doesn't create one million numbers and store them in a list.

Then:

numbers = iter(range(1, 1_000_001))

Step 2:
Create iterator
Space: O(1)

The iterator keeps track of its current position rather than storing the entire sequence.

Conceptually:

Iterator
   ↓
current position = 1

Then:

for number in numbers:

Python asks the iterator for the next value.

Step 3:
next() → 1
Memory: O(1)

Then:

Step 4:
next() → 2
Memory: O(1)

Then:

Step 5:
next() → 3
Memory: O(1)

And this continues:

next() → 4       O(1)
next() → 5       O(1)
next() → 6       O(1)
...
next() → 999,999 O(1)
next() → 1,000,000 O(1)

At no point does the iterator need to store all one million values.

Question: In an iterator, when next() is invoked to retrieve the next element, how is memory managed for the previously returned value? Does the iterator discard the old value, overwrite it with the new one, or maintain it in memory until it is no longer referenced?

Memory comparison

List:
1, 2, 3, 4, 5, 6, 7, ............., 999,999, 1,000,000
└──────────────────────────────────────────────────┘
              stored in memory
Space: O(n)

Iterator:

Iterator
   ↓
current = 1

then
Iterator
   ↓
current = 2

then
Iterator
   ↓
current = 3
...

Only the information necessary to produce the next value is maintained.

Hence Space: O(1)

But There’s an Important Detail

Be careful while saying:

“Iterators always use O(1) space.”

That’s not universally true.

An iterator can internally store data, and some iterators may use O(n) memory.

The more accurate statement is:

Iterators allow data to be processed one item at a time, and many iterators — especially generators — can do this using O(1) additional space.

Example:

numbers = [i for i in range(n)]

creates the entire list and uses O(n) space.

Iterators in Data Science

In Data Science, this becomes much more practical because datasets can be huge — millions or even billions of rows.

The key idea is:

Instead of loading the entire dataset into RAM, an iterator lets you load and process a small portion at a time.

Let’s understand this with a real-world example.

Real-life example: Customer transactions

Imagine you’re a Data Scientist at an e-commerce company.

You have a CSV file:

transactions.csv

It contains 100 million transactions.

Each row might look like:

customer_id,product,amount,date
101,Laptop,75000,2026-08-01
102,Phone,25000,2026-08-01
103,Headphones,5000,2026-08-02
...

Your task is : Calculate the total sales.

Approach 1: Load the entire dataset

A beginner might write:

import pandas as pd

df = pd.read_csv("transactions.csv")
total_sales = df["amount"].sum()
print(total_sales)

This is easy.

But think about what happens in memory.

Suppose your CSV is 10 GB.

You might think:

CSV file = 10 GB
RAM required = 10 GB

But that’s not necessarily true.

When Pandas loads the CSV, it creates in-memory data structures. Depending on the data types, strings, indexes, intermediate objects, etc., the DataFrame can require considerably more or less than the file’s on-disk size.

Get Abhishek Jain’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

So you could have:

            RAM
              ↓
┌─────────────────────────────┐
│                             │
│   100 million rows          │
│                             │
│   customer_id               │
│   product                   │
│   amount                    │
│   date                      │
│                             │
└─────────────────────────────┘

Potential problem:

Available RAM = 16 GB
Dataset       = very large
             ↓
MemoryError / swapping / very slow processing

#Swapping means your computer starts using the SSD/HDD as temporary memory because RAM is running low.

Approach 2: Read the data in chunks

Instead of loading everything, we can process the CSV in chunks.

Pandas provides an iterator-like interface when we use chunksize.

import pandas as pd

chunks = pd.read_csv(
    "transactions.csv",
    chunksize=100_000
)
total_sales = 0
for chunk in chunks:
    total_sales += chunk["amount"].sum()
print(total_sales)

This is a very important pattern in Data Science.

Let’s understand what is happening.

Step 1: Create the chunk iterator

chunks = pd.read_csv(
    "transactions.csv",
    chunksize=100_000
)

We’re telling Pandas:

“Don’t give me the entire CSV. Give me 100,000 rows at a time.”

Conceptually:

  100 million rows
        ↓
┌──────────────────┐
│   100,000 rows   │ ← Chunk 1
└──────────────────┘
┌──────────────────┐
│   100,000 rows   │ ← Chunk 2
└──────────────────┘
┌──────────────────┐
│   100,000 rows   │ ← Chunk 3
└──────────────────┘
       ...
┌──────────────────┐
│   100,000 rows   │ ← Chunk 1000
└──────────────────┘

You don’t need all 100 million rows in RAM.

Step 2: Process the first chunk

for chunk in chunks:
    total_sales += chunk["amount"].sum()

Initially:

          RAM
           ↓
┌──────────────────────┐
│ Chunk 1              │
│ 100,000 rows         │
└──────────────────────┘
total_sales = 0

We calculate:

chunk["amount"].sum()

Suppose the result is:

₹50,00,00,000

Now:

total_sales = ₹50,00,00,000

Step 3: Move to the next chunk

The iterator gives us the next chunk.

Conceptually:

Chunk 1
   ↓
processed
   ↓
discarded

Chunk 2
   ↓
processing 
...

Memory doesn’t need to contain all previous chunks.

So:

  RAM
┌──────────────────────┐
│ Chunk 2              │
│ 100,000 rows         │
└──────────────────────┘
total_sales = previous total + current total

Step 4: Continue

The process continues:

Chunk 1 → Calculate → Add to total
Chunk 2 → Calculate → Add to total
Chunk 3 → Calculate → Add to total
Chunk 4 → Calculate → Add to total
...
Chunk 1000 → Calculate → Add to total

At the end:

total_sales = ₹XXXX

And we never needed to hold all 100 million rows in memory simultaneously.

Question:

If a dataset with 100 million rows is already stored on my computer, and I load it into a program, is the data loaded into RAM again? Or does the program just create a pointer/reference to the data already stored on disk?

The file on your disk and the data loaded into RAM are two different things.

Suppose you have:

transactions.csv
100 million rows
10 GB

That 10 GB is storage (disk/SSD) space, not RAM.

Then where do iterators help?

This is where our question becomes really important.

Suppose the CSV is:

10 GB on disk

and the complete DataFrame requires, say:

12 GB RAM

If your computer has:

RAM = 16 GB

you might be able to load it, but your system has very little RAM left for everything else.

Instead we handle it this way:

for chunk in pd.read_csv("transactions.csv", chunksize=100_000):
    process(chunk)

Pandas can read one portion at a time instead of reading everything at once.

Conceptually:

In simple terms:

• SSD/Disk: Your dataset, such as a CSV file, is physically stored here.
• OS File Cache: When the program reads the file, the operating system may keep recently read parts of the file in RAM so they can be accessed faster later.
• RAM: The data needed by your program is available in memory here.
• Pandas DataFrame: Pandas reads and parses the file and creates its own in-memory representation so you can perform operations such as filtering, grouping, and calculations.

Question

If the entire dataset is loaded into memory at once, will the OS file cache corresponding to that dataset also become large?

On the other hand, if we use an iterator or process the dataset in chunks, only a small portion of the dataset is loaded into memory at a time. Does that mean the OS file cache will also remain small?

Once the entire dataset has been processed, will the OS file cache be the same in both cases?

Finally, how does the operating system decide:

• Which parts of a file should be kept in the OS cache?
• How much memory should be allocated to the cache?
• When should cached data be removed (evicted) from memory?

Answer

• When you load the entire dataset using Pandas, the DataFrame takes up RAM, while the OS may also keep some of the file’s recently accessed data in its file cache.
• However, the OS cache is not necessarily as large as the dataset — it decides how much data to cache based on available RAM and other factors.
• When you use an iterator or process the dataset in chunks, Pandas only keeps a small portion of the data in memory at a time, but the OS can still cache more file data in the background.
• So, even after the entire dataset has been processed, the OS cache in both cases may or may not be the same.
• The OS continuously decides which file data to keep based mainly on what was recently accessed and how much RAM is available. If another application needs RAM, the OS can remove older cached data and use that memory for something else.