Bug report

Bug description:

SpooledTemporaryFile provides a temporary file backed by a buffer in memory that spills over to disk when it gets too large. However, the writelines() method only checks whether it should roll over after the entire lines iterator is exhausted. This causes unexpectedly high memory use when feeding it large iterators.

from tempfile import SpooledTemporaryFile

with SpooledTemporaryFile(mode="w", max_size=1024) as temp:
    temp.writelines(map(str, range(1000)))

With the above code, one might expect that the buffer doesn't grow (much) past 1024 bytes, but it grows to almost three times that size before finally rolling over.

CPython versions tested on:

3.12, 3.13, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-127371 Avoid unbounded growth SpooledTempfile.writelines #127372
• gh-127371: Fix high memory usage in SpooledTemporaryFile.writelines() when handling large iterators #127378
• [3.12] gh-127371 Avoid unbounded growth SpooledTempfile.writelines (GH-127372) #130885
• [3.13] gh-127371 Avoid unbounded growth SpooledTempfile.writelines (GH-127372) #130886