Skip to content

gh-117151: IO performance improvement, increase io.DEFAULT_BUFFER_SIZE to 128k #118144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Mar 7, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
adjust news and comments
  • Loading branch information
rmmancom committed Feb 6, 2025
commit eaa0df69b0f47774476593f9c7ab12c67ac8fa26
4 changes: 1 addition & 3 deletions Lib/_pyio.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,7 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
given, the default buffering policy works as follows:

* Binary files are buffered in fixed-size chunks; the size of the buffer
is set to `max(io.DEFAULT_BUFFER_SIZE, st_blksize)` using a heuristic
trying to determine the underlying device's "block size" when available
and falling back on `io.DEFAULT_BUFFER_SIZE`.
is the maximum of the DEFAULT_BUFFER_SIZE and the device block size.
On most systems, the buffer will typically be 131072 bytes long.

* "Interactive" text files (files for which isatty() returns True)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Increase ``io.DEFAULT_BUFFER_SIZE`` from 8k to 128k and adjust :func:`open` on
platforms where ``fstat`` provides a ``st_blksize`` field (such as Linux) to use
``max(io.DEFAULT_BUFFER_SIZE, device block size)`` rather than always using the
device block size. This should improve I/O performance.
platforms where :meth:`os.fstat` provides a ``st_blksize`` field (such as Linux)
to use ``max(io.DEFAULT_BUFFER_SIZE, device block size)`` rather than always
using the device block size. This should improve I/O performance.
Patch by Romain Morotti.
8 changes: 3 additions & 5 deletions Modules/_io/_iomodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -132,10 +132,8 @@ the size of a fixed-size chunk buffer. When no buffering argument is
given, the default buffering policy works as follows:

* Binary files are buffered in fixed-size chunks; the size of the buffer
is set to `max(io.DEFAULT_BUFFER_SIZE, st_blksize)` using a heuristic
trying to determine the underlying device's "block size" when available
and falling back on `io.DEFAULT_BUFFER_SIZE`.
On most systems, the buffer will typically be 131072 bytes long.
is the maximum of the DEFAULT_BUFFER_SIZE and the device block size.
On most systems, the buffer will typically be 131072 bytes long.

* "Interactive" text files (files for which isatty() returns True)
use line buffering. Other text files use the policy described above
Expand Down Expand Up @@ -201,7 +199,7 @@ static PyObject *
_io_open_impl(PyObject *module, PyObject *file, const char *mode,
int buffering, const char *encoding, const char *errors,
const char *newline, int closefd, PyObject *opener)
/*[clinic end generated code: output=aefafc4ce2b46dc0 input=bac1cd70f431fe9a]*/
/*[clinic end generated code: output=aefafc4ce2b46dc0 input=105f6f1cb63368c4]*/
{
size_t i;

Expand Down
8 changes: 3 additions & 5 deletions Modules/_io/clinic/_iomodule.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.