Skip to content

subprocess: Prefer close_range() to procfs-based fd closing #92301

Closed
@izbyshev

Description

@izbyshev

subprocess gained the ability to use close_range() in #84603, however, it's only used as a fallback if procfs is not available, which means that at least on Linux it's almost never used. Its performance benefits are significant (see numbers at the end). I propose to use it by default, with procfs being the second option.

This requires some extra changes because _Py_closerange() was designed to be standalone and tries all available methods, including brute force, and also because it's not required to be async-signal-safe, but we must preserve async-signal-safety of fd closing code on Linux. I'll open a PR with a proposed implementation.

Performance comparison that I did on a test version with the following script (may require you to raise the fd limit in your shell):

import os, subprocess, sys, timeit
from resource import *

soft, hard = getrlimit(RLIMIT_NOFILE)
setrlimit(RLIMIT_NOFILE, (hard, hard))

num_fds, num_iter = map(int, sys.argv[1:3])

for i in range(num_fds):
    os.open('/dev/null', os.O_RDONLY)

print(timeit.timeit(lambda: subprocess.run('/bin/true'), number=num_iter))

Before:

./python test.py 100 100
0.11497399304062128
./python test.py 1000 100
0.3999998280778527
./python test.py 10000 100
3.19203062588349

After:

./python test.py 100 100
0.07936713611707091
./python test.py 1000 100
0.09550889488309622
./python test.py 10000 100
0.288825839292258

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagetype-featureA feature request or enhancement

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions