Description
subprocess gained the ability to use close_range() in #84603, however, it's only used as a fallback if procfs is not available, which means that at least on Linux it's almost never used. Its performance benefits are significant (see numbers at the end). I propose to use it by default, with procfs being the second option.
This requires some extra changes because _Py_closerange() was designed to be standalone and tries all available methods, including brute force, and also because it's not required to be async-signal-safe, but we must preserve async-signal-safety of fd closing code on Linux. I'll open a PR with a proposed implementation.
Performance comparison that I did on a test version with the following script (may require you to raise the fd limit in your shell):
import os, subprocess, sys, timeit
from resource import *
soft, hard = getrlimit(RLIMIT_NOFILE)
setrlimit(RLIMIT_NOFILE, (hard, hard))
num_fds, num_iter = map(int, sys.argv[1:3])
for i in range(num_fds):
os.open('/dev/null', os.O_RDONLY)
print(timeit.timeit(lambda: subprocess.run('/bin/true'), number=num_iter))
Before:
./python test.py 100 100
0.11497399304062128
./python test.py 1000 100
0.3999998280778527
./python test.py 10000 100
3.19203062588349
After:
./python test.py 100 100
0.07936713611707091
./python test.py 1000 100
0.09550889488309622
./python test.py 10000 100
0.288825839292258