Race condition in pydoc._start_server

[itamaro]

Bug report

Bug description:

There's a race condition in pydoc._start_server - when _start_server() returns, we should get an object with a valid docserver attribute (set here). However, the function only checks that the serving attribute is truthy before returning (here).

The race is triggered if setting serving to True here happens before setting the docserver attribute here -- we observed this happening frequently in the Cinder ASAN test suite (originally observed and fixed by @jbower).

The race can be forced to happen by forcing a context switch after setting self.serving = True:

diff --git a/Lib/pydoc.py b/Lib/pydoc.py
index b0193b4a851..117a1dc8369 100755
--- a/Lib/pydoc.py
+++ b/Lib/pydoc.py
@@ -2511,6 +2511,7 @@ def run(self):

         def ready(self, server):
             self.serving = True
+            time.sleep(0.1)
             self.host = server.host
             self.port = server.server_port
             self.url = 'http://%s:%d/' % (self.host, self.port)

and running the test_pydoc.PydocServerTest.test_server test, which would fail and hang:

$ ./python.exe -m test test_pydoc -v -m 'test.test_pydoc.test_pydoc.PydocServerTest.test_server'
== CPython 3.13.0a4+ (heads/main-dirty:06565090339, Feb 29 2024, 11:49:21) [Clang 15.0.0 (clang-1500.1.0.2.5)]
== macOS-14.3.1-arm64-arm-64bit-Mach-O little-endian
== Python build: debug
== cwd: /Users/itamaro/work/pyexe/main-dbg/build/test_python_worker_66701æ
== CPU count: 12
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 792156149
0:00:00 load avg: 5.34 Run 1 test sequentially
0:00:00 load avg: 5.34 [1/1] test_pydoc.test_pydoc
test_server (test.test_pydoc.test_pydoc.PydocServerTest.test_server) ... ERROR
test_server (test.test_pydoc.test_pydoc.PydocServerTest.test_server) ... ERROR
Warning -- threading_cleanup() failed to clean up threads in 1.0 seconds
Warning --   before: thread count=0, dangling=1
Warning --   after: thread count=1, dangling=2
Warning -- Dangling thread: <_MainThread(MainThread, started 7977835584)>
Warning -- Dangling thread: <ServerThread(Thread-1, started 6150828032)>

======================================================================
ERROR: test_server (test.test_pydoc.test_pydoc.PydocServerTest.test_server)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/itamaro/work/cpython/Lib/test/test_pydoc/test_pydoc.py", line 1823, in test_server
    self.assertIn('localhost', serverthread.url)
                               ^^^^^^^^^^^^^^^^
AttributeError: 'ServerThread' object has no attribute 'url'

======================================================================
ERROR: test_server (test.test_pydoc.test_pydoc.PydocServerTest.test_server)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/itamaro/work/cpython/Lib/test/test_pydoc/test_pydoc.py", line 1821, in <lambda>
    lambda: serverthread.stop() if serverthread.serving else None
            ~~~~~~~~~~~~~~~~~^^
  File "/Users/itamaro/work/cpython/Lib/pydoc.py", line 2521, in stop
    self.docserver.quit = True
    ^^^^^^^^^^^^^^
AttributeError: 'ServerThread' object has no attribute 'docserver'

----------------------------------------------------------------------
Ran 1 test in 1.321s

FAILED (errors=2)
Warning -- threading._dangling was modified by test_pydoc.test_pydoc
Warning --   Before: {<weakref at 0x10499c280; to '_MainThread' at 0x102eb0a10>}
Warning --   After:  {<weakref at 0x10550f3f0; to '_MainThread' at 0x102eb0a10>, <weakref at 0x10550f380; to 'ServerThread' at 0x1049b81f0>}
test test_pydoc.test_pydoc failed
test_pydoc.test_pydoc failed (2 errors)

== Tests result: FAILURE ==

1 test failed:
    test_pydoc.test_pydoc

Total duration: 1.4 sec
Total tests: run=1 (filtered)
Total test files: run=1/1 (filtered) failed=1
Result: FAILURE

The race can be fixed by making sure the docserver attribute is also set before returning (PR incoming).

CPython versions tested on:

3.8, 3.10, 3.12, CPython main branch

Operating systems tested on:

Linux, macOS

Linked PRs

• gh-116143: Fix race condition in pydoc _start_server #116144
• [3.12] gh-116143: Fix race condition in pydoc _start_server (GH-116144) #116415
• [3.11] gh-116143: Fix race condition in pydoc _start_server (GH-116144) #116416