bpo-42888: Remove PyThread_exit_thread() calls from top-level thread … #24241
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…functions PyThread_exit_thread() uses pthread_exit() on POSIX systems. In glibc, pthread_exit() is implemented in terms of pthread_cancel(), requiring the stack unwinder implemented in libgcc. Further, in dynamically linked applications, calls of pthread_exit() in source code do not make libgcc_s.so a startup dependency: instead, it's lazily loaded by glibc via dlopen() when pthread_exit() is called the first time[1]. All of this makes otherwise finely working CPython fail in multithreaded applications on thread exit if dlopen() fails for any reason. While providing libgcc_s.so is the reponsibility of the user (or their package manager), this hidden dependency has been the source of countless frustrations(e.g. [2]) and, further, dlopen() may fail for other reasons([3]). But most calls to PyThread_exit_thread() in CPython are useless because they're done from the top-level thread function and hence are equivalent to simply returning. So remove all such calls, thereby avoiding the glibc cancellation machinery. The only exception are calls in take_gil() (Python/ceval_gil.h) which serve as a safety net for daemon threads attempting to acquire the GIL after Py_Finalize(). Unless a better model for daemon threads is devised or support for them is removed, those calls have to be preserved since we need to terminate the thread right now without touching any interpreter state. Of course, since PyThread_exit_thread() is a public API, any extension module can still call it and trip over the same issue. [1] https://sourceware.org/legacy-ml/libc-help/2014-07/msg00000.html [2] https://stackoverflow.com/questions/64797838/libgcc-s-so-1-must-be-installed-for-pthread-cancel-to-work [3] https://www.sourceware.org/bugzilla/show_bug.cgi?id=13119
|
This PR is stale because it has been open for 30 days with no activity. |
Those changes preserve the debug output ( |
|
I never ever used PYTHONTHREADDEBUG=1. I just tried: This debug mode produces so many logs that it looks basically useless :-( |
|
I started "Does anyone use threading debug PYTHONTHREADDEBUG=1 env var? Can I remove it?" thread on python-dev: |
|
I created https://bugs.python.org/issue44584 "Deprecate thread debugging PYTHONTHREADDEBUG=1". |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…functions
PyThread_exit_thread() uses pthread_exit() on POSIX systems. In glibc,
pthread_exit() is implemented in terms of pthread_cancel(), requiring
the stack unwinder implemented in libgcc. Further, in dynamically
linked applications, calls of pthread_exit() in source code do not
make libgcc_s.so a startup dependency: instead, it's lazily loaded
by glibc via dlopen() when pthread_exit() is called the first time[1].
All of this makes otherwise finely working CPython fail in multithreaded
applications on thread exit if dlopen() fails for any reason.
While providing libgcc_s.so is the reponsibility of the user
(or their package manager), this hidden dependency has been
the source of countless frustrations(e.g. [2]) and, further,
dlopen() may fail for other reasons([3]). But most calls
to PyThread_exit_thread() in CPython are useless because they're done
from the top-level thread function and hence are equivalent
to simply returning. So remove all such calls, thereby avoiding
the glibc cancellation machinery.
The only exception are calls in take_gil() (Python/ceval_gil.h)
which serve as a safety net for daemon threads attempting
to acquire the GIL after Py_Finalize(). Unless a better model for
daemon threads is devised or support for them is removed,
those calls have to be preserved since we need to terminate
the thread right now without touching any interpreter state.
Of course, since PyThread_exit_thread() is a public API,
any extension module can still call it and trip over the same issue.
[1] https://sourceware.org/legacy-ml/libc-help/2014-07/msg00000.html
[2] https://stackoverflow.com/questions/64797838/libgcc-s-so-1-must-be-installed-for-pthread-cancel-to-work
[3] https://www.sourceware.org/bugzilla/show_bug.cgi?id=13119
https://bugs.python.org/issue42888