bpo-34624: Allow regex for module passed via -W or PYTHONWARNINGS #9358
Conversation
|
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA). Unfortunately our records indicate you have not signed the CLA. For legal reasons we need you to sign this before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue. You can check yourself to see if the CLA has been received. Thanks again for your contribution, we look forward to reviewing it! |
It is nice that pytest supports regular expressions (as documented by Python itself already), but should do so also for cmdline options. It is likely to be fixed in Python eventually. Ref: python/cpython#9358
It is nice that pytest supports regular expressions (as documented by Python itself already), but should do so also for cmdline options. It is likely to be fixed in Python eventually. Ref: python/cpython#9358
|
FYI: rebased and changed it so that |
|
The inaccurate statements (and invalid regex) in the docs resulted in multiple hours of frustration in spyder-ide/qtpy#296 (as well as previous PRs) as to why the documented approach to matching warnings by regexes in |
I'm assuming the main blocking factor is developer bandwidth. By the way, while we're at it, would do you think about automatically appending |
|
Good question; that would certainly be a more helpful behavior to me. However, if we're going to do that, I would suggest either just changing it to match the behavior of I'm not a core dev or anything, but especially if its decided to go ahead with the latter, as a user would be something I think might be worth a What's New entry, since it has non-trivial direct user impact and changes the behavior of the popular |
As implied it should be possible by the documentation.
The previous expression seems to never have been functional at all.
Fixes:
/home/travis/build/python/cpython/Doc/library/warnings.rst:204:
Could not lex literal_block as "python3". Highlighting skipped.
|
This changeset may introduce a new attack vector and a potential REDOS vulnerability. |
|
@tiran Do you think it would be better to only fix the documentation then? Personally, I don't think that's a serious issue. There are more dangerous environment variables to attack around. Personally, my preferred option would be to modify the behaviour of |
That would potentially work for modules without requiring full regex support or escaping for what I would imagine are a large majority of use cases, though it would make the syntax and behavior of
Could you explain a little further as to the actual plausible attack scenario? As I understand it, typically ReDoS attacks involve inserting user input into vulnerable regexes, rather than manipulating regexes themselves. An attacker would have to already have control over the user/shell environment in which Python is running sufficient to set arbitrary environment variables in order to exploit this via the env var, which, generally speaking, would expose much more serious issues than this, I would think. Also, this would need to be weighed against the risk of developing silencing valid warnings (including those that may result in a security or service impact) either unintentionally or necessarily due to the limited expressiveness without regexes. |
|
Ok. At least the incorrect documentation seems to have been fixed at some point, no longer stating examples with (broken) regexes for PYTHONWARNINGS. I'm going to close this, as there doesn't seem to be any upstream capacity to deal with this if received as thirdparty PR. Edit: Actually, the documentation is still broken in parts, but I'll leave that to you. |
|
Its rather disappointing that this was summarily closed and the branch deleted after substantial discussion, given as explained this is a pretty serious limitation that results in substantially over-broad warning silencing, the proposed alternate approach would avoid the potential complexity and security issues raised, and in practice this makes potentially noisy but useful warnings, like |
Unfortunately, checking for strings without context is exceedingly prone to false positives, while missing anything that indirectly opens a file. Python 3.10 has a feature to warn about this though -- and it uses a runtime check which runs at the same time that the code fails to open files in the broken Windows locale. Set this up automatically when running the testsuite. Sadly, Python's builtin feature to change the warning level, e.g. by setting EncodingWarning to error at startup, is utterly broken if you want to limit it to only certain modules. This is tracked in order to be more efficiently ignored at https://bugs.python.org/issue34624 and python/cpython#9358 It is also very trigger happy and passing stuff around via environment variable either messes with the testsuite, or with thirdparty programs which are implemented in python *such as lots of gnome*, or perhaps both. Instead, add runtime code to meson itself, to add a hidden "feature". In the application source code, running the 'warnings' module, you can actually get the expected behavior that $PYTHONWARNINGS doesn't have. So check for a magic testsuite variable every time meson starts up, and if it does, then go ahead and initialize a warnings filter that makes EncodingWarning fatal, but *only* when triggered via Meson and not arbitrary subprocess scripts.
Unfortunately, checking for strings without context is exceedingly prone to false positives, while missing anything that indirectly opens a file. Python 3.10 has a feature to warn about this though -- and it uses a runtime check which runs at the same time that the code fails to open files in the broken Windows locale. Set this up automatically when running the testsuite. Sadly, Python's builtin feature to change the warning level, e.g. by setting EncodingWarning to error at startup, is utterly broken if you want to limit it to only certain modules. This is tracked in order to be more efficiently ignored at https://bugs.python.org/issue34624 and python/cpython#9358 It is also very trigger happy and passing stuff around via environment variable either messes with the testsuite, or with thirdparty programs which are implemented in python *such as lots of gnome*, or perhaps both. Instead, add runtime code to meson itself, to add a hidden "feature". In the application source code, running the 'warnings' module, you can actually get the expected behavior that $PYTHONWARNINGS doesn't have. So check for a magic testsuite variable every time meson starts up, and if it does, then go ahead and initialize a warnings filter that makes EncodingWarning fatal, but *only* when triggered via Meson and not arbitrary subprocess scripts.
Unfortunately, checking for strings without context is exceedingly prone to false positives, while missing anything that indirectly opens a file. Python 3.10 has a feature to warn about this though -- and it uses a runtime check which runs at the same time that the code fails to open files in the broken Windows locale. Set this up automatically when running the testsuite. Sadly, Python's builtin feature to change the warning level, e.g. by setting EncodingWarning to error at startup, is utterly broken if you want to limit it to only certain modules. This is tracked in order to be more efficiently ignored at https://bugs.python.org/issue34624 and python/cpython#9358 It is also very trigger happy and passing stuff around via environment variable either messes with the testsuite, or with thirdparty programs which are implemented in python *such as lots of gnome*, or perhaps both. Instead, add runtime code to meson itself, to add a hidden "feature". In the application source code, running the 'warnings' module, you can actually get the expected behavior that $PYTHONWARNINGS doesn't have. So check for a magic testsuite variable every time meson starts up, and if it does, then go ahead and initialize a warnings filter that makes EncodingWarning fatal, but *only* when triggered via Meson and not arbitrary subprocess scripts.
Unfortunately, checking for strings without context is exceedingly prone to false positives, while missing anything that indirectly opens a file. Python 3.10 has a feature to warn about this though -- and it uses a runtime check which runs at the same time that the code fails to open files in the broken Windows locale. Set this up automatically when running the testsuite. Sadly, Python's builtin feature to change the warning level, e.g. by setting EncodingWarning to error at startup, is utterly broken if you want to limit it to only certain modules. This is tracked in order to be more efficiently ignored at https://bugs.python.org/issue34624 and python/cpython#9358 It is also very trigger happy and passing stuff around via environment variable either messes with the testsuite, or with thirdparty programs which are implemented in python *such as lots of gnome*, or perhaps both. Instead, add runtime code to meson itself, to add a hidden "feature". In the application source code, running the 'warnings' module, you can actually get the expected behavior that $PYTHONWARNINGS doesn't have. So check for a magic testsuite variable every time meson starts up, and if it does, then go ahead and initialize a warnings filter that makes EncodingWarning fatal, but *only* when triggered via Meson and not arbitrary subprocess scripts.
|
I will leave this here, where we basically came to the same conclusion that the current https://discuss.python.org/t/how-to-filter-pipdeprecationwarning/21571/1 |
Enables passing in regexes for the module part via
-WorPYTHONWARNINGSas implied it should be possible by the documentation.https://bugs.python.org/issue34624
The same should should probably be done for the message part as this is implied to work as regex as well.
https://bugs.python.org/issue34624