gh-130167: Minor textwrap.dedent() optimization
#131925
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
All commit authors signed the Contributor License Agreement. |
|
It seems for the benchmarks handling large text, there's a consistent slowdown. Could you plot a chart of running the function with inputs of different sizes? Perhaps take the unicodeobject file and profile the first 500 lines, then the first 1000, 1500, etc, for this PR and the version using min/max. A |
|
It stays a consistently slower by 5-8% except for long lines test
|
|
From additional testing, this way is slightly faster in the ranges where the lines are between 1-18.
after analyzing all the function documentation in the Python codebase the average documentation length seems to be the following https://gist.github.com/Marius-Juston/08c574901317ee40837ab87ce0855685:
( excluding all 1 line comments ) so technically, this implementation is slightly faster for the average documentation length ( now is this performance increase really worth it, no probably not lol ) |
eendebakpt
reviewed
Mar 31, 2025
Lib/textwrap.py
Outdated
| l1 = min(non_blank_lines, default='') | ||
| l2 = max(non_blank_lines, default='') | ||
| margin = 0 | ||
| l1 = None |
There was a problem hiding this comment.
Suggested change
| l1 = None | |
| l1 = '' |
You can then skip the is None check in the loop. Similar for l2 (but you have to pick the default with care)
eendebakpt
reviewed
Mar 31, 2025
Lib/textwrap.py
Outdated
| if line and not line.isspace(): | ||
| if l1 is None or line < l1: | ||
| l1 = line | ||
| if l2 is None or line > l2: |
There was a problem hiding this comment.
Once you skip the if None check, this if can be an elseif
|
I'm against this change not only because it's harder to read but also because it only offers an overall <10% speed-up. In general, we accept optimizations exceeding that threshold. |
|
@eendebakpt with the new changes:
Small values
Large file
So on the large files we get 0 improvement, a 13% slight improvement on the synthetic test benches and on smaller values |
Agreed that this optimization is most likely not necessary, though this is very slightly more efficient memory wise since it does not have a |
eendebakpt
reviewed
Mar 31, 2025
Comment on lines
+435
to
+453
| val = False | ||
| for i, line in enumerate(lines): | ||
| # Compute min + max concurrently + normalize others | ||
| if line and not line.isspace(): | ||
| if val: | ||
| if line < l1: | ||
| l1 = line | ||
| elif line > l2: | ||
| l2 = line | ||
| else: | ||
| val = True | ||
| l1 = l2 = line | ||
|
|
||
| else: | ||
| lines[i] = '' | ||
|
|
||
| if not val or not l1: | ||
| return '\n'.join(lines) | ||
|
|
There was a problem hiding this comment.
Suggested change
| val = False | |
| for i, line in enumerate(lines): | |
| # Compute min + max concurrently + normalize others | |
| if line and not line.isspace(): | |
| if val: | |
| if line < l1: | |
| l1 = line | |
| elif line > l2: | |
| l2 = line | |
| else: | |
| val = True | |
| l1 = l2 = line | |
| else: | |
| lines[i] = '' | |
| if not val or not l1: | |
| return '\n'.join(lines) | |
| l1 = 'z' | |
| l2 = '' | |
| for i, line in enumerate(lines): | |
| # Compute min + max concurrently + normalize others | |
| if line and not line.isspace(): | |
| if line < l1: | |
| l1 = line | |
| if line > l2: | |
| l2 = line | |
| else: | |
| lines[i] = '' | |
| if not l1: | |
| return '\n'.join(lines) | |
| margin = 0 | |
| for margin, c in enumerate(l2): | |
| if c != l1[margin] or c not in ' \t': | |
| break | |
This saves chechking the val. Performance gain is only a bit though.
I agree with the other reviewers: unless we find out a way to make this much faster, the small gains are not worth the change.
@Marius-Juston Thanks for putting in the work to get all the performance numbers!
ZeroIntensity
requested changes
Mar 31, 2025
There was a problem hiding this comment.
Thanks for getting the benchmarks, sometimes its hard to get contributors to quote numbers, and you've gone above and beyond with benchmarking :).
But unfortunately, the numbers make me -1 on this.
min/maxare builtins that are written in C; they're pretty fast, especially compared to multiple lines of Python code. If we're concerned about those being slow, let's try to optimize those on the C level instead.- There is a significant amount of added complexity here, for seemingly not much benefit.
- Optimizing Python code by shifting things around is generally not a good idea, because when the interpreter changes, those optimizations become obsolete. We should definitely not be quoting memory usage, number of (pointer) copies, or number of lookups; those are all things that change on an interpreter-wide level.
I've learned that it can be hard to know when to stop painting. I think we should put down the brush, and enjoy the optimization that's already landed 🙂.
|
Makes sense. I am just having so much fun doing this it is definitely hard to stop lol! |
|
Closing this down then! ( I will be putting a feature request for the |
I think it's already been proposed but rejected so search through issues and https://discuss.python.org/c/ideas/6 first. |
I think I found it, the https://discuss.python.org/t/minmax-function-alongside-min-and-max/2834/73 issue got closed down. It seems like it got stale and just ignored? A shame. |
|
Feel free to open a new thread referencing the one you found. Personally I'd suggest A |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Minor optimization to @AA-Turner #131792 where you compute the min and max of the non_blank_lines simultaneously, removes additional use of
line.isspace()as well.reuses #130167