Skip to content

regrtest: allow to intermix --match and --ignore options #110918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
serhiy-storchaka opened this issue Oct 16, 2023 · 0 comments
Closed

regrtest: allow to intermix --match and --ignore options #110918

serhiy-storchaka opened this issue Oct 16, 2023 · 0 comments
Labels
3.11 only security fixes 3.12 only security fixes 3.13 bugs and security fixes tests Tests in the Lib/test dir type-feature A feature request or enhancement

Comments

@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented Oct 16, 2023 •
edited by bedevere-app bot
Loading

Feature or enhancement

Currently you can use --match and --ignore options for positive and negative filtering of test cases by name. But negative patterns always win. So you can select a class of tests and then exclude some tests from it, but you cannot exclude a class of tests except some tests.

Many programs that have similar options (can also be named --include/--exclude) use different algorithm. They apply rules in the order, and the last rule wins. I propose to implement this in Python regrtests too. Examples:

Run FileTests tests in test_os, excluding test_write:

./python -m test test_os -m FileTests -i test_write

Run all tests in test_os, excluding FileTests tests, but including test_write:

./python -m test test_os -i FileTests -m test_write

And, of course, any combinations are valid. This applies also to options that read patterns from files: --matchfile and --ignorefile. The implementation is actually somewhat simpler than the current one.

Linked PRs
@serhiy-storchaka serhiy-storchaka added type-feature A feature request or enhancement tests Tests in the Lib/test dir 3.11 only security fixes 3.12 only security fixes 3.13 bugs and security fixes labels Oct 16, 2023
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Oct 16, 2023
@serhiy-storchaka
pythongh-110918: regrtest: allow to intermix --match and --ignore opt…
40e37f1 
…ions

Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
@bedevere-app bedevere-app bot mentioned this issue Oct 16, 2023
Merged
serhiy-storchaka added a commit that referenced this issue Oct 21, 2023
@serhiy-storchaka
gh-110918: regrtest: allow to intermix --match and --ignore options (G…
9a1fe09 
…H-110919)

Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Oct 21, 2023
@serhiy-storchaka
[3.12] pythongh-110918: regrtest: allow to intermix --match and --ign…
72c8c61 
…ore options (pythonGH-110919)

Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored..
(cherry picked from commit 9a1fe09)

Co-authored-by: Serhiy Storchaka <[email protected]>
@bedevere-app bedevere-app bot mentioned this issue Oct 21, 2023
Merged
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Oct 21, 2023
@serhiy-storchaka
[3.11] pythongh-110918: regrtest: allow to intermix --match and --ign…
b7cf01e 
…ore options (pythonGH-110919)

Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored..
(cherry picked from commit 9a1fe09)

Co-authored-by: Serhiy Storchaka <[email protected]>
@bedevere-app bedevere-app bot mentioned this issue Oct 21, 2023
Merged
serhiy-storchaka added a commit that referenced this issue Oct 21, 2023
@serhiy-storchaka
[3.11] gh-110918: regrtest: allow to intermix --match and --ignore op…
4222dd9 
…tions (GH-110919) (GH-111168)

Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
(cherry picked from commit 9a1fe09)
serhiy-storchaka added a commit that referenced this issue Oct 21, 2023
@serhiy-storchaka
[3.12] gh-110918: regrtest: allow to intermix --match and --ignore op…
1ea9302 
…tions (GH-110919) (GH-111167)

Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
(cherry picked from commit 9a1fe09)
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
@serhiy-storchaka @aisk
pythongh-110918: regrtest: allow to intermix --match and --ignore opt…
f32f0eb 
…ions (pythonGH-110919)

Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
vstinner added a commit to vstinner/cpython that referenced this issue Mar 13, 2024
@vstinner
pythongh-110918: Fix side effects of regrtest test_match_tests()
5cd5355 
test_match_tests now saves and restores patterns.

Add get_match_tests() function to libregrtest.filter.

Previously, running test_regrtest multiple times in a row only ran
tests once: "./python -m test test_regrtest -R 3:3.
@bedevere-app bedevere-app bot mentioned this issue Mar 13, 2024
Merged
vstinner added a commit that referenced this issue Mar 13, 2024
@vstinner
gh-110918: Fix side effects of regrtest test_match_tests() (#116718)
612f1ec 
test_match_tests now saves and restores patterns.

Add get_match_tests() function to libregrtest.filter.

Previously, running test_regrtest multiple times in a row only ran
tests once: "./python -m test test_regrtest -R 3:3.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Mar 13, 2024
@vstinner @miss-islington
pythongh-110918: Fix side effects of regrtest test_match_tests() (pyt…
03d7815 
…honGH-116718)

test_match_tests now saves and restores patterns.

Add get_match_tests() function to libregrtest.filter.

Previously, running test_regrtest multiple times in a row only ran
tests once: "./python -m test test_regrtest -R 3:3.
(cherry picked from commit 612f1ec)

Co-authored-by: Victor Stinner <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Mar 13, 2024
@vstinner @miss-islington
pythongh-110918: Fix side effects of regrtest test_match_tests() (pyt…
406ae57 
…honGH-116718)

test_match_tests now saves and restores patterns.

Add get_match_tests() function to libregrtest.filter.

Previously, running test_regrtest multiple times in a row only ran
tests once: "./python -m test test_regrtest -R 3:3.
(cherry picked from commit 612f1ec)

Co-authored-by: Victor Stinner <[email protected]>
This was referenced Mar 13, 2024
Merged
Merged
vstinner added a commit that referenced this issue Mar 13, 2024
@miss-islington @vstinner
[3.12] gh-110918: Fix side effects of regrtest test_match_tests() (GH…
283dda8 
…-116718) (#116726)

gh-110918: Fix side effects of regrtest test_match_tests() (GH-116718)

test_match_tests now saves and restores patterns.

Add get_match_tests() function to libregrtest.filter.

Previously, running test_regrtest multiple times in a row only ran
tests once: "./python -m test test_regrtest -R 3:3.
(cherry picked from commit 612f1ec)

Co-authored-by: Victor Stinner <[email protected]>
vstinner added a commit that referenced this issue Mar 13, 2024
@miss-islington @vstinner
[3.11] gh-110918: Fix side effects of regrtest test_match_tests() (GH…
cde47e2 
…-116718) (#116727)

gh-110918: Fix side effects of regrtest test_match_tests() (GH-116718)

test_match_tests now saves and restores patterns.

Add get_match_tests() function to libregrtest.filter.

Previously, running test_regrtest multiple times in a row only ran
tests once: "./python -m test test_regrtest -R 3:3.
(cherry picked from commit 612f1ec)

Co-authored-by: Victor Stinner <[email protected]>
vstinner added a commit to vstinner/cpython that referenced this issue Mar 20, 2024
@vstinner
pythongh-110918: Fix side effects of regrtest test_match_tests() (pyt…
f7230ee 
…hon#116718)

test_match_tests now saves and restores patterns.

Add get_match_tests() function to libregrtest.filter.

Previously, running test_regrtest multiple times in a row only ran
tests once: "./python -m test test_regrtest -R 3:3.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-110918: Fix side effects of regrtest test_match_tests() (pyt…
a40f093 
…hon#116718)

test_match_tests now saves and restores patterns.

Add get_match_tests() function to libregrtest.filter.

Previously, running test_regrtest multiple times in a row only ran
tests once: "./python -m test test_regrtest -R 3:3.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-110918: Fix side effects of regrtest test_match_tests() (pyt…
0109a62 
…hon#116718)

test_match_tests now saves and restores patterns.

Add get_match_tests() function to libregrtest.filter.

Previously, running test_regrtest multiple times in a row only ran
tests once: "./python -m test test_regrtest -R 3:3.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@serhiy-storchaka @Glyphack
pythongh-110918: regrtest: allow to intermix --match and --ignore opt…
ef7b6e4 
…ions (pythonGH-110919)

Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.11 only security fixes 3.12 only security fixes 3.13 bugs and security fixes tests Tests in the Lib/test dir type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@serhiy-storchaka