Skip to content

test_doctest cannot be run directly: let's remove this option #111348

Closed
@sobolevn

Description

@sobolevn

Bug report

When running ./python.exe Lib/test/test_doctest.py you get:

» ./python.exe Lib/test/test_doctest.py   
.....................................F................FF.........
======================================================================
FAIL: basics (__main__.test_DocTestFinder)
Doctest: __main__.test_DocTestFinder.basics
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/doctest.py", line 2353, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for __main__.test_DocTestFinder.basics
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 445, in basics

----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 469, in __main__.test_DocTestFinder.basics
Failed example:
    print(tests)  # doctest: +ELLIPSIS
Expected:
    [<DocTest sample_func from test_doctest.py:32 (1 example)>]
Got:
    [<DocTest sample_func from /Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py:32 (1 example)>]
----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 576, in __main__.test_DocTestFinder.basics
Failed example:
    for t in tests:
        print('%2s  %s' % (len(t.examples), t.name))
Expected:
     1  some_module
     3  some_module.SampleClass
     3  some_module.SampleClass.NestedClass
     1  some_module.SampleClass.NestedClass.__init__
     1  some_module.SampleClass.__init__
     1  some_module.SampleClass.a_cached_property
     2  some_module.SampleClass.a_classmethod
     1  some_module.SampleClass.a_classmethod_property
     1  some_module.SampleClass.a_property
     1  some_module.SampleClass.a_staticmethod
     1  some_module.SampleClass.double
     1  some_module.SampleClass.get
     1  some_module.__test__.c
     2  some_module.__test__.d
     1  some_module.sample_func
Got:
     1  some_module
     1  some_module.__test__.c
     2  some_module.__test__.d


======================================================================
FAIL: test_pdb_set_trace (__main__)
Doctest: __main__.test_pdb_set_trace
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/doctest.py", line 2353, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for __main__.test_pdb_set_trace
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 1984, in test_pdb_set_trace

----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 2043, in __main__.test_pdb_set_trace
Failed example:
    try:
        runner.run(test)
    finally:
        sys.stdin = real_stdin
Expected:
    --Return--
    > <doctest test.test_doctest.test_pdb_set_trace[7]>(3)calls_set_trace()->None
    -> import pdb; pdb.set_trace()
    (Pdb) print(y)
    2
    (Pdb) up
    > <doctest foo-bar@baz[1]>(1)<module>()
    -> calls_set_trace()
    (Pdb) print(x)
    1
    (Pdb) continue
    TestResults(failed=0, attempted=2)
Got:
    --Return--
    > <doctest __main__.test_pdb_set_trace[7]>(3)calls_set_trace()->None
    -> import pdb; pdb.set_trace()
    (Pdb) print(y)
    2
    (Pdb) up
    > <doctest foo-bar@baz[1]>(1)<module>()
    -> calls_set_trace()
    (Pdb) print(x)
    1
    (Pdb) continue
    TestResults(failed=0, attempted=2)


======================================================================
FAIL: test_pdb_set_trace_nested (__main__)
Doctest: __main__.test_pdb_set_trace_nested
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/doctest.py", line 2353, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for __main__.test_pdb_set_trace_nested
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 2118, in test_pdb_set_trace_nested

----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 2154, in __main__.test_pdb_set_trace_nested
Failed example:
    try:
        runner.run(test)
    finally:
        sys.stdin = real_stdin
    # doctest: +REPORT_NDIFF
Differences (ndiff with -expected +actual):
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(5)calls_set_trace()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(5)calls_set_trace()
    ?             ^^^^^^^
      -> self.f1()
      (Pdb) print(y)
      1
      (Pdb) step
      --Call--
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(7)f1()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(7)f1()
    ?             ^^^^^^^
      -> def f1(self):
      (Pdb) step
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(8)f1()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(8)f1()
    ?             ^^^^^^^
      -> x = 1
      (Pdb) step
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(9)f1()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(9)f1()
    ?             ^^^^^^^
      -> self.f2()
      (Pdb) step
      --Call--
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(11)f2()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(11)f2()
    ?             ^^^^^^^
      -> def f2(self):
      (Pdb) step
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(12)f2()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(12)f2()
    ?             ^^^^^^^
      -> z = 1
      (Pdb) step
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(13)f2()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(13)f2()
    ?             ^^^^^^^
      -> z = 2
      (Pdb) print(z)
      1
      (Pdb) up
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(9)f1()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(9)f1()
    ?             ^^^^^^^
      -> self.f2()
      (Pdb) print(x)
      1
      (Pdb) up
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(5)calls_set_trace()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(5)calls_set_trace()
    ?             ^^^^^^^
      -> self.f1()
      (Pdb) print(y)
      1
      (Pdb) up
      > <doctest foo-bar@baz[1]>(1)<module>()
      -> calls_set_trace()
      (Pdb) print(foo)
      *** NameError: name 'foo' is not defined
      (Pdb) continue
      TestResults(failed=0, attempted=2)


----------------------------------------------------------------------
Ran 65 tests in 0.942s

FAILED (failures=3)

These issues are not easy to fix. Moreover, since they are literally written in docstrings, it is not very readable.

The second problem is that -c option is claimed to be supported:

def test_coverage(coverdir):
trace = import_helper.import_module('trace')
tracer = trace.Trace(ignoredirs=[sys.base_prefix, sys.base_exec_prefix,],
trace=0, count=1)
tracer.run('test_main()')
r = tracer.results()
print('Writing coverage results...')
r.write_results(show_missing=True, summary=True,
coverdir=coverdir)
if __name__ == '__main__':
if '-c' in sys.argv:
test_coverage('/tmp/doctest.cover')
else:
unittest.main()

However:

  • test_main is not defined, so it fails
  • When changed to unittest.main() (with all of the other test failures fixed), it still does not work on macos sonoma, no file is created, nothing

I propose to:

  • raise an exception that running test_doctest directly is not supported
  • remove test_coverage

Linked PRs

Metadata

Metadata

Assignees

Labels

testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions