SystemError for pep695 type parameter with the same name as the inner class

[15r10nk]

Bug report

Bug description:

The following script shows the SystemError:

class name_1[name_4]:

    class name_4[name_2](name_4):
        name_4

output (Python 3.12.0rc2+):

SystemError: compiler_lookup_arg(name='name_4') with reftype=3 failed in <generic parameters of name_4>; freevars of code name_4: ('.type_params', 'name_4')

minimizing it further leads some different crashes:
The propably most interresting one is this, which does not look special at all.

class name_1[name_3]:
    class name_4[name_2](name_5):
        pass

output (Python 3.12.0rc2+):

Traceback (most recent call last):
  File "/home/frank/projects/pysource-codegen/bug3.py", line 1, in <module>
    class name_1[name_3]:
  File "/home/frank/projects/pysource-codegen/bug3.py", line 1, in <generic parameters of name_1>
    class name_1[name_3]:
  File "/home/frank/projects/pysource-codegen/bug3.py", line 2, in name_1
    class name_4[name_2](name_5):
  File "/home/frank/projects/pysource-codegen/bug3.py", line 2, in <generic parameters of name_4>
    class name_4[name_2](name_5):
                         ^^^^^^
NameError: name 'name_5' is not defined. Did you mean: 'name_2'?
Modules/gcmodule.c:113: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f2c57a12930
object refcount : 1
object type     : 0x557ad9ef1fe0
object type name: dict
object repr     : {'__module__': '__main__', '__qualname__': 'name_1', '__type_params__': (name_3,)}

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x0000557ada04eba0)

Current thread 0x00007f2c57f21280 (most recent call first):
  Garbage-collecting
  <no Python frame>

@JelleZijlstra I think this is another one for you. Feel free to create a second issue if you think that they are unrelated.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-109219: propagate free vars through type param scopes #109377
• gh-109219: propagate used free vars through type param scopes [v2] #109389
• [3.12] gh-109219: propagate free vars through type param scopes (GH-109377) #109410

[JelleZijlstra]

Thanks. The second one should be fixed by #109123. I'll have to look into what is causing the SystemError on your first example.

[JelleZijlstra]

I spent some time last night trying to get this to work without success. I'll try more later today, but here are my thoughts so far on this example:

class name_1[name_4]:

    class name_4[name_2](name_4):
        name_4

The name_4 in the body of the inner class should refer to the type param of the outer class, because the bodies of inner classes can't see names defined in outer classes, but they can see names defined in outer function-like scopes.

The name_4 in the bases of the inner class should be a NameError. If a name is bound anywhere in a class body, then we look only at the class scope and the global scope, not any intervening scopes. I discuss this in https://jellezijlstra.github.io/pep695 in the section that starts "A few more edge cases came up later." In this case, the name is bound in the class body, but there is no binding yet when the bases of the inner class are evaluated, so next we look at the globals, where there is also no name_4, so we raise a NameError.

cc @carljm, would appreciate if you could see whether you agree with this diagnosis, and if you can see a simple way to implement the desired behavior in the compiler.

[carljm]

Your analysis looks correct to me! I'll take a look at implementation...

[JelleZijlstra]

I suppose the problem is that inside the annotation scope for the inner class, name_4 needs to be present with two different meanings: a GLOBAL_IMPLICIT for the use in the annotation scope itself, and a free variable, so that it can be put in the cell for the class scope. I'm not sure the compiler has a way to do that in general, though there are some cases involving class scopes that do work like this.

[carljm]

A similar example that still reproduces the bug, but simplifies and perhaps clarifies a few things (replace the outer annotation scope with a regular function scope, and use a normal re-definition of the typevar in the class scope instead of circularly naming the inner class the same as the typevar; this example should run successfully rather than raise a NameError):

def f():
    T = str
    class C:
        T = int
        class D[U](T):
            T
    assert int in C.D.__bases__

[carljm]

In my latter example, the T = str in the outer f() function (equivalent to the presence of name_4 in outer class type params in the original example) is key to reproducing the bug. It is this definition of T (not the redefinition in the class) which causes the innermost T (in the body of the inner class) to have scope FREE rather than GLOBAL_IMPLICIT (as it does if you remove the line T = str from my example.)

I haven't quite sorted out yet what should be happening here; I'll come back to it tomorrow.

[carljm]

I'm using this test case to debug, and focusing on the discrepancy between the behavior with and without the [U] type param on the inner class, since the presence of that type param should not change the behavior of the name T. This test case fails with the SystemError as written, and passes if the [U] is removed:

    def test_complex_nested_classes(self):
        code = """
            def f():
                T = str
                class C:
                    T = int
                    class D[U](T):
                        x = T
                return C
        """
        C = run_code(code)["f"]()
        self.assertIn(int, C.D.__bases__)
        self.assertIs(C.D.x, str)

[carljm]

FWIW, I also wrote this PyST_Dump function to help debug this: https://gist.github.com/carljm/ecf76f8d0b4d77b00a5ae79b93d1418a

Maybe we should commit this, ifdef-ed out by default, similar to the dump_* functions in flowgraph.c.

[carljm]

#109377 fixes this case. I'm still working through some scenarios to see whether it needs more tweaks to avoid breaking any other case.

The fix is to use the symbol flag DEF_CLASS_FREE, which permits adding names to freevars in a class scope, so as to propagate the free var from an inner scope up to a wrapping function-like scope that defines the name in a cell, while the actual uses of the name in the class body can still have a scope other than FREE. So the fix here is to consider a can_see_class_scope scope within a class (i.e. the type param scope here) to be equivalent to a class body when it comes to applying the DEF_CLASS_FREE tag in this situation.

I think the PR may still need an adjustment to ensure we only do this if the name is bound in an enclosing scope.

[15r10nk]

hi, I generated some more examples.

I used cpython from commit 21f4e6d.
I have no idea if they are all caused by the same bug, but I hope that they are helpful.

class name_1[**name_0]:
    try:

        class name_3[name_3](name_4=name_0):
            name_0
    except {name_0} as name_5:
        from name_4 import name_0
    else:
        pass
    finally:
        pass

# output:
# Traceback (most recent call last):
#   File "/home/frank/projects/pysource-playground/test_code.py", line 6, in <module>
#     compile(source,sys.argv[1],"exec")
# SystemError: compiler_lookup_arg(name='name_0') with reftype=3 failed in <generic parameters of name_3>; freevars of code name_3: ('.type_params', 'name_0')

class name_1[*name_1]:

    async def name_1[**name_0](name_5: name_1, /): # type: 
        name_1

# output:
# Traceback (most recent call last):
#   File "/home/frank/projects/pysource-playground/test_code.py", line 6, in <module>
#     compile(source,sys.argv[1],"exec")
# SystemError: compiler_lookup_arg(name='name_1') with reftype=3 failed in <generic parameters of name_1>; freevars of code name_1: ('name_1',)

class name_0[name_2]:
    (name_2 := name_1)

    def name_5[*name_4]() -> name_2: # type: ignoresome text
        name_2

# output:
# Traceback (most recent call last):
#   File "/home/frank/projects/pysource-playground/test_code.py", line 6, in <module>
#     compile(source,sys.argv[1],"exec")
# SystemError: compiler_lookup_arg(name='name_2') with reftype=3 failed in <generic parameters of name_5>; freevars of code name_5: ('name_2',)

class name_4[**name_0]:

    class name_0[name_4](**name_0):
        name_0

# output:
# Traceback (most recent call last):
#   File "/home/frank/projects/pysource-playground/test_code.py", line 6, in <module>
#     compile(source,sys.argv[1],"exec")
# SystemError: compiler_lookup_arg(name='name_0') with reftype=3 failed in <generic parameters of name_0>; freevars of code name_0: ('.type_params', 'name_0')

class name_5[**name_5]:

    class name_5[name_1](**name_5):
        {name_4: name_4 for name_1 in name_3 if name_5}

# output:
# Traceback (most recent call last):
#   File "/home/frank/projects/pysource-playground/test_code.py", line 6, in <module>
#     compile(source,sys.argv[1],"exec")
# SystemError: compiler_lookup_arg(name='name_5') with reftype=3 failed in <generic parameters of name_5>; freevars of code name_5: ('.type_params', 'name_5')

class name_0[name_5]:

    @(name_5 := 0)
    class name_3[*name_4](name_5=name_5):
        {name_0 for name_0 in name_4 if name_5}

# output:
# Traceback (most recent call last):
#   File "/home/frank/projects/pysource-playground/test_code.py", line 6, in <module>
#     compile(source,sys.argv[1],"exec")
# SystemError: compiler_lookup_arg(name='name_5') with reftype=3 failed in <generic parameters of name_3>; freevars of code name_3: ('.type_params', 'name_4', 'name_5')

class name_2[*name_0]:
    match something:
        case {**name_4}:
            name_0 = name_3 # type: ignoresome text
    with name_0: # type: ignoresome text

        class name_2[name_5](name_0):
            name_0

# output:
# Traceback (most recent call last):
#   File "/home/frank/projects/pysource-playground/test_code.py", line 6, in <module>
#     compile(source,sys.argv[1],"exec")
# SystemError: compiler_lookup_arg(name='name_0') with reftype=3 failed in <generic parameters of name_2>; freevars of code name_2: ('.type_params', 'name_0')

Background:

I'm currently updating pysource-minimize and pysource-codegen to 3.12.
I created therefor a little test repository https://github.com/15r10nk/pysource-playground, which I used to generate this examples.
This code is still work in progress and the examples might be not as minimal as they could be.
I'm not able to continue my work until next week (family ...).
You can test a new python version with this repository in the meantime if you want. Create an issue in this repository if you have any problems, I will try to help you.

[carljm]

@15r10nk Thank you! At first glance just based on the error messages, I suspect these are all the same bug, but I will turn them into test cases and verify the fix handles all of them.

[carljm]

I tested all of the above cases and #109377 fixes them all; they all compile successfully and then fail as expected at runtime with either NameError or TypeError. I don't think they even need to be added to the PR as separate regression tests, since they all test for the same bug.

[JelleZijlstra]

Merged and backported, thanks @15r10nk for the report and @carljm for the fix!