Bug or new behavior of generics parameterization?

from typing import TypeVar, Generic, Callable


T = TypeVar("T")
P = TypeVar("P")

class Test(Generic[T]): ...


S = Callable[[Test], P]

def test(arg: S[T]): ...

On python3.11 i can't run this code. Code falls on an attempt to parameterize S
Traceback:

Traceback (most recent call last):
  File "C:\Users\KIRILL-1\PycharmProjects\dataclass_factory\exp.py", line 12, in <module>                  
    def test(arg: S[T]): ...                                                                               
                  ~^^^                                                                                     
  File "C:\Users\KIRILL-1\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 360, in inner       
    return cached(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\KIRILL-1\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 1391, in __getitem__
    new_args = self._determine_new_args(args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\KIRILL-1\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 1440, in _determine_new_args
    subargs.append(new_arg_by_param[x])
                   ~~~~~~~~~~~~~~~~^^^
KeyError: ~T

How i understand, in 3.11, an attempt is being made to parameterize Test in S, but in 3.10 - not
Is it new behavior or a bug?

PR: gh-98852: Fix subscription of type aliases #98920

PR: [3.11] gh-98852: Fix subscription of type aliases (GH-98920) #98952

PR: [3.10] gh-98852: Fix subscription of types.GenericAlias instances (GH-98920) #98969

This is how it used to work in 3.10: https://github.com/python/cpython/blame/85f88f63d96b07208c98a725391af7cb710fe06b/Lib/typing.py#L1060-L1081

>>> from typing import TypeVar, Generic, Callable
>>> 
>>> 
>>> T = TypeVar("T")
>>> P = TypeVar("P")
>>> 
>>> class Test(Generic[T]): ...
... 
>>> 
>>> S = Callable[[Test], P]
>>> 
>>> def test(arg: S[T]): ...
... 
>>> test.__annotations__
{'arg': typing.Callable[[__main__.Test], ~T]}

Clearly looks like a bug to me. I am investigating what is going on :)

I know about how it works in 3.10 =)
My simple solution:

...
            if substfunc:
                new_arg = substfunc(new_arg_by_param[old_arg])
            elif not isinstance(old_arg, (_GenericAlias, GenericAlias, types.UnionType)):
                new_arg = old_arg
            else:
 ...

With that, code above works fine.

>>> test.__annotations__
{'arg': typing.Callable[[__main__.Test], ~T]}

What you think about this?

Yes, looks like it might work. Previously the same logic was guarded by this check: https://github.com/python/cpython/blame/85f88f63d96b07208c98a725391af7cb710fe06b/Lib/typing.py#L1071

Would you like to send a PR with this proposal? :)

@JelleZijlstra

Could we first find which PR introduced the bug?

Also, @JelleZijlstra ^^

It is bpo-44796 (#31143)

Indeed, there is a bug in _collect_parameters which causes to skipping generic types.

Yet one interesting example:

>>> class Test(Generic[T]): ...
... 
>>> Test.__parameters__
(~T,)
>>> class Test2(Test): ...
... 
>>> Test2.__parameters__
()

But after fixing this code I got other test failure. And it seems that we have contradictory tests. Test in test_complex_subclasses expects that we can successfully create a class inheriting from generic base and Generic[].

cpython/Lib/test/test_typing.py

Lines 3898 to 3900 in 018b248

    
           # see gh-94607: this fails in that bug 
        
           class Sub(Base, Generic[T]): 
        
               ...

But tests in test_generic_errors and test_generic_inheritance expect that Generic[] should include all typing variables, otherwise the class creation should fail.

cpython/Lib/test/test_typing.py

Lines 3047 to 3048 in 018b248

    
           with self.assertRaises(TypeError): 
        
               class MyGeneric(List[T], Generic[S]): ...

cpython/Lib/test/test_typing.py

Lines 6492 to 6494 in 018b248

    
           with self.assertRaises(TypeError): 
        
               class Point3D(Point2DGeneric[T], Generic[KT]): 
        
                   c: KT

The only difference between these examples is that in the former case the generic base is a generic class, and in the latter cases it is a generic alias. I do not think that there should be difference between them in this context. We should either require Generic[] containing all typing variables or allow it only containing some of them.

See also #94607.

I don't see a contradiction. In class Sub(Base, Generic[T]):, Base should be equivalent to Base[Any], which is the general behavior for generics missing a subscription.

It makes sense if you look at it this way. The problem is with distinguishing free and bound variables. Both are represented in __parameters__.

pythongh-98852: Fix subscription of type aliases

The issue is more serious and is not completely new.

In the following code:

from typing import *
T = TypeVar("T")
T2 = TypeVar("T2")
class A(Generic[T2]): ...

Tuple[A, T][int]
tuple[A, T][int]
Tuple[TypeVar, T][int]
tuple[TypeVar, T][int]

3.11 raises exception for the 1st and 3rd examples, produces incorrect result for the 2nd example, and crashes on the last example.

3.10 raises exception for the 2nd examples.

gh-98852: Fix subscription of type aliases (GH-98920)

pythongh-98852: Fix subscription of type aliases (pythonGH-98920)

gh-98852: Fix subscription of type aliases (GH-98920)

[3.10] pythongh-98852: Fix subscription of types.GenericAlias instanc…

[3.10] gh-98852: Fix subscription of types.GenericAlias instances (GH…

Is there anything left to do here?

It is completed.

Navigation Menu

Repository navigation

Bug or new behavior of generics parameterization? #98852

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	# see gh-94607: this fails in that bug
	class Sub(Base, Generic[T]):
	...

	with self.assertRaises(TypeError):
	class MyGeneric(List[T], Generic[S]): ...

	with self.assertRaises(TypeError):
	class Point3D(Point2DGeneric[T], Generic[KT]):
	c: KT

Bug or new behavior of generics parameterization? #98852

Description

Activity

sobolevn commented on Oct 29, 2022

Eclips4 commented on Oct 29, 2022

sobolevn commented on Oct 29, 2022

gvanrossum commented on Oct 29, 2022

Eclips4 commented on Oct 30, 2022

serhiy-storchaka commented on Oct 30, 2022

JelleZijlstra commented on Oct 30, 2022

serhiy-storchaka commented on Oct 31, 2022

serhiy-storchaka commented on Oct 31, 2022

hauntsaninja commented on Nov 6, 2022

serhiy-storchaka commented on Nov 6, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions