Self and self are not bound until after class definition, becoming Any/Never when passed through a decorator #16554

Activity

1. jace

   added

   bugmypy got something wrongmypy got something wrong

   on Nov 24, 2023
2. jace commented on Nov 25, 2023

   

   jace

   on Nov 25, 2023

   Author

   More actions

   I think Mypy's handling of self isn't completely wrong, even though it breaks typing via the decorator but works for built-ins like @property and @classmethod. The type of the self/cls parameter is only known at call time. The type assigned at definition time (host class's type or a subtype) is only the usual type for the usual descriptor-based access (cls.method or obj.method). Which means the problem is in the decorator's typing for its descriptor:

   def __get__(self, obj: T | None, cls: type[T]) -> Callable[P, R]: ...

   There is no indication that the Callable is typed T, so the Self type in R can't be linked to T. Maybe it can be done using a Protocol?

   class WrappedMethod(Protocol[T, P, R]):
       def __call__(self: T, *args: P.args, **kwargs: P.kwargs) -> R: ...
   
   
   class Decorator:
       ...
       def __get__(self, obj: T | None, cls: type[T]) -> WrappedMethod[T, P, R]: ...
3. jace commented on Nov 25, 2023

   

   jace

   on Nov 25, 2023

   Last edited by jace

   Author

   More actions

   Okay, checked this up. Specifying a type on self in Protocol.__call__ makes it not callable, so dead-end there.

   Pyright also correctly processes Self when the decorated method is accessed via a subclass. I tried the pre-Self method of a bound TypeVar but that doesn't work in both Pyright and Mypy:

   class Decorator(Generic[P, R]):
       def __init__(self, fget: Callable[Concatenate[Any, P], R]): ...
       @overload
       def __get__(self, obj: None, cls: Type[T]) -> Callable[Concatenate[T, P], R]: ...
       @overload
       def __get__(self, obj: T, cls: Type[T]) -> Callable[P, R]: ...
       def __get__(self, obj: Optional[T], cls: Type[T]) -> Union[Callable[P, R], Callable[Concatenate[T, P], R]]: ...
   
   ParentType = TypeVar('ParentType', bound='Parent')
   
   class Parent:
       @Decorator
       def method_self(self, arg: str) -> Self:
           return self
   
       @Decorator
       def method_typevar(self: ParentType, arg: str) -> ParentType:
           return self
   
   class Child(Parent):
       pass
   
   reveal_type(Parent.method_self)
   # Pyright: "(arg: str) -> Parent"  (correct)
   # Mypy: "(arg: str) -> Any"
   
   reveal_type(Parent.method_typevar)
   # Pyright: "(arg: str) -> Any"
   # Mypy:  "(arg: str) -> Any"
   
   reveal_type(Child.method_self)
   # Pyright: "(arg: str) -> Child" (correct)
   # Mypy: "(arg: str) -> Any"
   
   reveal_type(Child.method_typevar)
   # Pyright: "(arg: str) -> Any"
   # Mypy:  "(arg: str) -> Any"

   PEP 673 – Self Type makes no mention of how Self should be processed in a generic decorator except for @classmethod (which is anyway special-cased in type checkers), so this looks like a gap in the spec.

   The difference in how Pyright and Mypy bind Self comes out when calling an unbound method:

   class Parent:
       def method(self) -> Self: return self
   
   class Child(Parent): pass
   
   s1 = Child()
   s2 = Parent.method(s1)
   reveal_type(s2)
   # Runtime: Child
   # Pyright: Parent (wrong)
   # Mypy: Parent (correct)

   So: Pyright is binding Self from the class or instance, which is usually right, but wrong when calling an unbound method, whereas Mypy is more accurately binding it from the self parameter, but that is lost in the decorator's __get__ signature. If only there was a way to add it back with something like MethodType[T, Callable[P, R]] (or with Protocol).

   Related: #16558, cc: @erictraut and @AlexWaygood
4. jace commented on Nov 25, 2023

   

   jace

   on Nov 25, 2023

   Author

   More actions

   Test code: https://mypy-play.net/?mypy=latest&python=3.11&gist=e8de541c111bccf640907937f8794a1b
5. erictraut commented on Nov 25, 2023

   

   erictraut

   on Nov 25, 2023

   More actions

   Pyright is binding Self from the class or instance ... [which is] wrong when calling an unbound method

   I think this result is still correct from a type perspective. In your example above, the runtime produces a value which is a Child instance, and this is a valid subtype of Parent, which is the type evaluated by pyright. In other words, this isn't a "wrong" type evaluation. It's perhaps not as narrow as you'd like, but that can't be helped if binding is done when it should be.

   I think binding and partial specialization must be done when evaluating the attribute access expression (Parent.method), not as part of the call expression (Parent.method(s1)). This becomes clear when you consider generic classes. Consider if Parent were generic and accepted a single type parameter, as in the modified example below:

   from typing import Generic, Self, TypeVar
   
   T = TypeVar("T")
   
   class Parent(Generic[T]):
       def method(self) -> Self:
           return self
   
   class Child(Parent[int]):
       pass
   
   s1 = Child()
   s2 = Parent[str].method(s1) # Pyright: Type violation (correct), mypy: no error
   reveal_type(s2)

   Here, mypy's approach results in a clear false negative.

   Binding rules are admittedly underspecified in the Python typing spec. This would be a good topic for the newly-formed typing council to clarify so we can get consistent behavior between type checkers.

   👍React with 👍1Reacted by Kiran Jonnalagadda
6. jace commented on Nov 25, 2023

   

   jace

   on Nov 25, 2023

   Author

   More actions

   I think binding and partial specialization must be done when evaluating the attribute access expression

   I agree. This has the nice side effect that the generic decorator is simpler because it doesn't have to bind on type.

   In my first code sample above, I made the mistake of defining self type as invariant because that's the default for TypeVar, but self has to be covariant. I had no idea what these terms meant last month, and I imagine it's going to cause grief to many others writing type annotations for the first time.