TypeVar bound to Protocol behaves wrong around decorators

[bo5o]

Bug report

I am experiencing the following behaviour:

import functools
from typing import Any, Hashable, Iterable, Protocol, TypeVar, cast


class IterableFirst(Protocol):
    def __call__(self, __a: Iterable[Hashable], *args, **kwargs) -> Any:
        ...


F = TypeVar("F", bound=IterableFirst)


def my_decorator(func: F) -> F:
    @functools.wraps(func)
    def decorator(__a, *args, **kwargs):
        return func(__a, *args, **kwargs)

    return cast(F, decorator)


@my_decorator  # ERROR: Value of type variable "F" of "my_decorator" cannot be "Callable[[Iterable[str]], Any]"
def foo(a: Iterable[str]):
    return sum(map(hash, a))

@my_decorator  # same error in Python 3.8.1; in Python 3.7.6 with typing_extension no error
def bar(a: Iterable[Hashable], b, c):
    return sum(map(hash, a))

Expected behaviour

If I am correct, both examples (foo, bar) should be fine.

Version

$ mypy --version
mypy 0.761
$ python -V
Python 3.8.1

Same behaviour on current master (1104a9f)

Extra flags

I am using default mypy settings.

[ilevkivskyi]

FWIW this is actually a legitimate error because Callable[[Iterable[str]], Any] is not a subtype of Callable[[Iterable[Hashable]], Any] because argument types in callables are contravariant, you can read some docs about this, see https://mypy.readthedocs.io/en/stable/generics.html#variance-of-generic-types

[ilevkivskyi]

Oh, the second example is a bit different, it is still technically unsafe, but we might well allow such things see #5876

[bo5o]

Ah, thank you. I understand now.

However, how do I behave correctly in this kind of situation? The decorator should only accept functions that have an iterable of hashable objects as first argument, so Iterable[str], Iterable[int] and custom objects that implement __hash__ should all be accepted.

[ilevkivskyi]

For this you would probably need type variables with lower bounds. I think this was proposed before but is unlikely to be implemented in foreseeable future. So you can only use Iterable[Any].

[bo5o]

I tried with Iterable[Any] but even that throws an error. I've condensed the code to a minimal example:

import functools
from typing import Any, Iterable, Protocol, TypeVar, cast


class IterableFirst(Protocol):
    def __call__(self, __a: Iterable[Any], *args, **kwargs) -> Any:
        ...


foo: IterableFirst


def bar(a: Iterable[str], b, c):
    pass


foo = bar  # ERROR: Incompatible types in assignment (expression has type "Callable[[str, Any, Any], Any]", variable has type "IterableFirst")

Mypy output:

$ mypy main.py
main.py:16: error: Incompatible types in assignment (expression has type "Callable[[Iterable[str], Any, Any], Any]", variable has type "IterableFirst")
main.py:16: note: "IterableFirst.__call__" has type "Callable[[Iterable[Any], VarArg(Any), KwArg(Any)], Any]"
Found 1 error in 1 file (checked 1 source file)
$ mypy -V
mypy 0.761
$ python -V
Python 3.8.1

I can resolve the issue by not using *args and **kwargs.

import functools
from typing import Any, Iterable, Protocol, TypeVar, cast


class IterableFirst(Protocol):
    def __call__(self, __a: Iterable[Any], __b, __c) -> Any:
        ...


foo: IterableFirst


def bar(a: Iterable[str], b, c):
    pass


foo = bar  # no error

And the same goes for the initial problem

import functools
from typing import Any, Hashable, Iterable, Protocol, TypeVar, cast


class IterableFirst(Protocol):
    def __call__(self, __a: Iterable[Any], __b: Any, __c: Any) -> Any:
        ...


F = TypeVar("F", bound=IterableFirst)


def my_decorator(func: F) -> F:
    @functools.wraps(func)
    def decorator(__a, *args):
        return func(__a, *args)

    return cast(F, decorator)


@my_decorator  # no error
def foo(a: Iterable[str], b, c):
    return sum(map(hash, a))


@my_decorator  # no error
def bar(a: Iterable[Hashable], b, c):
    return sum(map(hash, a))

Is typing *args and **kwargs in Protocols not allowed?

By the way, I observe the same behaviour with normal Callable types:

import functools
from typing import Any, Callable, Iterable

from mypy_extensions import KwArg, VarArg

IterableFirst = Callable[[Iterable[Any], VarArg(Any), KwArg(Any)], Any]


def bar(a: Iterable[str], b, c) -> str:
    return next(iter(a), "default")

# ERROR: Incompatible types in assignment (expression has type "Callable[[Iterable[str], Any, Any], str]", variable has type "Callable[[Iterable[Any], VarArg(Any), KwArg(Any)], Any]")
foo: IterableFirst = bar

$ mypy main.py
main.py:13: error: Incompatible types in assignment (expression has type "Callable[[Iterable[str], Any, Any], str]", variable has type "Callable[[Iterable[Any], VarArg(Any), KwArg(Any)], Any]")
Found 1 error in 1 file (checked 1 source file)

[shaunc]

Would "ParamSpec" help here?

https://peps.python.org/pep-0612/