Multipe errors in the interaction between Protocol, Callable, ParamSpec and TypeVar

[Avasam]

Bug Report

I was trying to get some practice writing a stub with a complex Callable type. See the following wrapper function in aiofiles.os:

import asyncio
from functools import partial, wraps

def wrap(func:):
    @wraps(func)
    async def run(*args, loop=None, executor=None, **kwargs):
        if loop is None:
            loop = asyncio.get_event_loop()
        pfunc = partial(func, *args, **kwargs)
        return await loop.run_in_executor(executor, pfunc)

    return run

If I'm not mistaken, the inline-typing should look something like this:

from __future__ import annotations
import os
import asyncio
from functools import partial, wraps
from collections.abc import Callable
from typing import Any, TypeVar
from typing_extensions import ParamSpec
_T = TypeVar("_T")
_P = ParamSpec("_P")

def wrap(func: Callable[_P, _T]):  # pyright: ignore[reportInvalidTypeVarUse]  # return type is still infered correctly
    @wraps(func)
    async def run(*args: _P.args, loop: AbstractEventLoop | None=None, executor: Any | None=None, **kwargs: _P.kwargs):
        if loop is None:
            loop = asyncio.get_event_loop()
        pfunc = partial[_T](func, *args, **kwargs)
        return await loop.run_in_executor(executor, pfunc)

    return run

To Reproduce
Trying to do the following in a stub file:

import os
from asyncio.events import AbstractEventLoop
from collections.abc import Callable, Coroutine
from typing import Any, Protocol
from typing_extensions import ParamSpec, TypeVar

_T = TypeVar("_T")
_T_co = TypeVar("_T_co", covariant=True)
_P = ParamSpec("_P")

class WrappedCallable(Protocol[_P, _T_co]):
    def __call__(
        self, *args: _P.args, loop: AbstractEventLoop | None = ..., executor: Any = ..., **kwargs: _P.kwargs
    ) -> Coroutine[Any, Any, _T_co]: ...

def wrap(func: Callable[_P, _T]) -> WrappedCallable[_P, _T]: ...

stat = wrap(os.stat)

async def foo() -> None:
    stat_result = await stat()
    reveal_type(stat)
    reveal_type(stat_result)

Expected Behavior

I'm unsure, but I would like to see this being possible.
pyright actually gets the type of stat_result right, and gets wrapped callable type mostly right (it only loose the name of the generic arguments, but manages to keep both loop and executor. Which idk if there's anything that can be done about that anyway)

Actual Behavior

I get the following output:

import os.pyi:11: error: Free type variable expected in Protocol[...]  [misc]
import os.pyi:14: error: Variable "import os._T_co" is not valid as a type  [valid-type]
import os.pyi:14: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
import os.pyi:16: error: Variable "import os._T" is not valid as a type  [valid-type]
import os.pyi:16: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
import os.pyi:22: note: Revealed type is "import os.WrappedCallable[[[path: Union[builtins.int, builtins.str, builtins.bytes, os.PathLike[builtins.str], os.PathLike[builtins.bytes]], *, dir_fd: Union[builtins.int, None] =, follow_symlinks: builtins.bool =], _T?]]"
import os.pyi:23: note: Revealed type is "_T_co?"
Found 3 errors in 1 file (checked 1 source file)

Your Environment

• Mypy version used: 0.990 (compiled: yes)
• Mypy command-line flags: None
• Mypy configuration options from mypy.ini (and other config files): None
• Python version used: 3.9.13

[A5rocks]

At the moment (as of mypy v1) the output of this is:

main.py:16: error: Missing return statement  [empty-body]
main.py:22: note: Revealed type is "__main__.WrappedCallable[[path: Union[builtins.int, builtins.str, builtins.bytes, os.PathLike[builtins.str], os.PathLike[builtins.bytes]], *, dir_fd: Union[builtins.int, None] =, follow_symlinks: builtins.bool =], Tuple[builtins.int, builtins.int, builtins.int, builtins.int, builtins.int, builtins.int, builtins.int, builtins.float, builtins.float, builtins.float, fallback=os.stat_result]]"
main.py:23: note: Revealed type is "Tuple[builtins.int, builtins.int, builtins.int, builtins.int, builtins.int, builtins.int, builtins.int, builtins.float, builtins.float, builtins.float, fallback=os.stat_result]"

I'm not so sure about the fallback=os.stat_result thing but it seems to work?

[erictraut]

The code above uses keyword arguments between *args: P.args and **kwargs: P.kwargs, which is expressly forbidden by PEP 612 for reasons explained in the spec. Mypy doesn't currently emit an error for this condition. If these are moved before the *args, then the code sample type checks without errors in the latest version of mypy. I recommend closing.

The code above actually has a number of errors and missing import statements. To save others time, here's a fixed version of the code.

from asyncio import AbstractEventLoop, get_event_loop
from functools import partial, wraps
from typing import Any, ParamSpec, Callable

_P = ParamSpec("_P")

def wrap(func: Callable[_P, Any]):
    @wraps(func)
    async def run(
        loop: AbstractEventLoop | None = None,
        executor: Any | None = None,
        *args: _P.args,
        **kwargs: _P.kwargs,
    ):
        if loop is None:
            loop = get_event_loop()
        pfunc = partial(func, *args, **kwargs)
        return await loop.run_in_executor(executor, pfunc)

    return run

[Avasam]

Thanks for your input @erictraut , always appreciated.

which is expressly forbidden by PEP 612 for reasons explained in the spec

https://peps.python.org/pep-0612/#id2 and https://peps.python.org/pep-0612/#concatenating-keyword-parameters are the sections that describe this in details.

So it sounds like something that should be changed upstream in the aiofiles library if this is still an issue I encounter and would like "fixed" https://github.com/Tinche/aiofiles/blob/main/src/aiofiles/ospath.py#L9

I'll close this as I agree mypy likely doesn't need to try and support this use case as it is explicitly prohibited by PEP 612.

[A5rocks]

(Not erroring on something not allowed is a bug, but not sure if an issue about that already exists: if not, you could make an issue about that specifically!)

[erictraut]

I recall that this is already logged elsewhere, but I can't find the exact bug number at the moment.

[Avasam]

That would probably be it: #14832 (Invalid ParamSpec usage in function with added kwargs not reported as error) or #13966 (ParamSpec P.args and P.kwargs are not defined when using explicit keyword argument).
I've upvoted them.