Support expand_type with ParamSpec.{args,kwargs}

[sterliakov]

Fixes #19839.

Looks like it was relatively easy to do the right way, let me try! When splitting a callable/parameters into args and kwargs, we have the following options:

• posonly - only goes to *args, required unless has a default. If we encounter such required arg, all previously collected optional args become required (this only happens due to faulty TVT expansion somewhere; probably I should look into that too)
• kwonly - only goes to **kwargs, required unless has a default
• pos-or-kw - goes to both
• vararg - only goes to *args as an Unpack (possibly normalized by tuple constructor)
• kwargs - only goes to **kwargs and is only used if there are no kwargs with known names, because PEP 728 is not yet implemented, so we have to choose between dict and TypedDict. (thoughts? Maybe it is better to prefer dict with union(kwarg, *kwargs.values()) as value type? Either way I do not consider this question important as PEP728 will be eventually implemented, and we'll have extra_items for our TypedDicts)

Applying these steps to every argument in order, we collect required and optional args and kwargs candidates. Now, the type of **kwargs is a TypedDict if we know any keys, dict[str, KwargType] if we only have something like **kw: str, and dict[str, Never] if no kwargs were found.

The type of *args is union of all prefixes of optional_args concatenated with required_args: all required args must be there, and optional args can only be passed in order. Since it is uncommon to have a function with more than 10-20 args, I think this union is a reasonable solution.

[sterliakov]

This is correct, but ugly. Would making it "nice" justify complexity increase?

[ilevkivskyi]

I think this is fine.

[Hnasar]

Very nice!

[Hnasar]

Why did you make these classmethods? They don't use cls. Staticmethod or plain method seem less surprising.

[sterliakov]

This is nowhere near performance-critical code (we got a crash report on failing assert only a few years later after it was introduced), so it doesn't matter how optimal mypyc-compiled code is, so it's mostly a matter of preference. I prefer using classmethod because it makes later refactoring easier: it allows accessing other methods without extra diff lines and hardcoding the class name (and not regular methods to keep this code externally usable).

[ilevkivskyi]

Here are some more detailed comments, but I now think these are unnecessary complications. We should simply return upper bounds (i.e. something like tuple[object, ...] or dict[str, object]), and only consider something more precise if people will complain.

[ilevkivskyi]

Maybe I am missing something, but where do we handle the presence of the default?

[sterliakov]

It was a stupid cleanup attempt, should be fixed now

[ilevkivskyi]

What happens in examples here and and above if the callable is itself generic (including variadic/paramspec-generic)?

[sterliakov]

Fixed - it was leaking typevars indeed. It took me a while to construct a case where that happens, though...

[ilevkivskyi]

OK, LG. This is not really critical anyway since it is not well specified in the PEP.

[ilevkivskyi]

I think this is fine.

[ilevkivskyi]

@sterliakov Btw I just noticed that many of the unions in revealed types can be simplified by mypy.binder.collapse_variadic_union(), is it just a coincidence? If it is not a coincidence, then maybe you can re-use that helper here? (This will probably require moving it to typevartuples.py)

[ilevkivskyi]

Hm, looking at this again this is likely just a coincidence, because of

def f5(x: int, y: int = 0) -> int:
    return 0
def f6(x: int, *args: int) -> int:
    return 0

if the types would be different (not both ints), then there would be no compact representation.

[github-actions]

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅