type inference failure on custom iterable class unpacking

[danmur97]

Bug Report

When unpacking a custom iterable class to build another iterable, mypy does not correctly infer the resulting type i.e. CustomIter[CustomIter[_T]] is equal to CustomIter[_T] (that should not be true)

To Reproduce

proof-of-concept

Expected Behavior

Nested iterable unpacking type checking works fine when using standard types (e.g. list[list[_T]]) however custom iterable classes should have the same behavior.

Actual Behavior

As presented on the playground proof-of-concept the issue is that mypy does not mark the failure on this incorrectly defined function:

def case_1(iter_of_iter: CustomIter[list[int]]) -> CustomIter[int]:
    # Mypy fails
    #
    # Mypy does not flag that the return type does not match.
    # It is `CustomIter[list[int]]` instead of `CustomIter[int]`
    return CustomIter.new(*iter_of_iter)

Your Environment

• details available on the playground example.

[sterliakov]

Condensed repro:

# mypy: disable-error-code=empty-body
from __future__ import annotations

from typing import Generic, Iterator, TypeVar

_T = TypeVar("_T")

class CustomIter(Generic[_T]):
    def __iter__(self) -> Iterator[_T]: ...


def case_1(iter_of_iter: CustomIter[list[int]]) -> None:
    reveal_type([*iter_of_iter])  # N: Revealed type is "builtins.list[Any]"
    reveal_type(list(iter_of_iter))  # N: Revealed type is "builtins.list[builtins.list[builtins.int]]"

The likely cause is this (very old) mess:

mypy/mypy/checker.py

Lines 1085 to 1118 in 88cf8e0

	def get_generator_yield_type(self, return_type: Type, is_coroutine: bool) -> Type:
	"""Given the declared return type of a generator (t), return the type it yields (ty)."""
	return_type = get_proper_type(return_type)
	if isinstance(return_type, AnyType):
	return AnyType(TypeOfAny.from_another_any, source_any=return_type)
	elif isinstance(return_type, UnionType):
	return make_simplified_union(
	[self.get_generator_yield_type(item, is_coroutine) for item in return_type.items]
	)
	elif not self.is_generator_return_type(
	return_type, is_coroutine
	) and not self.is_async_generator_return_type(return_type):
	# If the function doesn't have a proper Generator (or
	# Awaitable) return type, anything is permissible.
	return AnyType(TypeOfAny.from_error)
	elif not isinstance(return_type, Instance):
	# Same as above, but written as a separate branch so the typechecker can understand.
	return AnyType(TypeOfAny.from_error)
	elif return_type.type.fullname == "typing.Awaitable":
	# Awaitable: ty is Any.
	return AnyType(TypeOfAny.special_form)
	elif return_type.args:
	# AwaitableGenerator, Generator, AsyncGenerator, Iterator, or Iterable; ty is args[0].
	ret_type = return_type.args[0]
	# TODO not best fix, better have dedicated yield token
	return ret_type
	else:
	# If the function's declared supertype of Generator has no type
	# parameters (i.e. is `object`), then the yielded values can't
	# be accessed so any type is acceptable. IOW, ty is Any.
	# (However, see https://github.com/python/mypy/issues/1933)
	return AnyType(TypeOfAny.special_form)

mypy expects a Generator for some reason, and Iterator is not a Generator - the latter has nothing to do with yielded type in general, it can go through the same __iter__ protocol.

Ough, sorry, that block still looks wrong, but that's irrelevant here. Our problem is here:

mypy/mypy/argmap.py

Lines 201 to 207 in 9c1406c

	if isinstance(actual_type, Instance) and actual_type.args:
	from mypy.subtypes import is_subtype
	if is_subtype(actual_type, self.context.iterable_type):
	return map_instance_to_supertype(
	actual_type, self.context.iterable_type.type
	).args[0]

In this case is_subtype should be true (structural subtyping), but map_instance_to_supertype can't handle that:

mypy/mypy/maptype.py

Lines 8 to 14 in f33c9a3

	def map_instance_to_supertype(instance: Instance, superclass: TypeInfo) -> Instance:
	"""Produce a supertype of `instance` that is an Instance
	of `superclass`, mapping type arguments up the chain of bases.
	If `superclass` is not a nominal superclass of `instance.type`,
	then all type arguments are mapped to 'Any'.
	"""

[randolf-scholz]

This is fixed on my rework star args branch (randolf-scholz#6) (context: #19651 (comment)), not sure when I'll have time to submit / clean up that monster PR though.