Skip to content

Add type stubs for BenchmarkFixture#309

Open
hwelch-fle wants to merge 12 commits into
ionelmc:masterfrom
hwelch-fle:add-type-stub-benchmarkfixture
Open

Add type stubs for BenchmarkFixture#309
hwelch-fle wants to merge 12 commits into
ionelmc:masterfrom
hwelch-fle:add-type-stub-benchmarkfixture

Conversation

@hwelch-fle

Copy link
Copy Markdown

Added type stub for BenchmarkFixture class. The typing is currently relegated to a fixture.pyi file to prevent modification of the actual fixture code.

There is some existing work on this in #290, but this should be hinting for most of the public API.

I have also included a new test_stubs.py file that tests different argument options to test that return types are properly inferred.

Methods:

  • __call__: Infers *args and **kwargs types from ParamSpec of function_to_benchmark
  • pedantic: Shows Type Error when setup is passed with args and or kwargs, infers return from Param Spec
  • weave: Probably not neccessary since **kwargs are not for the target

Attributes

  • name: str
  • fullname: str
  • disabled: bool
  • param: str | None
  • params: tuple[str, ...] | None
  • group: str | None
  • has_error: bool
  • extra_info: dict[str, Any]
  • skipped: bool
  • cprofile: cProfile.Profile
  • cprofile_loops: int | None
  • cprofile_dump: str | None
  • cprofile_stats: dict[Any, Any] | None
  • stats: dict[Any, Any] | None

I'm not sure if it's best to roll this into the actual codebase or keep it in a pyi sidecar, but if it's decided that the hints should be inlined, these headers can be directly copied over.

resolves #212

@hwelch-fle

hwelch-fle commented May 24, 2026

Copy link
Copy Markdown
Author

I am a bit unsure about some of the stats objects. Are they cProfile stats or pytest-benchmark stats? I think both of them implement __getitem__ so maybe those should be changed to Mapping[str, Any] ?

The cProfile stats do use a tuple key (typed internally as _Label so I originally had them as Any, Any so you don't get complaints if you know what you're doing)

Also added newline at end of each file
Moved all type tests under a `TYPE_CHECKING` block since they should only be run by mypy

Additionally used old style TypeVar/ParamSpec pattern to allow use in pre 3.12 Python versions
Since stats are exposed by a public interface in `BenchmarkFixture`, we should probably add stubs for them too
@hwelch-fle

Copy link
Copy Markdown
Author

Sorry for all the additional commits, I'm getting used to your tox setup and trying to make sure everything's compliant and doesn't add any warnings/errors to your current CI

- Remove overloads for `as_dict` in favor of a union of the flat/nested TypedDicts
- User can narrow the type by checking for keys (e.g. 'min' in m.as_dict() -> _FlatMetadataDict)
@hwelch-fle

Copy link
Copy Markdown
Author

These could also go in the typeshed if you don't want to maintain them here, but I saw you added py.typed, so I opened this here instead.

I will also be happy to maintain these stubs in the future. I added them to my local environment and it cleared all the Unknown type cheker errors I was getting.

There are also some existing hints in the modules, I added those to the stubs, but the stubs could also be marked partial by adding partial\n to py.typed and defer type resolution to the actual modules if/when inline hints are implemented there.

Since this extension still supports 3.10, it's probably best to keep the typing separate from the codebase due to the changes that have happened between 3.10 and 3.14. The stubs will be totally ignored at runtime so we only really need to make sure that they work with current type checkers without having to deal with version specific imports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add type annotations

1 participant