Skip to content

add pytest-benchmark to benchmarking section #596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
samcunliffe merged 4 commits into from
Jan 23, 2026
Merged

add pytest-benchmark to benchmarking section #596

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
22d708a
add pytest-benchmark to benchmarking section
K-Meech Jan 16, 2026
cd0df7f
Merge branch 'main' into km/expand-benchmarks-section
samcunliffe Jan 16, 2026
2481824
update pytest-benchmark text
K-Meech Jan 16, 2026
b8bf82d
remove backticks in pytest-benchmark description
K-Meech Jan 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/pages/benchmarking-profiling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ layout: default

## Benchmarking

| Name | Short description | 🚦 |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-----------------------------------------: |
| [asv](https://asv.readthedocs.io/en/stable/) | A tool for benchmarking Python packages over their lifetime. Allows you to write benchmarks and then run them against every commit in the repository, to identify where performance increased or decreased. Comparative benchmarks can also be run, which can be useful for [running them in CI using GitHub runners](https://labs.quansight.org/blog/2021/08/github-actions-benchmarks). | <span class="label label-green">Best</span> |
| Name | Short description | 🚦 |
| --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------: |
| [asv](https://asv.readthedocs.io/en/stable/) | A tool for benchmarking Python packages over their lifetime. Allows you to write benchmarks and then run them against every commit in the repository, to identify where performance increased or decreased. Comparative benchmarks can also be run, which can be useful for [running them in CI using GitHub runners](https://labs.quansight.org/blog/2021/08/github-actions-benchmarks). | <span class="label label-green">Best</span> |
| [pytest-benchmark](https://pytest-benchmark.readthedocs.io/en/stable) | Provides a `benchmark` fixture with full `pytest` integration. A simple, light-weight alternative to `asv`, for cases that don't require tracking performance over many commits. | <span class="label label-yellow">Good</span> |

@connoraird connoraird Jan 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can also run comparative benchmarks (regression-tests) using pytest-benchmark. We've done this in the glass project

@connoraird connoraird Jan 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although we are a little sceptical of their reliability but I think that matches what the linked blog says

@K-Meech K-Meech Jan 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @connoraird - good to know! I guess it is still not as fully featured as asv's comparison though? i.e. it's more geared towards comparing specific runs / subsets of runs, rather than the full project history?

How about something like:
Provides a benchmark fixture with full pytest integration, and the ability to compare performance between multiple runs. A simple, light-weight alternative to asv.

@samcunliffe samcunliffe Jan 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| [asv](https://asv.readthedocs.io/en/stable/) | A tool for benchmarking Python packages over their lifetime. Allows you to write benchmarks and then run them against every commit in the repository, to identify where performance increased or decreased. Comparative benchmarks can also be run, which can be useful for [running them in CI using GitHub runners](https://labs.quansight.org/blog/2021/08/github-actions-benchmarks). | <span class="label label-green">Best</span> |
| [pytest-benchmark](https://pytest-benchmark.readthedocs.io/en/stable) | Provides a `benchmark` fixture with full `pytest` integration. A simple, light-weight alternative to `asv`, for cases that don't require tracking performance over many commits. | <span class="label label-yellow">Good</span> |
| [asv](https://asv.readthedocs.io/en/stable/) | A tool for benchmarking Python packages over their lifetime. Allows you to write benchmarks and then run them against every commit in the repository, to identify where performance increased or decreased. Comparative benchmarks can also be run, which can be useful for [running them in CI using GitHub runners](https://labs.quansight.org/blog/2021/08/github-actions-benchmarks). | <span class="label label-green">Best</span> |
| [pytest-benchmark](https://pytest-benchmark.readthedocs.io/en/stable) | Provides a benchmark fixture with full pytest integration, and the ability to compare performance between multiple runs. A simple, light-weight alternative to asv that we've found to be useful. | <span class="label label-yellow">Good</span> |

Because you said

It was a useful alternative to asv

... feels worth mentioning.

@K-Meech K-Meech Jan 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks both - I've updated the text now 👍


## Profiling

Expand Down
Loading