test: Add memory usage benchmarks#1737
Open
ferbetanzo wants to merge 4 commits into
Open
Conversation
Contributor
Bencher Report
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1737 +/- ##
==========================================
- Coverage 93.51% 93.50% -0.02%
==========================================
Files 133 133
Lines 12766 12786 +20
==========================================
+ Hits 11938 11955 +17
- Misses 828 831 +3 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ferbetanzo
commented
May 14, 2026
| with: | ||
| mode: simulation | ||
| run: uv run pytest tests/ --codspeed -k "not executable" | ||
| run: uv run pytest tests/ --codspeed -k "not executable and not mem_compile" |
Contributor
Author
There was a problem hiding this comment.
Skipping the memory usage benchmarks in the codspeed execution since we're tracking those metrics with bencher.
maximilianruesch
changed the title
There was a problem hiding this comment.
Pull request overview
Adds peak Python-allocation memory reporting to the benchmark suite by introducing a tracemalloc-based helper, wiring it into existing *_compile benchmarks, and exporting the metric to Bencher/CodSpeed workflows.
Changes:
- Added
compile_and_get_peak_memorytest utility to collect peak traced memory during compilation. - Extended several existing compile benchmarks to record
benchmark.extra_info["memory"]and export it ascompile_memin Bencher JSON. - Introduced a new
test_mem_usagebenchmark module and adjusted CI benchmark selection/threshold configuration.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
tests/util.py |
Adds compile_and_get_peak_memory helper using tracemalloc. |
tests/benchmarks/test_queue_push_pop.py |
Records compilation peak-memory extra metric for queue compile benchmarks. |
tests/benchmarks/test_mem_usage.py |
New memory-usage benchmark module for compiling small guppy programs. |
tests/benchmarks/test_ctrl_flow.py |
Records compilation peak-memory extra metric for ctrl-flow compile benchmark. |
tests/benchmarks/test_big_array.py |
Records compilation peak-memory extra metric for big-array compile benchmark. |
tests/bencher.py |
Exports the new memory extra metric as Bencher measure compile_mem. |
pyproject.toml |
Adds a global warning filter related to pytest-benchmark warnings. |
justfile |
Adds Bencher threshold checks for compile_mem. |
.github/workflows/benchmarks_pr.yml |
Excludes mem_compile tests from PR CodSpeed benchmark run. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+53
to
+61
| try: | ||
| gc.disable() | ||
| for _ in range(n_compilations): | ||
| guppy_fn.compile() | ||
| _, peak = tracemalloc.get_traced_memory() | ||
| gc.enable() | ||
| _ = gc.collect() | ||
| return peak | ||
| finally: |
Member
There was a problem hiding this comment.
@ferbetanzo waiting for this to be resolved as well before taking a closer look.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Includes:
compile_and_get_peak_memoryutil to get the memory used when compiling a guppy function usingtracemalloc.get_traced_memory().*_compiletests in the big array, many ctrl flow and queue suites now report used peak memory.test_mem_usagesuite that benchmarks mem usage of single and multiple compilations of guppy funcs that use array and list comprehensions.