Skip to content

Make benchmarks stable, the second attempt - #1206

Merged
asvetlov merged 1 commit into
aio-libs:masterfrom
asvetlov:stable-bench
Jun 25, 2025
Merged

Make benchmarks stable, the second attempt#1206
asvetlov merged 1 commit into
aio-libs:masterfrom
asvetlov:stable-bench

Conversation

@asvetlov

Copy link
Copy Markdown
Member

Problematic benchmarks now ask not the single key, but three sibling keys.
These keys have different hashes, maybe it could help with amortizing benchmark time after hash collisions.

See #1201 for details

@asvetlov asvetlov added the bot:chronographer:skip This PR does not need to include a change note label Jun 25, 2025
@asvetlov
asvetlov marked this pull request as ready for review June 25, 2025 17:04
@codspeed-hq

codspeed-hq Bot commented Jun 25, 2025

Copy link
Copy Markdown

CodSpeed Performance Report

Merging #1206 will degrade performances by 79.79%

Comparing asvetlov:stable-bench (d6c8577) with master (06f1301)

Summary

❌ 12 (👁 12) regressions
✅ 232 untouched benchmarks

Benchmarks breakdown

Benchmark BASE HEAD Change
👁 test_cimultidict_getall_istr_hit[c] 102.4 µs 261.1 µs -60.78%
👁 test_cimultidict_getall_istr_hit[py] 1.6 ms 8.1 ms -79.79%
👁 test_cimultidict_getall_istr_miss[c] 28.5 µs 50.8 µs -43.87%
👁 test_cimultidict_getall_istr_miss[py] 176.4 µs 537.5 µs -67.18%
👁 test_multidict_getall_str_hit[ci-c] 112.7 µs 284.6 µs -60.39%
👁 test_multidict_getall_str_hit[ci-py] 1.6 ms 8.1 ms -79.7%
👁 test_multidict_getall_str_hit[cs-c] 99.1 µs 245.5 µs -59.64%
👁 test_multidict_getall_str_hit[cs-py] 1.7 ms 8 ms -78.5%
👁 test_multidict_getall_str_miss[ci-c] 39.1 µs 74.5 µs -47.48%
👁 test_multidict_getall_str_miss[ci-py] 173.8 µs 576.8 µs -69.87%
👁 test_multidict_getall_str_miss[cs-c] 25.6 µs 37.4 µs -31.57%
👁 test_multidict_getall_str_miss[cs-py] 139 µs 439.8 µs -68.38%

@codecov

codecov Bot commented Jun 25, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.32%. Comparing base (06f1301) to head (d6c8577).
⚠️ Report is 31 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1206      +/-   ##
==========================================
- Coverage   99.85%   98.32%   -1.53%     
==========================================
  Files          25       27       +2     
  Lines        3380     3874     +494     
  Branches      229      706     +477     
==========================================
+ Hits         3375     3809     +434     
- Misses          3       18      +15     
- Partials        2       47      +45     
Flag Coverage Δ
CI-GHA 98.32% <100.00%> (-1.53%) ⬇️
MyPy 80.47% <100.00%> (?)
pytest 99.85% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@asvetlov
asvetlov merged commit e7b77ca into aio-libs:master Jun 25, 2025
64 checks passed
@asvetlov
asvetlov deleted the stable-bench branch June 25, 2025 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:skip This PR does not need to include a change note

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant