bench(sirun): report operations per second#9039
Conversation
Loop benchmarks now emit an operations-per-second metric from the same measured-loop window that the startup guard already owns, without adding a second timer around each benchmark.
Bench variants now use OPERATIONS as the single loop-count input, so the startup guard reports throughput from the same counter each benchmark already runs with.
Overall package sizeSelf size: 6.32 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.2.0 | 104.26 kB | 843.44 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
BenchmarksBenchmark execution time: 2026-06-25 11:46:28 Comparing candidate commit fd3327d in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2266 metrics, 17 unstable metrics.
|
This makes sure the ops/sec metric only uses values that the measured loop actually consumes, while live request benchmarks keep their request-count knobs separate.
🎉 All green!🧪 All tests passed 🔗 Commit SHA: fd3327d | Docs | Datadog PR Page | Give us feedback! |
…hit variant Forcing an OPERATIONS count on every loop and asserting it was wrong for loops whose iteration is not a single operation: profiler counts fixed busy-work rounds, async_hooks batches promises across intervals, and test-codeowners runs whole-corpus passes. Revert those three to their original count vars and have startup-guard warn and skip the metric when OPERATIONS is unset, so a bench reports throughput only when one iteration is one operation. Also restore the debugger control flow. The restructure moved the install-only path out of the DD_DYNAMIC_INSTRUMENTATION_ENABLED branch and dropped its keepAlive, so the enabled-but-breakpoint-not-hit variant could exit before the probe finished installing and measure nothing.
BridgeAR
requested review from
khanayan123 and
wconti27
and removed request for
a team
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd3327d68d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| return | ||
| } | ||
|
|
||
| statsd ??= new (require('./statsd'))() |
There was a problem hiding this comment.
For loop benchmarks run under sirun, SIRUN_READY_FD resets wall.time to the post-ready period, and the process is still timed until it exits; this lazy require('./statsd') plus UDP socket construction runs after the hot loop but before exit, so it is included in sirun wall.time even though the duration used for .ops and the startup-share assertion exclude it. On shorter variants this adds a fixed post-loop cost to every sample and can mask small regressions; initialize/cache the StatsD client before loopStart() or otherwise outside the post-ready timing window.
Useful? React with 👍 / 👎.
![datadog-datadog-prod-us1-2[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
The PR consistently and correctly migrates all loop benchmarks to a unified OPERATIONS env var for throughput reporting. Every active work-loop variant in meta.json has OPERATIONS set; the one missing entry (debugger/enabled-but-breakpoint-not-hit) intentionally skips the work loop via INSTALL_ONLY=true. The startup-guard's existing startup-share assertion acts as a second safety net: if OPERATIONS were ever accidentally omitted, the 0-iteration loop would make startup share ≈ 100%, which fails loudly. No behavioral regressions found.
📊 Validated against 8 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit fd3327d · What is Autotest? · Any feedback? Reach out in #autotest
Loop benchmarks now emit an operations-per-second metric from the same measured-loop window that the startup guard already owns, without adding a second timer around each benchmark. Bench variants now use OPERATIONS as loop-count input, so the startup guard reports throughput from the same counter each benchmark already runs with.
2 participants
Summary
Loop benchmarks now emit an operations-per-second metric from the startup guard's measured window. The count comes from a single
OPERATIONSenv var, so benchmark metadata and benchmark code agree on the loop count used for throughput.Test plan
node benchmark/sirun/run.js spans --iterations 1 --append --outfile check-runs.txtnode benchmark/sirun/run.js propagation --iterations 1 --append --outfile check-runs.txtnode benchmark/sirun/run.js async_hooks --iterations 1 --append --outfile check-runs.txtnpm run lintRefs: #5606