Skip to content

fix(iroh-dns-server): DNS write benchmark measurement#4382

Merged
flub merged 2 commits into
n0-computer:mainfrom
jmqd:jm/dns-write-benchmark
Jul 1, 2026
Merged

fix(iroh-dns-server): DNS write benchmark measurement#4382
flub merged 2 commits into
n0-computer:mainfrom
jmqd:jm/dns-write-benchmark

Conversation

@jmqd

@jmqd jmqd commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Measure only the DNS write loop in the Criterion benchmark.

The old benchmark used b.iter, so Criterion measured runtime creation, config loading, server binding, client setup, packet creation, the publish loop, and shutdown. It also republished the same signed packet for every iteration.

This patch moves setup outside the measured duration, switches to iter_custom, and generates distinct signed packets for the measured publish batch. This makes dns_server_writes report write-path throughput instead of setup-heavy duplicate-publish timing.

   +-------+----------------------+----------------------+---------------------+----------------------+----------------------+--------------------------+
   | iters | before time (median) | after time (median)  | median time delta   | before throughput    | after throughput     | Criterion verdict        |
   +-------+----------------------+----------------------+---------------------+----------------------+----------------------+--------------------------+
   |    10 |            36.372 ms |             4.009 ms |  -32.363 ms / -89.0%|        274.93 elem/s |      2.494 Kelem/s   | improved, p < 0.05       |
   |   100 |            69.348 ms |            37.477 ms |  -31.871 ms / -46.0%|      1.442 Kelem/s   |      2.668 Kelem/s   | improved, p < 0.05       |
   |   250 |           113.170 ms |           102.680 ms |  -10.490 ms /  -9.3%|      2.209 Kelem/s   |      2.435 Kelem/s   | improved, p = 0.05       |
   |  1000 |           364.230 ms |           414.710 ms |  +50.480 ms / +13.9%|      2.746 Kelem/s   |      2.411 Kelem/s   | regressed, p < 0.05      |
   +-------+----------------------+----------------------+---------------------+----------------------+----------------------+--------------------------+

Per-element view:

  +-------+------------------------+------------------------+----------------------+
  | iters | before median / elem   | after median / elem    | interpretation       |
  +-------+------------------------+------------------------+----------------------+
  |    10 |            3637.20 us  |             400.91 us  | setup dominated old  |
  |   100 |             693.48 us  |             374.77 us  | setup still visible  |
  |   250 |             452.68 us  |             410.72 us  | closer to write path |
  |  1000 |             364.23 us  |             414.71 us  | unique writes cost   |
  +-------+------------------------+------------------------+----------------------+

Change checklist

  • Self-review.
  • Tests if relevant.
  • This PR was created by a human that thought critically about the
    proposed change and wrote an as clear and concise description as
    they could.
  • This PR isn't slop, and is carefully crafted to do have the
    intended effect.

Measure only the DNS write loop in the Criterion benchmark.

The old benchmark used `b.iter`, so Criterion measured runtime
creation, config loading, server binding, client setup, packet
creation, the publish loop, and shutdown. It also republished the same
signed packet for every iteration.

This patch moves setup outside the measured duration, switches to
`iter_custom`, and generates distinct signed packets for the measured
publish batch. This makes `dns_server_writes` report write-path
throughput instead of setup-heavy duplicate-publish timing.
@n0bot n0bot Bot added this to iroh Jun 30, 2026
@github-project-automation github-project-automation Bot moved this to 🚑 Needs Triage in iroh Jun 30, 2026

@flub flub left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@flub flub changed the title Fix DNS write benchmark measurement fix(iroh-dns-server): DNS write benchmark measurement Jul 1, 2026
@flub flub merged commit 9541a6d into n0-computer:main Jul 1, 2026
38 checks passed
@github-project-automation github-project-automation Bot moved this from 🚑 Needs Triage to ✅ Done in iroh Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

2 participants