test(proto): Implement bandwidth-limited routing by matheus23 · Pull Request #720 · n0-computer/noq

matheus23 · 2026-06-23T12:27:01Z

Description

Refactors the RoutingDecision::Deliver case to give it a recv_time: Instant field. This allows Routing to decide when to deliver packets.
Adds a BwLimitedRouting routing implementation that implements a queue that delays packets the more packets are queued and tail-drops packets at a certain max queue length.
Adds a throughput test as a quick way of testing noq-proto and its congestion control against BwLimitedRouting.

Breaking Changes

None. Test changes only.

Notes & open questions

Started this work because it might be really useful in catching regressions in congestion control, and to prepare work for when we send on multiple paths simultaneously.

Still draft because this needs a little more work in the test itself (it doesn't actually assert anything yet).

But we can now generate neat pictures from noq-proto itself!

The qlog file for this is generated in 0.2 seconds on my machine, even though it is a simulated 10MB transfer over a 1MB/s connection, so this simulates ~10s of actual time.
The other neat thing about it is that it is fairly deterministic. There's still some sources of randomness in noq, but all of the "measured" transfer speeds fall into a 1% range.

Change checklist

Self-review.
Documentation updates following the style guide, if relevant.
Tests if relevant.

github-actions · 2026-06-23T12:29:54Z

Documentation for this PR has been generated and is available at: https://n0-computer.github.io/noq/pr/720/docs/noq/

Last updated: 2026-07-01T08:11:28Z

github-actions · 2026-06-23T12:36:44Z

Performance Comparison Report

796d86b12e6a3fb99971fa3de246a15dd55fbbd1 - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5587.6 Mbps	7896.7 Mbps	-29.2%	95.8% / 100.0%
medium-concurrent	5418.7 Mbps	7983.6 Mbps	-32.1%	94.8% / 99.5%
medium-single	3991.1 Mbps	4615.6 Mbps	-13.5%	98.1% / 151.0%
small-concurrent	3954.0 Mbps	5292.2 Mbps	-25.3%	93.3% / 102.0%
small-single	3600.8 Mbps	4674.9 Mbps	-23.0%	93.7% / 102.0%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	N/A	4022.5 Mbps	N/A
lan	N/A	810.4 Mbps	N/A
wan	N/A	83.8 Mbps	N/A

Summary

noq is 26.0% slower on average

---

0c79f22c7b2a70f628e475829b347a670890bcb6 - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5450.5 Mbps	7744.4 Mbps	-29.6%	94.0% / 99.0%
medium-concurrent	5256.1 Mbps	7767.4 Mbps	-32.3%	94.2% / 99.0%
medium-single	3800.0 Mbps	4748.9 Mbps	-20.0%	93.9% / 100.0%
small-concurrent	3835.1 Mbps	5358.7 Mbps	-28.4%	90.8% / 99.1%
small-single	3516.7 Mbps	4798.9 Mbps	-26.7%	89.3% / 97.2%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	3086.8 Mbps	4092.4 Mbps	-24.6%
lan	782.4 Mbps	810.4 Mbps	-3.5%
lossy	69.8 Mbps	69.8 Mbps	~0%
wan	83.8 Mbps	83.8 Mbps	~0%

Summary

noq is 27.0% slower on average

---

9cedbd99a9c48819bd910f8618c569c92465a93f - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5450.8 Mbps	8057.1 Mbps	-32.3%	96.7% / 98.2%
medium-concurrent	5469.7 Mbps	7652.7 Mbps	-28.5%	96.7% / 98.0%
medium-single	4089.4 Mbps	4563.3 Mbps	-10.4%	95.9% / 98.3%
small-concurrent	3791.2 Mbps	5214.6 Mbps	-27.3%	97.7% / 100.0%
small-single	3551.4 Mbps	4622.5 Mbps	-23.2%	96.0% / 98.3%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	3030.9 Mbps	4064.0 Mbps	-25.4%
lan	782.4 Mbps	803.1 Mbps	-2.6%
lossy	69.8 Mbps	69.8 Mbps	~0%
wan	83.8 Mbps	83.8 Mbps	~0%

Summary

noq is 25.1% slower on average

---

567199ca671d0360c40bf062c8b86bf9bfe35340 - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5477.1 Mbps	7957.4 Mbps	-31.2%	97.3% / 98.9%
medium-concurrent	5520.8 Mbps	7473.3 Mbps	-26.1%	96.0% / 97.6%
medium-single	4061.7 Mbps	4694.9 Mbps	-13.5%	95.7% / 98.0%
small-concurrent	3844.4 Mbps	5285.1 Mbps	-27.3%	97.0% / 99.6%
small-single	3522.6 Mbps	4665.7 Mbps	-24.5%	95.8% / 98.2%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	3051.2 Mbps	3942.2 Mbps	-22.6%
lan	782.4 Mbps	801.6 Mbps	-2.4%
lossy	60.0 Mbps	69.8 Mbps	-14.1%
wan	83.8 Mbps	83.8 Mbps	~0%

Summary

noq is 24.5% slower on average

---

8ae85b7fc58813eecf7851d1f5713c3d7c6e3b36 - artifacts

No results available

---

9baf9262ed5265d93022d9bd5e33efc38c939b89 - artifacts

No results available

---

fa1c9b49a60aa42afc656cc233d02e5c9b9766ce - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5383.2 Mbps	7968.1 Mbps	-32.4%	94.8% / 99.4%
medium-concurrent	5370.1 Mbps	7665.4 Mbps	-29.9%	96.1% / 147.0%
medium-single	4032.3 Mbps	4469.8 Mbps	-9.8%	97.3% / 149.0%
small-concurrent	3743.9 Mbps	5112.7 Mbps	-26.8%	94.2% / 102.0%
small-single	3444.1 Mbps	4705.8 Mbps	-26.8%	92.2% / 101.0%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	N/A	4032.2 Mbps	N/A
lan	N/A	796.4 Mbps	N/A
lossy	N/A	69.8 Mbps	N/A
wan	N/A	83.8 Mbps	N/A

Summary

noq is 26.6% slower on average

…testing

Also: Reduce test runtime by sending fewer bytes in total, and make it more robust by stepping forward until the stream datagram arrives.

flub

Why is BwLimitedConfig better as a separate routing impl instead of part of the normal BasicRouting? It seems otherwise it is identitical?

flub · 2026-07-02T13:15:36Z

+    let mut bytes_received = 0;
+
+    let start = pair.time;
+    let client_stream = pair.conn_mut(Client).streams().open(Dir::Bi).unwrap();


ConnPair::streams(Client).open(Dir::Bi).unwrap()?

flub · 2026-07-02T13:15:54Z

+    // send the first batch to ensure the other side created the stream
+    bytes_to_send -= pair
+        .conn_mut(Client)
+        .send_stream(client_stream)


ConnPair.send_stream?

I guess I'll stop commenting on this, but we should probably stick to the ConnPair methods for new tests.

Ugh sorry, I get confused about which ones are available.

flub · 2026-07-02T13:31:12Z


        if let Some(latency) = latency {
-            pair.latency = latency;
+            pair.routes.as_basic_mut().latency = latency;


Hum, it's a bit unfortunate to tie this to basic routing. Previously this would work for every routing table. Why can we not do that now as well?

We can. I can implement latency for all routing algorithms and add a fn set_latency to the Routing enum. I'll do that.

matheus23 · 2026-07-02T16:26:33Z

Why is BwLimitedConfig better as a separate routing impl instead of part of the normal BasicRouting? It seems otherwise it is identitical?

I didn't want to affect other tests! Switching from BasicRouting to BwLimitedRouting will change the timing for when packets arrive. I'm not sure if that will mess up things here and there - e.g. calculating PTO times and checking against them.

I can also try to make it part of BasicRouting, but idk, this will just make simpler tests more complicated to reason about, no?

(Also I'm not sure what you mean by "otherwise it's identical" - apart from all the latency calculations, queuing and dropping packets? Yes, apart from being bandwidth-limited, BwLimitedRouting is the same as BasicRouting...)

matheus23 self-assigned this Jun 23, 2026

n0bot Bot added this to iroh Jun 23, 2026

github-project-automation Bot moved this to 🚑 Needs Triage in iroh Jun 23, 2026

flub reviewed Jun 24, 2026

View reviewed changes

Comment thread noq-proto/src/tests/multipath.rs Outdated

Comment thread noq-proto/src/tests/util.rs

Comment thread noq-proto/src/tests/util.rs

matheus23 added 5 commits June 26, 2026 16:37

refactor(proto): Let the routing own the delivery time

6e24ca9

test(proto): Implement BwLimitedRouting and write throughput test

6b9762a

Implement Routing being able to ECN-mark

85ffba8

Add InboundQueue and address review

96d675a

Add assertion

9cedbd9

matheus23 force-pushed the matheus23/throughput-testing branch from 0c79f22 to 9cedbd9 Compare June 26, 2026 17:18

matheus23 marked this pull request as ready for review June 26, 2026 17:19

matheus23 added 2 commits July 1, 2026 09:27

Merge remote-tracking branch 'origin/main' into matheus23/throughput-…

567199c

…testing

Use current timestamp for re-queuing withheld datagrams

9baf926

matheus23 force-pushed the matheus23/throughput-testing branch from 8ae85b7 to 9baf926 Compare July 1, 2026 08:01

Move throughput test from multipath.rs to tests/mod.rs

fa1c9b4

Also: Reduce test runtime by sending fewer bytes in total, and make it more robust by stepping forward until the stream datagram arrives.

matheus23 requested a review from flub July 1, 2026 08:11

flub reviewed Jul 2, 2026

View reviewed changes

Uh oh!

Conversation

matheus23 commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Breaking Changes

Notes & open questions

Change checklist

Uh oh!

github-actions Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Comparison Report

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

flub left a comment

Choose a reason for hiding this comment

Uh oh!

flub Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

flub Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

matheus23 Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

flub Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

matheus23 Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

matheus23 commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

matheus23 commented Jun 23, 2026 •

edited

Loading

github-actions Bot commented Jun 23, 2026 •

edited

Loading

github-actions Bot commented Jun 23, 2026 •

edited

Loading

matheus23 commented Jul 2, 2026 •

edited

Loading