test(vsock): add latency test by aaron-ang · Pull Request #5603 · firecracker-microvm/firecracker

aaron-ang · 2026-01-06T02:04:15Z

Create simple ping latency test in vsock_helper host tools

Changes

Reason

Add latency measurement to the vsock performance suite by extending vsock_helper.c with ping and ping-uds commands that emit per‑request RTTs
Add latency tests in test_vsock.py that exercise guest -> host and host -> guest paths.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

This functionality cannot be added in rust-vmm.

aaron-ang · 2026-01-06T02:29:14Z

hi @raduiliescu @Manciukic could you please take a look?

Manciukic · 2026-01-08T12:05:12Z

Hey @aaron-ang, thanks for the contribution! I only had a very quick look, but I was wondering whether we need both ping and ping-uds. Intuitively, the ping is symmetric and already measures both g2h and h2g so I would think only the ping mode is needed. Did you notice any difference in measurement between the two tests? Can you share some data/output on your reference machine to get a feel on how that looks like?

JamesC1305 · 2026-02-25T16:53:05Z

Hi @aaron-ang,

Just wanted to see if you had a chance to read our previous update? Let us know if you need anything more from us.

Thank you!

aaron-ang · 2026-02-25T17:44:52Z

Hi @JamesC1305, apologies as I have been really busy the past few weeks. I will look into this within this week.

aaron-ang · 2026-02-26T04:16:18Z

My setup: "cpu_model": "Intel(R) N100", "host_kernel": "linux-6.17", "rootfs": "ubuntu-24.04.squashfs".

I did some stats aggregation (ping-uds has slightly higher overall latency):

Direction	n	mean (µs)	median (µs)	p95 (µs)	p99 (µs)
g2h/ping	3600	633.0	375.5	2201.3	3012.1
h2g/ping-uds	3600	861.5	508.0	2610.2	3427.1

I think both should be kept.ping (g2h test) measures guest-initiated AF_VSOCK traffic to a host echo server, while ping-uds (h2g test) measures host-initiated UDS CONNECT traffic into Firecracker and then to a guest echo server.

Manciukic

Overall LGTM, just a nit to reduce boilerplate code. I need to double check this works, I will kickoff our performance CI as well so we can run these tests

Manciukic · 2026-03-04T14:43:31Z

+
+@pytest.mark.nonci
+@pytest.mark.parametrize("vcpus", [1, 2], ids=["1vcpu", "2vcpu"])
+def test_vsock_latency_g2h(uvm_plain_acpi, vcpus, metrics, bin_vsock_path):


nit: we can avoid some duped code by having the test setup in a separate fixture for all the tests in the file

@pytest.fixture def vsock_uvm(uvm_plain_acpi, request): """Fixture to initialize a microVM with vsock device.""" vcpus = request.param if hasattr(request, "param") else 1 mem_size_mib = 1024 vm = uvm_plain_acpi vm.spawn(log_level="Info", emit_metrics=True) vm.basic_config(vcpu_count=vcpus, mem_size_mib=mem_size_mib) vm.add_net_iface() vm.api.vsock.put(vsock_id="vsock0", guest_cid=3, uds_path="/" + VSOCK_UDS_PATH) vm.start() vm.pin_threads(0) return vm

then it can be used as

@pytest.mark.parametrize("vsock_uvm", [1, 2], indirect=True, ids=["1vcpu", "2vcpu"])

@aaron-ang can you use the fixture here as well?

Manciukic · 2026-03-04T14:51:01Z

Perf test dry-run: https://buildkite.com/firecracker/performance-a-b-tests/builds/833
PR tests: https://buildkite.com/firecracker/firecracker-pr/builds/16110
PR tests (optional): https://buildkite.com/firecracker/firecracker-pr-optional/builds/15364

codecov · 2026-03-04T14:55:22Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.79%. Comparing base (ae57b7a) to head (a7d1313).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5603   +/-   ##
=======================================
  Coverage   82.79%   82.79%           
=======================================
  Files         276      276           
  Lines       29764    29764           
=======================================
  Hits        24643    24643           
  Misses       5121     5121

Flag	Coverage Δ
5.10-m5n.metal	`83.08% <ø> (+<0.01%)`	⬆️
5.10-m6a.metal	`82.41% <ø> (ø)`
5.10-m6g.metal	`79.70% <ø> (ø)`
5.10-m6i.metal	`83.09% <ø> (+<0.01%)`	⬆️
5.10-m7a.metal-48xl	`82.40% <ø> (ø)`
5.10-m7g.metal	`79.70% <ø> (ø)`
5.10-m7i.metal-24xl	`83.05% <ø> (ø)`
5.10-m7i.metal-48xl	`83.05% <ø> (+<0.01%)`	⬆️
5.10-m8g.metal-24xl	`79.70% <ø> (ø)`
5.10-m8g.metal-48xl	`79.70% <ø> (ø)`
5.10-m8i.metal-48xl	`83.05% <ø> (-0.01%)`	⬇️
5.10-m8i.metal-96xl	`83.05% <ø> (-0.01%)`	⬇️
6.1-m5n.metal	`83.11% <ø> (+<0.01%)`	⬆️
6.1-m6a.metal	`82.44% <ø> (ø)`
6.1-m6g.metal	`79.70% <ø> (-0.01%)`	⬇️
6.1-m6i.metal	`83.11% <ø> (-0.01%)`	⬇️
6.1-m7a.metal-48xl	`82.43% <ø> (-0.01%)`	⬇️
6.1-m7g.metal	`79.70% <ø> (-0.01%)`	⬇️
6.1-m7i.metal-24xl	`83.12% <ø> (ø)`
6.1-m7i.metal-48xl	`83.12% <ø> (+<0.01%)`	⬆️
6.1-m8g.metal-24xl	`79.69% <ø> (-0.01%)`	⬇️
6.1-m8g.metal-48xl	`79.70% <ø> (ø)`
6.1-m8i.metal-48xl	`83.12% <ø> (ø)`
6.1-m8i.metal-96xl	`83.12% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Manciukic · 2026-03-11T16:05:07Z

Plotting the results, it's pretty slow compared to what we would have expected. Surprisingly some instances are much much faster. I think removing the delay would make this better.

test_vsock_latency 1 vcpu 6.1/6.1 p50 (unit=us)

aaron-ang · 2026-03-25T05:26:20Z

Plotting the results, it's pretty slow compared to what we would have expected. Surprisingly some instances are much much faster. I think removing the delay would make this better.

test_vsock_latency 1 vcpu 6.1/6.1 p50 (unit=us)

cool, thanks! I will incorporate your feedback this week

aaron-ang · 2026-03-30T02:19:42Z

@Manciukic please review

Manciukic

Just one nit but I haven't looked too deeply. I'll give it another go in our perf pipeline

Manciukic · 2026-04-01T14:42:12Z

+
+@pytest.mark.nonci
+@pytest.mark.parametrize("vcpus", [1, 2], ids=["1vcpu", "2vcpu"])
+def test_vsock_latency_g2h(uvm_plain_acpi, vcpus, metrics, bin_vsock_path):


@aaron-ang can you use the fixture here as well?

Manciukic · 2026-04-01T14:43:56Z

Perf test dry run : https://buildkite.com/firecracker/performance-a-b-tests/builds/866/

Manciukic · 2026-04-01T16:36:03Z

It looks better now! Not as good as network, but that's a separate problem (unit is microseconds)

Manciukic · 2026-04-08T14:17:51Z

@aaron-ang I kicked off CI again and there's another style issue. You can run this checks on your side before submitting running ./tools/devtool checkstyle

integration_tests/style/test_gitlint.py:21: in test_gitlint
    assert rc == 0, "Commit message violates gitlint rules: {}".format(stderr)
E   AssertionError: Commit message violates gitlint rules: Commit e273e3b9e0:
E     3: B6 Body message is missing
E     
E     Commit f371d4417d:
E     3: B6 Body message is missing
E     
E   assert 2 == 0
        _          = ''
        rc         = 2
        stderr     = ('Commit e273e3b9e0:\n'

aaron-ang · 2026-04-10T03:47:41Z

@aaron-ang I kicked off CI again and there's another style issue. You can run this checks on your side before submitting running ./tools/devtool checkstyle

integration_tests/style/test_gitlint.py:21: in test_gitlint
    assert rc == 0, "Commit message violates gitlint rules: {}".format(stderr)
E   AssertionError: Commit message violates gitlint rules: Commit e273e3b9e0:
E     3: B6 Body message is missing
E     
E     Commit f371d4417d:
E     3: B6 Body message is missing
E     
E   assert 2 == 0
        _          = ''
        rc         = 2
        stderr     = ('Commit e273e3b9e0:\n'

rebased with main and resolved git format error, but i'm still getting unrelated errors when running ./tools/devtool checkstyle regarding pci

Manciukic · 2026-04-10T08:52:06Z

weird, in any case it passed on the CI. Maybe there's an issue with that command on the local environment.

aaron-ang · 2026-04-15T01:28:05Z

is this good to merge?

Manciukic

sorry for the delay! Overall LGTM, just a few more ironing to get it in good shape

Manciukic · 2026-04-22T17:04:28Z

+        long long end_us = end.tv_sec * 1000000LL + end.tv_nsec / 1000;
+        long long rtt_us = end_us - start_us;
+
+        printf("rtt=%.3f us seq=%d\n", (double)rtt_us, seq);


We can just write the integer here as it'd always be .000.

Suggested change

printf("rtt=%.3f us seq=%d\n", (double)rtt_us, seq);

printf("rtt=%lld us seq=%d\n", rtt_us, seq);

Manciukic · 2026-04-22T17:04:51Z

+        long long end_us = end.tv_sec * 1000000LL + end.tv_nsec / 1000;
+        long long rtt_us = end_us - start_us;
+
+        printf("rtt=%.3f us seq=%d\n", (double)rtt_us, seq);


ditto

Suggested change

printf("rtt=%.3f us seq=%d\n", (double)rtt_us, seq);

printf("rtt=%lld us seq=%d\n", rtt_us, seq);

Manciukic · 2026-04-22T17:05:29Z

these changes are unrelated. Could you move them to another commit? Also, if we could share some code between the functional and the performance test, it could be interesting.

split the functional test change to a new commit and moved the helper for the fixture to utils_vsock.py

Manciukic · 2026-04-22T17:07:19Z

+                f"/tmp/vsock_helper ping 2 {ECHO_SERVER_PORT} {requests_per_round}"
+            )
+            samples.extend(consume_vsock_ping_output(ping_output))
+


we should verify we actually got the expected number of records, just in case

assert len(samples) == rounds * requests_per_round, ( f"Expected {rounds * requests_per_round} samples, got {len(samples)}" )

Manciukic · 2026-04-22T17:07:32Z

+            check=True,
+        )
+        samples.extend(consume_vsock_ping_output(result.stdout))
+


ditto

assert len(samples) == rounds * requests_per_round, ( f"Expected {rounds * requests_per_round} samples, got {len(samples)}" )

aaron-ang · 2026-04-23T01:48:52Z

sorry for the delay! Overall LGTM, just a few more ironing to get it in good shape

thanks for the comments, i've addressed them. please take a look :)

JamesC1305 · 2026-04-24T15:16:12Z

+        for _ in range(rounds):
+            _, ping_output, _ = vm.ssh.check_output(
+                f"/tmp/vsock_helper ping 2 {ECHO_SERVER_PORT} {requests_per_round}"
+            )


I noticed that there's a spike in latency at the beginning of every round. On my test instance (c7i.metal-24xl) the median was ~50us, but every 30th sample was ~1000us. Perhaps we can add an extra request at the beginning of the sample that gets excluded to negate this warmup?

This is for both h2g and g2h

implemented by adding 1 to requests_per_round then filtering out the first sample when adding it so samples list.

JamesC1305 · 2026-04-24T15:16:34Z

+    for _ in range(rounds):
+        result = subprocess.run(
+            [
+                bin_vsock_path,
+                "ping-uds",
+                uds_path,
+                str(ECHO_SERVER_PORT),
+                str(requests_per_round),
+            ],
+            capture_output=True,
+            text=True,
+            check=True,
+        )
+        samples.extend(consume_vsock_ping_output(result.stdout))


Same as above: https://github.com/firecracker-microvm/firecracker/pull/5603/changes#r3138630625

Add ping and ping-uds RTT commands to vsock_helper, plus g2h and h2g latency tests that drop the first sample per round to negate cold-start warmup spikes. Signed-off-by: Aaron Ang <aaron.angyd@gmail.com>

Use shared boot_vsock_vm helper for fixture setup. Add functional tests for vsock uds path override on snapshot restore. Signed-off-by: Aaron Ang <aaron.angyd@gmail.com>

aaron-ang force-pushed the vsock-latency-perf branch from 90ff871 to 75e4ea8 Compare January 6, 2026 02:22

JamesC1305 assigned Manciukic Jan 14, 2026

JamesC1305 added the Status: Awaiting author Indicates that an issue or pull request requires author action label Jan 14, 2026

aaron-ang force-pushed the vsock-latency-perf branch from 75e4ea8 to 1a80bd9 Compare February 26, 2026 04:04

Manciukic reviewed Mar 4, 2026

View reviewed changes

Manciukic reviewed Mar 11, 2026

View reviewed changes

Comment thread tests/integration_tests/performance/test_vsock.py Outdated

aaron-ang force-pushed the vsock-latency-perf branch from e7912cd to dd13239 Compare March 30, 2026 02:18

aaron-ang force-pushed the vsock-latency-perf branch from dd13239 to aa6f57b Compare March 30, 2026 02:25

Manciukic reviewed Apr 1, 2026

View reviewed changes

Comment thread tests/integration_tests/functional/test_vsock.py Outdated

aaron-ang force-pushed the vsock-latency-perf branch from aa6f57b to e273e3b Compare April 6, 2026 03:54

aaron-ang requested a review from Manciukic April 7, 2026 03:38

aaron-ang force-pushed the vsock-latency-perf branch from 08f5641 to 0aebe4a Compare April 10, 2026 03:46

Manciukic removed the Status: Awaiting author Indicates that an issue or pull request requires author action label Apr 15, 2026

Manciukic added the Status: Awaiting review Indicates that a pull request is ready to be reviewed label Apr 15, 2026

Manciukic reviewed Apr 22, 2026

View reviewed changes

aaron-ang force-pushed the vsock-latency-perf branch from 232274a to 97c26aa Compare April 23, 2026 01:46

JamesC1305 reviewed Apr 24, 2026

View reviewed changes

aaron-ang added 2 commits April 24, 2026 19:40

test(vsock): latency performance test

b7a348c

Add ping and ping-uds RTT commands to vsock_helper, plus g2h and h2g latency tests that drop the first sample per round to negate cold-start warmup spikes. Signed-off-by: Aaron Ang <aaron.angyd@gmail.com>

test(vsock): refactor fixtures and add uds path override tests

577bbd3

Use shared boot_vsock_vm helper for fixture setup. Add functional tests for vsock uds path override on snapshot restore. Signed-off-by: Aaron Ang <aaron.angyd@gmail.com>

aaron-ang force-pushed the vsock-latency-perf branch from 97c26aa to 577bbd3 Compare April 24, 2026 23:40

Merge branch 'main' into vsock-latency-perf

a7d1313

Manciukic approved these changes Apr 29, 2026

View reviewed changes

	printf("rtt=%.3f us seq=%d\n", (double)rtt_us, seq);
	printf("rtt=%lld us seq=%d\n", rtt_us, seq);

Conversation

aaron-ang commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Reason

License Acceptance

PR Checklist

Uh oh!

aaron-ang commented Jan 6, 2026

Uh oh!

Manciukic commented Jan 8, 2026

Uh oh!

JamesC1305 commented Feb 25, 2026

Uh oh!

aaron-ang commented Feb 25, 2026

Uh oh!

aaron-ang commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Manciukic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Manciukic commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Manciukic commented Mar 11, 2026

Uh oh!

aaron-ang commented Mar 25, 2026

Uh oh!

aaron-ang commented Mar 30, 2026

Uh oh!

Manciukic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Manciukic commented Apr 1, 2026

Uh oh!

Manciukic commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Manciukic commented Apr 8, 2026

Uh oh!

aaron-ang commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Manciukic commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aaron-ang commented Apr 15, 2026

Uh oh!

Manciukic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aaron-ang Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aaron-ang commented Jan 6, 2026 •

edited

Loading

aaron-ang commented Feb 26, 2026 •

edited

Loading

Manciukic commented Mar 4, 2026 •

edited

Loading

codecov Bot commented Mar 4, 2026 •

edited

Loading

Manciukic commented Apr 1, 2026 •

edited

Loading

aaron-ang commented Apr 10, 2026 •

edited

Loading

Manciukic commented Apr 10, 2026 •

edited

Loading

aaron-ang Apr 23, 2026 •

edited

Loading

aaron-ang Apr 24, 2026 •

edited

Loading