Skip to content

tests: add periodic sanitizer integration test pipeline#5722

Open
aaron-ang wants to merge 1 commit intofirecracker-microvm:mainfrom
aaron-ang:test/sanitizer-integration-pipeline
Open

tests: add periodic sanitizer integration test pipeline#5722
aaron-ang wants to merge 1 commit intofirecracker-microvm:mainfrom
aaron-ang:test/sanitizer-integration-pipeline

Conversation

@aaron-ang
Copy link
Copy Markdown

@aaron-ang aaron-ang commented Feb 26, 2026 •
edited
Loading

Changes

  • Add tests/integration_tests/build/test_sanitizers.py to run Rust vmm integration tests under ASAN (address) as a nonci build test.
  • Add .buildkite/pipeline_sanitizers.py to generate Buildkite sanitizer steps per architecture (m6i.metal, m7g.metal).
  • Update tests/README.md with sanitizer test run instructions and pipeline reference.
  • Fix tools/test.sh to avoid unbound-variable failure outside Buildkite by changing "$BUILDKITE" to "${BUILDKITE:-false}".

Reason

Closes #1662.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

  • I have read and understand CONTRIBUTING.md.
  • I have run tools/devtool checkbuild --all to verify that the PR passes
    build checks on all supported architectures.
  • I have run tools/devtool checkstyle to verify that the PR passes the
    automated style checks.
  • I have described what is done in these changes, why they are needed, and
    how they are solving the problem in a clear and encompassing way.
  • I have updated any relevant documentation (both in code and in the docs)
    in the PR.
  • I have mentioned all user-facing changes in CHANGELOG.md.
    N/A: this PR does not introduce user-facing behavior changes.
  • If a specific issue led to this PR, this PR closes the issue.
  • When making API changes, I have followed the
    Runbook for Firecracker API changes.
    N/A: this PR does not make API changes.
  • I have tested all new and changed functionalities in unit tests and/or
    integration tests.
  • I have linked an issue to every new TODO.
  • This functionality cannot be added in rust-vmm.

@aaron-ang aaron-ang force-pushed the test/sanitizer-integration-pipeline branch 2 times, most recently from 34ee078 to 4de013b Compare February 26, 2026 06:19
@ilstam ilstam added the Status: Awaiting review Indicates that a pull request is ready to be reviewed label Mar 4, 2026
@zulinx86 zulinx86 removed the Status: Awaiting review Indicates that a pull request is ready to be reviewed label Mar 4, 2026
zulinx86
zulinx86 reviewed Mar 8, 2026
View reviewed changes
Comment thread tests/integration_tests/build/test_sanitizers.py Outdated
Comment thread tests/integration_tests/build/test_sanitizers.py Outdated
Comment thread tests/integration_tests/build/test_sanitizers.py Outdated
@zulinx86
Copy link
Copy Markdown
Contributor

zulinx86 commented Mar 8, 2026

Hi @aaron-ang , sorry for being late to review and thank you so much of your contribution! I left some comments, so could you take a look at them?

@aaron-ang
Copy link
Copy Markdown
Author

aaron-ang commented Mar 25, 2026

Hi @aaron-ang , sorry for being late to review and thank you so much of your contribution! I left some comments, so could you take a look at them?

hi, apologies for the delay. I will make the necessary edits this week.

@aaron-ang aaron-ang force-pushed the test/sanitizer-integration-pipeline branch from 4de013b to 99c2b1f Compare April 6, 2026 03:29
@aaron-ang aaron-ang requested a review from zulinx86 April 7, 2026 03:39
@zulinx86 zulinx86 requested a review from micz010 as a code owner April 15, 2026 08:21
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 15, 2026 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.07%. Comparing base (636f7fa) to head (e1af357).

⚠️ Current head e1af357 differs from pull request most recent head 271859e

Please upload reports for the commit 271859e to get more accurate results.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #5722   +/-   ##
=======================================
  Coverage   83.07%   83.07%           
=======================================
  Files         276      276           
  Lines       29541    29541           
=======================================
  Hits        24541    24541           
  Misses       5000     5000
Flag Coverage Δ
5.10-m5n.metal 83.40% <ø> (+<0.01%) ⬆️
5.10-m6a.metal 82.73% <ø> (+<0.01%) ⬆️
5.10-m6g.metal 79.99% <ø> (ø)
5.10-m6i.metal 83.39% <ø> (-0.01%) ⬇️
5.10-m7a.metal-48xl 82.72% <ø> (ø)
5.10-m7g.metal 79.99% <ø> (ø)
5.10-m7i.metal-24xl 83.36% <ø> (ø)
5.10-m7i.metal-48xl 83.37% <ø> (+<0.01%) ⬆️
5.10-m8g.metal-24xl 79.99% <ø> (ø)
5.10-m8g.metal-48xl 79.99% <ø> (ø)
5.10-m8i.metal-48xl 83.38% <ø> (+<0.01%) ⬆️
5.10-m8i.metal-96xl 83.37% <ø> (ø)
6.1-m5n.metal 83.42% <ø> (+<0.01%) ⬆️
6.1-m6a.metal 82.76% <ø> (ø)
6.1-m6g.metal 79.99% <ø> (-0.01%) ⬇️
6.1-m6i.metal 83.42% <ø> (ø)
6.1-m7a.metal-48xl 82.75% <ø> (ø)
6.1-m7g.metal 79.99% <ø> (-0.01%) ⬇️
6.1-m7i.metal-24xl 83.43% <ø> (ø)
6.1-m7i.metal-48xl 83.44% <ø> (+<0.01%) ⬆️
6.1-m8g.metal-24xl 79.99% <ø> (ø)
6.1-m8g.metal-48xl 79.99% <ø> (ø)
6.1-m8i.metal-48xl 83.44% <ø> (+<0.01%) ⬆️
6.1-m8i.metal-96xl 83.44% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

zulinx86
zulinx86 reviewed Apr 15, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@zulinx86 zulinx86 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for address the previous comments! I left additional comments for the buildkite python script.

Comment thread .buildkite/pipeline_sanitizers.py
Comment thread .buildkite/pipeline_sanitizers.py Outdated
@aaron-ang
test: add sanitizer integration pipeline
271859e 
Add a dedicated integration test that runs vmm Rust integration tests
under ASAN, and wire a Buildkite pipeline generator for periodic
sanitizer runs.

Also document the run command and harden tools/test.sh for local
runs under set -u by guarding BUILDKITE variable access.

Signed-off-by: Aaron Ang <aaron.angyd@gmail.com>
@aaron-ang aaron-ang force-pushed the test/sanitizer-integration-pipeline branch from e1af357 to 271859e Compare April 16, 2026 01:04
@zulinx86
Copy link
Copy Markdown
Contributor

zulinx86 commented Apr 16, 2026

Thank you for addressing them! LGTM

I kicked the CI and I'm seeking another reviewer on this change...

@zulinx86 zulinx86 added the Status: Awaiting review Indicates that a pull request is ready to be reviewed label Apr 16, 2026
@zulinx86
Copy link
Copy Markdown
Contributor

zulinx86 commented Apr 16, 2026

To make it easier for another reviewer to review, the following is the log when executed on my dev env.

$ ./tools/devtool -y test --no-build -- -m nonci -s --log-cli-level=DEBUG integration_tests/build/test_sanitizers.py
(omitted)

integration_tests/build/test_sanitizers.py::test_rust_integration_tests_under_sanitizer[address] 
---------------------------------------------------------------------------------- live log call -----------------------------------------------------------------------------------
07:48:46.747 commands: DEBUG 
[75] Command:
rustup toolchain list | grep nightly
[75] stdout:
nightly-2025-11-21-x86_64-unknown-linux-gnu

[75] stderr:

Returned error code: 0
07:48:57.839 commands: DEBUG 
[86] Command:
CARGO_TARGET_DIR="/firecracker/build/cargo_target/sanitizers/address" RUSTFLAGS="-Zsanitizer=address" cargo +nightly-2025-11-21-x86_64-unknown-linux-gnu test --target x86_64-unknown-linux-gnu -p vmm --test devices --test integration_tests --test io_uring -- --test-threads=1
[86] stdout:

running 3 tests
test test_issue_hangup ... ok
test test_issue_serial_hangup_anon_pipe_while_registered_stdin ... ok
test test_issue_serial_hangup_anon_pipe_while_unregistered_stdin ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


running 9 tests
test test_build_and_boot_microvm ... Hello, world!
Hello, world!
Hello, world!
Hello, world!
ok
test test_build_microvm ... Hello, world!
Hello, world!
Hello, world!
Hello, world!
ok
test test_create_and_load_snapshot ... ok
test test_dirty_bitmap_success ... ok
test test_disallow_dump_cpu_config_without_pausing ... ok
test test_disallow_snapshots_without_pausing ... ok
test test_pause_resume_microvm ... ok
test test_preboot_load_snap_disallowed_after_boot_resources ... ok
test test_snapshot_load_sanity_checks ... ok

test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 10.61s


running 8 tests
test test_eventfd ... ok
test test_read ... ok
test test_restrictions ... ok
test test_ring_new ... ok
test test_ring_push ... ok
test test_ring_submit ... ok
test test_submit_and_wait_all ... ok
test test_write ... ok

test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.31s


[86] stderr:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.11s
     Running tests/devices.rs (build/cargo_target/sanitizers/address/x86_64-unknown-linux-gnu/debug/deps/devices-3e74fdce113b37d2)
     Running tests/integration_tests.rs (build/cargo_target/sanitizers/address/x86_64-unknown-linux-gnu/debug/deps/integration_tests-33d69b9015ea9dde)
     Running tests/io_uring.rs (build/cargo_target/sanitizers/address/x86_64-unknown-linux-gnu/debug/deps/io_uring-a50fee5bb309b0be)

Returned error code: 0
PASSED

----------------------------------------------------------------------------------- JSON report ------------------------------------------------------------------------------------
report saved to: ../test_results/test-report.json
=============================================================================== slowest 10 durations ===============================================================================
11.11s call     integration_tests/build/test_sanitizers.py::test_rust_integration_tests_under_sanitizer[address]
0.00s setup    integration_tests/build/test_sanitizers.py::test_rust_integration_tests_under_sanitizer[address]
0.00s teardown integration_tests/build/test_sanitizers.py::test_rust_integration_tests_under_sanitizer[address]
================================================================================ 1 passed in 11.13s ================================================================================
[Firecracker devtool 2026-04-16T07:48:58+00:00] Finished test run ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zulinx86 zulinx86 zulinx86 approved these changes

@xmarcalx xmarcalx Awaiting requested review from xmarcalx

@kalyazin kalyazin Awaiting requested review from kalyazin kalyazin is a code owner

@pb8o pb8o Awaiting requested review from pb8o

@Manciukic Manciukic Awaiting requested review from Manciukic Manciukic is a code owner

@micz010 micz010 Awaiting requested review from micz010 micz010 is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

@zulinx86 zulinx86

Labels

Status: Awaiting review Indicates that a pull request is ready to be reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Tests] Periodically run Firecracker integration tests under sanitizers

3 participants

@aaron-ang @zulinx86 @ilstam