test(bench): add px4bench hardware bench-test suite for release qualification by mrpollo · Pull Request #27823 · PX4/PX4-Autopilot

mrpollo · 2026-07-06T17:47:30Z

CI builds every NuttX target but never boots one, and SITL runs the same code on a host OS whose threading, scheduling, and libc behave differently from the RTOS on the flight controller. Whole classes of defects only exist on real hardware: boot-time initialization ordering, link lifecycle across USB re-enumeration, storage and filesystem behavior, loop-rate drift, and concurrency bugs that are invisible on glibc but fatal on NuttX. A board can pass every SITL test and still ship with a dead telemetry link. This adds px4bench, a semi-automated bench qualification suite under Tools/bench_test/ that closes that gap with repeatable, scriptable verification on real boards: qualify a release candidate before cutting a branch, baseline a board across a firmware upgrade, or hand a beta to a test team knowing the basics hold.

Structure: a px4bench library package (connection, nsh shell over SERIAL_CONTROL, param/mission/MAVFTP protocol helpers, PASS/FAIL reporting), bench/ with seven real-firmware tests, sih/ with a simulation-in-hardware flight test, and a run_bench_suite.py orchestrator with a per-test watchdog. Every operation has a timeout and a timeout is reported as a FAIL naming what stalled; a hang is the finding. Scripts run directly with pymavlink and pyserial only; pip install -e is optional and ships just the library. The README carries a risk map tying each test to the subsystem it exercises, plus the rationale for using raw pymavlink rather than MAVSDK (the tests probe exactly the protocol layers MAVSDK abstracts away).

Bench tests, none of which arm the vehicle: boot_health (shell snapshot plus a --baseline mode that diffs uORB rates across firmware versions), reboot_loop, usb_replug (operator-assisted), link_forwarding (dual-link stress for the nested-send lock path from #27813), param_stress, mission_stress, log_transfer (logger plus MAVFTP download with ULog verification).

sih/flight_mission.py flies a full auto mission in simulation on the FMU itself (SYS_HITL=2, pwm_out_sim in place of real outputs): SIH airframe swap, takeoff, waypoint square, RTL, land detection, auto-disarm, post-flight ULog download into the report directory, then restores the original board configuration. With --viewer it tees MAVLink to UDP so a local Hawkeye renders the flight live.

Everything was validated against a px4_fmu-v6xrt over USB on macOS: the bench suite passes 5/5 and the flight test 15/15, repeatable back-to-back. The suite caught real issues during its own bring-up, including a live reproduction of the #27809 USB mavlink deadlock on the pre-fix firmware. Example qualification flight log: https://logs.px4.io/plot_app?log=dc4666e9-62ea-4cc1-bdc9-f32be52c5337

Draft until the two remaining hardware paths get bench time: usb_replug (needs an operator) and link_forwarding (needs a telemetry radio). Note for early adopters: running sih/flight_mission.py requires firmware built with CONFIG_MODULES_SIMULATION_SIMULATOR_SIH=y; a companion PR enables it on px4_fmu-v6xrt.

github-actions · 2026-07-06T17:56:54Z

🔎 FLASH Analysis

px4_fmu-v5x [Total VM Diff: 0 byte (0 %)]

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.0%     +55  [ = ]       0    .debug_abbrev
  -0.0%      -2  [ = ]       0    .debug_info
  -0.0%     -13  [ = ]       0    .debug_line
   -85.7%      -6  [ = ]       0    [Unmapped]
    -0.0%      -7  [ = ]       0    [section .debug_line]
  +0.0%     +40  [ = ]       0    TOTAL

px4_fmu-v6x [Total VM Diff: 0 byte (0 %)]

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.0%     +55  [ = ]       0    .debug_abbrev
  -0.0%      -2  [ = ]       0    .debug_info
  -0.0%      -5  [ = ]       0    .debug_line
     +67%      +2  [ = ]       0    [Unmapped]
    -0.0%      -7  [ = ]       0    [section .debug_line]
  +0.0%     +48  [ = ]       0    TOTAL

Updated: 2026-07-08T03:39:19

Semi-automated bench tests for qualifying v1.18 on real Pixhawk-class hardware after the TSAN/concurrency series (#27606, #27809, #27813). CI builds NuttX but never boots it, and the failure mode of this class of regression is a silent hang, not a crash: every operation here has a hard timeout and a timeout is reported as FAIL naming what stalled. Tests map to the reworked subsystems: boot_health (WorkQueue/uORB snapshot with baseline diffing), dual_link_forwarding (nested-send lock path under simultaneous two-link load), param_torture (set/readback and reboot persistence on SDLOG_UTC_OFFSET), mission_torture (dataman and the mission shared-state mutex, 220 items, link alternation), mavftp_log (logger on/off plus MAVFTP download and ULog verification), reboot_loop and the operator-assisted usb_replug (mavlink instance and RAM lifecycle across re-enumeration). run_bench_suite.py orchestrates the non-interactive tests with a per-test watchdog. Requires pymavlink/pyserial, pyulog optional. Nothing arms the vehicle. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

First hardware run (fmu-v6xrt over USB, macOS) surfaced three issues: - param_torture consumed PARAM_VALUE echoes positionally; one duplicate broadcast desynced every subsequent check. Drain stale messages before each set and match the echo against the expected value, reporting all observed values so a genuine wrong-echo bug remains visible. - the shell sentinel's second safety echo leaked into the next command's captured output; strip any sentinel line, not just the current one. - pymavlink 2.4.49 add_message() intermittently crashes with TypeError when the first message of an instanced type arrives with its instance field unset; wrap it defensively until fixed upstream. Full suite now passes 5/5 against fmu-v6xrt. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

sih_flight.py switches the board to a SIH airframe (SYS_HITL=2, physics on the FMU), flies takeoff, a 3-waypoint square, and RTL as an auto mission, asserting arming, takeoff, waypoint progression, landing, and auto-disarm with per-phase timeouts, then restores the original config. Covers the flight-logic paths the bench tests cannot reach; pwm_out_sim replaces real outputs so nothing is driven on the rails. --viewer tees every received MAVLink frame to UDP (frame per datagram, same framing as the SITL viewer channel) and enables the HIL_STATE_QUATERNION/HIL_ACTUATOR_CONTROLS streams on the board, so Hawkeye renders the flight live from a serial-connected board. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

…and sih/ split Turn the flat script pile into a standalone project so contributors can navigate and extend it: - px4bench/ shared library package: core primitives in __init__ (Reporter, connect, MavlinkShell, reboot/replug, viewer tee, mavlink status parsers, pymavlink add_message workaround) plus protocol modules params.py, missions.py, and ftp.py extracted from the tests. Zero helper duplication remains across scripts. - bench/ holds the real-firmware tests (boot_health, reboot_loop, usb_replug, link_forwarding, param_stress, mission_stress, log_transfer); sih/ holds the simulated flight (flight_mission), making the simulation/no-simulation boundary explicit. - pyproject.toml (px4bench 0.1.0, BSD-3-Clause, pymavlink/pyserial deps, pyulog extra) so pip install -e Tools/bench_test works; every script remains directly runnable without installation via a parent-dir path shim. - README rewritten contributor-first: architecture, per-test justification tied to the v1.18 risk areas, why pymavlink over MAVSDK, how to add a test, baseline workflow. - Consistent CLI surface (shared connection args; --report-dir replaces log_transfer's --outdir); decorative section banners removed. All hardware-learned behavior is preserved exactly: param echo drain and match-by-value, shell sentinel strip-all, the add_message workaround, explicit param save before reboot, mission clear before upload, RTL in MAV_FRAME_MISSION, and the post-flight ULog download. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

The README opened by framing the whole suite around one merged PR series. The value is general: CI never boots NuttX and SITL runs on a host OS, so defects in boot ordering, link lifecycle, storage, loop rates, and RTOS-only concurrency are invisible until a board is on a bench. Reword the introduction around that gap and the timeout-first design rule instead of the originating incident. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

Today's runs executed against whatever firmware happened to be on the board (an old April build at first) and nothing checked. For a qualification tool that is unacceptable: before any test starts the suite now establishes, and can control, exactly which build it is testing. New px4bench/firmware.py: .px4 metadata parsing and validation (fields verified against Tools/px_mkfw.py and px4_uploader.py), board identity via ver all over the nsh shell, flashing through Tools/px4_uploader.py with streamed output and a hard timeout, HW-arch to build-target inference by enumerating boards/, an early wrong-board check against firmware.prototype board_id, and the named firmware_identity check (git-hash prefix match against the artifact). run_bench_suite.py preflight supports four firmware sources, all converging on the same flash + verify path: keep what is on the board (--any-firmware), flash a local file (--firmware), build the inferred target from this source tree (--build, --target, --build-timeout), or download a GitHub release artifact (--release TAG|latest, via gh; asset naming verified against the v1.17.0 release). --expect-hash verifies without flashing. Flags are mutually exclusive; with none given, a TTY gets an operator menu covering the same four sources, and automation exits with an error before touching the board. The detected identity is printed, written to firmware.json in the suite report dir, and stamped into every test's report dir via PX4BENCH_FIRMWARE_INFO. If the board's mavlink is wedged the uploader cannot soft-reboot it into the bootloader (hit live today); after repeated reboot attempts the gate prints an operator instruction to replug USB so the uploader catches the bootloader at power-on. sih/flight_mission.py gains a verify-only --expect-hash and stamps the firmware identity into its report dir; it never flashes. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

New page in Platform Testing and CI covering px4bench: why on-hardware verification exists alongside SITL and CI, the firmware traceability gate, the bench test inventory, the SIH hardware-in-the-loop flight test with Hawkeye, baseline comparison for upgrade regression and golden-unit workflows, production end-of-line usage, and CI integration. Linked from the section index and the sidebar. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

github-actions · 2026-07-07T18:19:12Z

No broken links found in changed files.

Signed-off-by: Ramon Roche <mrpollo@gmail.com>

--target px4_fmu-v6xrt_bench previously skipped the early board_id check because only bare vendor_model targets were known; a label selects a .px4board config within the same board, so strip it and resolve to the board dir. The uploader still enforces board_id against the bootloader at flash time. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

… gate The simulated flight is the only test that exercises commander, navigator, and land-detector flight logic on real NuttX scheduling; leaving it out of the default suite meant the most valuable coverage was routinely skipped. It now runs last in the sequence. Arming safety gets two explicit paths: on a TTY the operator sees a prominent warning that the flight controller WILL ARM (simulated flight, pwm_out_sim replaces the real outputs, board must be bare) and must type 'arm'; declining records a skip, not a failure. For automation and manufacturing lines --allow-arming bypasses the prompt, and a non-TTY run without it skips the flight with a warning naming the flag. Before touching any configuration the test probes the live firmware for the SIH module ('simulator_sih status'; NuttX nsh replies 'nsh: <cmd>: command not found', apps/nshlib/nsh_parse.c) and skips with a warning when the module is absent. Shared plumbing in px4bench: EXIT_SKIP (75) recorded as SKIP by the orchestrator, shell_command_exists() for probe-and-skip, and arming_gate() used by both the orchestrator flow and the standalone flight test. Config mutation by the flight test is expected; it restores the original airframe and SYS_HITL afterwards. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

…em commands The firmware already ships sd_bench, sd_stress, and serial_test; the suite was not exploiting them. bench/storage_stress.py joins the default sequence before log_transfer (storage health first, then the test that depends on storage). It runs sd_bench with data verification and parses write/read throughput and fsync latency into named checks with generous FAIL floors (defaults justified in the README: 100 KB/s write, 200 KB/s read, 500 ms fsync; a working-but-slow card prints a WARNING instead of flapping), then sd_stress for file churn with a short default iteration count. Both phases probe-and-skip independently: a firmware without a command or a board without an SD card records SKIP with a warning, never FAIL. bench/serial_loopback.py is a standalone operator/fixture test for manufacturing: with a TX-RX loopback jumper installed on a chosen UART, serial_test transmits a known pattern and the session summary (rx/tx/rx err counts) plus pattern-error lines decide the verdict. Not in the default sequence because it needs a physical fixture. The firmware gate's --build path now prints a bench capability report from one shared table: after reading default.px4board plus the label overlay (labels merge both, cmake/kconfig.cmake:47-54) it lists whether the image will contain simulator_sih, sd_bench, sd_stress, and serial_test and which suite tests will run or skip. For anything missing it names a sibling variant that has it (exact --target) and the exact config line, and interactively offers to append the line to the local board config, stating that the edit is local and uncommitted. All four options are 'default n' in Kconfig, so the merged .px4board lines decide presence. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

…report The flight test is now part of the default suite; both the bench README and the bench testing guide said it ran separately. Update the sequence description, the risk map and suite tables (storage_stress and flight_mission rows, serial_loopback for fixtures), the SIH sections (probe-and-skip, the typed 'arm' confirmation, --allow-arming for automation), the production section (--allow-arming on the line, serial_loopback with a jumper on the fixture), and the firmware gate --build description (bench capability report with variant and config line recommendations). Storage threshold defaults are justified where the flags are documented. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

…nsh tasks A hardware run exhausted the board's task/fd table after a few probe cycles: shell open failed while heartbeat still worked, and continued writes eventually stalled the link. Not a firmware hang, a resource leak the tooling drove. The firmware spawns a new nsh task plus two pipes (four fds) on the first SERIAL_CONTROL write of a session (Mavlink::get_shell) and only frees them when a SERIAL_CONTROL message arrives with the RESPOND flag cleared (Mavlink::close_shell). Two tooling bugs combined to leak them: - MavlinkShell.open() returned True on any buffered byte, including leftover output from a prior session, so it reported a live shell without one. It now confirms an actual 'nsh>' prompt or a completed echo-sentinel round-trip before returning True, keeping the same timeout-and-report-failure behavior. - open/close cycles reopened before the single flags=0 teardown was processed, spawning a second shell before the first was freed. close() now sends the flags=0 teardown and then briefly drains incoming SERIAL_CONTROL until quiet (bounded, never a hang) so the firmware runs close_shell before a caller can reopen. Lifecycle: every MavlinkShell user now opens one session, runs all its work on it, and closes it in a finally so no error path leaks a shell. The SIH flight and storage tests run all their probes on a single shell (shell_command_exists already takes an open shell); flight_mission no longer opens a throwaway probe shell separate from the flight session beyond the unavoidable reboot boundary in enter_sih. Folds in the sentinel fix (run() sends the echo sentinels on their own input line, because nsh aborts a ';' chain when a command fails, which previously lost the sentinel for a failing probe and read as a stall). Signed-off-by: Ramon Roche <mrpollo@gmail.com>

…tence test The full-suite persistence check failed on hardware with 'after reboot SDLOG_UTC_OFFSET = 740 (expected 777)'. Firmware persistence is solid; this was a test race. A PARAM_SET propagates through the param system asynchronously, and phase_persistence ran 'param save' on the strength of the PARAM_VALUE echo alone. When save raced ahead of the commit it persisted the PRIOR value (740, the last value Phase 2 wrote). The echo was not even proof of the new value: wait_param_echo returned seen=[740, 777], a stale queued PARAM_VALUE from Phase 2 having slipped past drain_param_values. Make persistence deterministic with two gates instead of timing: 1. After setting the marker, read it back from the board until it reports MARKER_VALUE (read_until) before saving. This confirms the value is committed to RAM and trusts the board's read over any stale echo. 2. After 'param save', confirm the saved state with 'param show <name>' and require the '+' (saved) flag AND the marker value before rebooting (retry save once first). '*' means unsaved; the test never reboots on an unsaved marker. Flag columns per src/systemcmds/param/param.cpp:822 (x used, + saved, * unsaved, l locked). phase_set_readback now also trusts read_until over the echo, so the same stale-queued-echo does not fail it either; its pass criteria stay the readback match. Restores in phase_persistence and the final cleanup use read_until too. New shared helpers in px4bench.params: read_until(mav, name, expected, timeout) -> (ok, last_seen), and param_is_saved(shell, name) plus its parse_param_show() parser for the 'param show' saved flag and value. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

The --viewer path (tee MAVLink frames to UDP and enable the HIL_STATE_QUATERNION/HIL_ACTUATOR_CONTROLS streams so Hawkeye could render an SIH flight live) was never validated end to end: Hawkeye never actually rendered from it. Rather than ship an unproven feature, drop it. Removes px4bench.attach_viewer_tee (and its function-local socket use), the flight_mission --viewer/--viewer-port args, the viewer tee call and the stream-enable block, and the now-unused --board-dev arg that only fed those stream commands. The SIH flight logic, firmware gate, storage/serial tests, and the arming gate are unchanged. README and the bench testing docs page drop the Hawkeye section and all --viewer mentions. Signed-off-by: Ramon Roche <mrpollo@gmail.com>

Signed-off-by: Ramon Roche <mrpollo@gmail.com>

DronecodeBot · 2026-07-08T14:46:13Z

This pull request has been mentioned on Dronecode Forum | Open Source Drone Development. There might be relevant details there:

https://discuss.px4.io/t/px4-dev-call-jul-08-2026-team-sync-and-community-q-a/49199/1

github-actions Bot added kind:test Adds or improves tests. scope:tools Scripts, developer tools, packaging, or setup helpers. labels Jul 6, 2026

mrpollo force-pushed the mrpollo/bench-test-tools branch from dd454f9 to ac6f03d Compare July 6, 2026 21:37

mrpollo changed the title ~~test(bench): add NuttX bench smoke-test suite for release qualification~~ test(bench): add px4bench hardware bench-test suite for release qualification Jul 6, 2026

mrpollo mentioned this pull request Jul 6, 2026

feat(boards/fmu-v6xrt): enable SIH simulator module #27825

Closed

mrpollo added 6 commits July 7, 2026 09:51

mrpollo force-pushed the mrpollo/bench-test-tools branch from 848313b to d8b04a2 Compare July 7, 2026 16:56

github-actions Bot added the scope:docs label Jul 7, 2026

github-actions Bot added the scope:boards Board-specific changes or hardware definitions. label Jul 7, 2026

mrpollo added 2 commits July 7, 2026 14:13

feat(boards): add 6XRT variant with SIH HITL

047a292

Signed-off-by: Ramon Roche <mrpollo@gmail.com>

mrpollo force-pushed the mrpollo/bench-test-tools branch from a34312d to 37838cd Compare July 7, 2026 21:14

github-actions Bot added the scope:build-system CMake, Kconfig, board config, or build tooling. label Jul 7, 2026

mrpollo added 7 commits July 7, 2026 14:32

feat(boards): fmu-v5x bench tools to test variant

fdab3ac

Signed-off-by: Ramon Roche <mrpollo@gmail.com>

mrpollo marked this pull request as ready for review July 8, 2026 03:34

github-actions Bot added the scope:testing Unit, integration, fuzzing, or test data. label Jul 8, 2026

mrpollo merged commit 1343ad1 into main Jul 8, 2026
83 checks passed

mrpollo deleted the mrpollo/bench-test-tools branch July 8, 2026 04:33

farhangnaderi mentioned this pull request Jul 9, 2026

px4bench: 1.17 backport issue tracker #27840

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(bench): add px4bench hardware bench-test suite for release qualification#27823

test(bench): add px4bench hardware bench-test suite for release qualification#27823
mrpollo merged 16 commits into
mainfrom
mrpollo/bench-test-tools

mrpollo commented Jul 6, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

Uh oh!

DronecodeBot commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mrpollo commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔎 FLASH Analysis

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

Uh oh!

DronecodeBot commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mrpollo commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading