feat(hardfault): add SRAM hardfault/watchdog DroneCAN logging by marinauterion · Pull Request #27787 · PX4/PX4-Autopilot

marinauterion · 2026-07-01T15:46:25Z

Solved Problem

On boards without a usable persistant memory (filesystem, BBSRAM, PROGMEM, SSARC, …), there is no possibility to save the full-hardfault-context (registers + interrupt/user stack).
Neither does the task_watchdog log starvation via dronecan.
There is currently no possibility to preprocess the linker-file.

UAVcan-Node does not publish its status.
If a critical task (like hardfault_streamer) does publish its log-msgs to early (before an ID was assigned),
log-msgs get lost

Solution

Hardfault-Dump:
This PR introduces a .noinit SRAM region, that survives warm resets.
On the next boot, the hardfault_stream streams the raw hex dump with a CRC32 via PX4_ERR so it is forwarded over DroneCAN LogMessage.

Task_Watchdog
The same streaming path is used by task_watchdog: when CPU-starvation is
detected the task dump and a cpuload snapshot are emitted via PX4_ERR.

Changes:

- introduce BOARD_HAS_RAM_HARDFAULT_DUMP to tell the system, that there is no persistent storage (filesystem, BBSRAM, PROGMEM, SSARC, …) available
- hardfault_log.h: add BOARD_HAS_RAM_HARDFAULT_DUMP branch with px4_ram_hardfault_dump_s; fix #if HAS_BBSRAM (was #if defined()) so STM32F4 boards without BBSRAM fall through to the RAM dump path
- nuttx/CMakeLists.txt: add logic to inject custom cmake-files
- board_crashdump.c: save hardfault to SRAM (.noinit) and stamp magic
- hardfault_stream: stream hardfault-hex dump + CRC via DroneCAN if configured
- task_watchdog: stream dump + cpuload via PX4_INFO if configured

UAVcan_node
Does now publish it status via uORB
This status is used by hardfault-streamer and task_watchdog.

Test:

trigger a hardfault on a board
collect it via dronecan and save it. (eg hardfault_dump.dmp) it should look like:

125	14:54:47.217	ERROR	hardfault_stream	0000 0f000000fcb80120ad000000d6000000d0b90120f0000000efffffffac20002003000000a8b201200000
125	14:54:47.367	ERROR	hardfault_stream	002a 0000000000000000000000000000e9ffffff000000000000000000000000000000000000000000000000
125	14:54:47.517	ERROR	hardfault_stream	0054 000000000000000000000000000000000000000000000000000000000000000000000000000000000000
125	14:55:02.518	ERROR	hardfault_stream	10bc efbeaddeefbeaddeefbeaddeefbeaddeefbeaddeefbeaddeefbeaddeefbeaddeefbeaddeefbeaddeefbe
…
125	14:55:05.367	ERROR	hardfault_stream	13da 0000000000003d000000486a01200000000000000000801b00208870012084d709080000000000000000
125	14:55:05.517	ERROR	hardfault_stream	1404 586a0120dcd70908efd9ef7df203000010b10120c4d70908810000008000000000000000940009080000
125	14:55:05.668	ERROR	hardfault_stream	142e 000000000000c1e5040890b1012000000000b0b10120c1e50408a0b10120319e0000486b0120486a0120
125	14:55:05.815	ERROR	hardfault_stream	crc b2f8ddd0

used the provided python-script to convert the hex-blob into a emdbg-compatible file-format

python3 Tools/hardfaults_rawhex_decode.py memcpy_to_null/dump.dmp
WARN: 1 missing chunk(s) at offsets: 0x0516
WARN: CRC mismatch - expected 0xb2f8ddd0, got 0x23c5d065 (possible bit-flip in stream)
Size:  total=5208 B  istack=768 B  ustack=4096 B
Task:  'task_watchdog'  pid=214  line=173
File:  'armv7-m/arm_hardfault.c'
PC:    0x0805b41e  LR: 0x08010e20  SP: 0x2001b9d0
Wrote: memcpy_to_null/dump.hardfault.log
Wrote: memcpy_to_null/coredump_dump.txt

call postmortem-dbg

python3 -m emdbg.bench.fmu --target TARGET \
  --coredump PATH/TO/DUMP/memcpy_to_null/coredump_dump.txt

Reading symbols from …

515	int TaskWatchdog::print_usage(const char *reason)
   0x0805b414 <_ZN13task_watchdog12TaskWatchdog14custom_commandEiPPc+76>:	2280    	movs	r2, #128	@ 0x80
   0x0805b416 <_ZN13task_watchdog12TaskWatchdog14custom_commandEiPPc+78>:	4669    	mov	r1, sp
  …

(gdb) bt
#0  0x0805b41e in task_watchdog::TaskWatchdog::custom_command (argc=<optimized out>, argv=<optimized out>)
    at px4/src/modules/task_watchdog/TaskWatchdog.cpp:515
#1  0x0807f1b4 in ModuleBase::main (desc=..., argc=3, argv=0x2001b2a8)
    at px4/platforms/common/module_base.cpp:82
#2  0x08018b42 in nxtask_startup (entrypt=0x805bb15 <task_watchdog_main(int, char**)>, entrypt@entry=0x3, argc=3, argv=0x2001b2a8)
    at px4/platforms/nuttx/NuttX/nuttx/libs/libc/sched/task_startup.c:70
#3  0x08012bd2 in nxtask_start ()
    at px4/platforms/nuttx/NuttX/nuttx/sched/task/task_start.c:134
#4  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

github-actions · 2026-07-01T17:04:04Z

🔎 FLASH Analysis

px4_fmu-v5x [Total VM Diff: 8 byte (0 %)]

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.0%      +8  +0.0%      +8    .text
    +0.6%      +8  +0.6%      +8    McAutotuneAttitudeControl::updateStateMachine()
    +0.0%      +4  +0.0%      +4    [section .text]
   -30.8%      -4 -30.8%      -4    g_nullstring
  +0.0%     +97  [ = ]       0    .debug_abbrev
  +0.0%     +56  [ = ]       0    .debug_info
  +0.0%     +55  [ = ]       0    .debug_line
    [NEW]      +5  [ = ]       0    [Unmapped]
    +0.0%     +50  [ = ]       0    [section .debug_line]
  +0.0%     +36  [ = ]       0    .debug_loclists
  +0.0%     +11  [ = ]       0    .debug_rnglists
   -33.3%      -1  [ = ]       0    [Unmapped]
    +0.0%     +12  [ = ]       0    [section .debug_rnglists]
  +0.0%      +5  [ = ]       0    .debug_str
  -0.1%      -8  [ = ]       0    [Unmapped]
  +0.0%    +260  +0.0%      +8    TOTAL

px4_fmu-v6x [Total VM Diff: 8 byte (0 %)]

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.0%      +8  +0.0%      +8    .text
    +0.6%      +8  +0.6%      +8    McAutotuneAttitudeControl::updateStateMachine()
  +0.0%     +97  [ = ]       0    .debug_abbrev
  +0.0%     +56  [ = ]       0    .debug_info
  +0.0%     +47  [ = ]       0    .debug_line
   -75.0%      -3  [ = ]       0    [Unmapped]
    +0.0%     +50  [ = ]       0    [section .debug_line]
  +0.0%     +36  [ = ]       0    .debug_loclists
  +0.0%     +11  [ = ]       0    .debug_rnglists
   -33.3%      -1  [ = ]       0    [Unmapped]
    +0.0%     +12  [ = ]       0    [section .debug_rnglists]
  +0.0%      +5  [ = ]       0    .debug_str
  -0.1%      -8  [ = ]       0    [Unmapped]
  +0.0%    +252  +0.0%      +8    TOTAL

Updated: 2026-07-09T14:12:50

github-actions · 2026-07-02T10:14:59Z

No broken links found in changed files.

alexcekay · 2026-07-02T15:15:52Z

@hamishwillee: Could you also please have a look at the docs part here? Thanks a lot.

alexcekay · 2026-07-02T15:21:23Z

+	char line[90];
+
+#ifdef BOARD_HAS_RAM_HARDFAULT_DUMP
+	wait_for_transport_ready();


This is dangerous! You do a blocking wait here while the TaskWatchdog is in boosted priority. This has to run as quickly as possible and must terminate quickly otherwise you will starve other tasks in case the node for whatever reason does not reach MODE_OPERATIONAL.

This is in general problematic as the whole output will be starved for 30s due to all the log waiting.

Please check the priorities of tasks.

wait_for_transport_ready() does call sleep(1s)
thus the task does not starve other tasks

the watchdog loses his higher prio afet 1.5s anyway

if we would use eg a timeout

we were able to catch other starving-tasks, but cuz we assume, in that case, that the transport-medium is dead, there is no need to be ready for other starvations. -> instead, if the transport gets possible after time x, we still get the output of the starvation-reason; which is the most valuable

hamishwillee · 2026-07-02T21:57:55Z

@@ -0,0 +1,109 @@
+# Hardfault Logging on Boards Without Non-Volatile Memory
+
+This section introduces what is need by a board-config to persist a hardfault over warm-reset (only), on boards without usable non-volatile storage.


Suggested change

This section introduces what is need by a board-config to persist a hardfault over warm-reset (only), on boards without usable non-volatile storage.

This section introduces what is needed by a board-config to persist a hardfault over warm-reset (only), on boards without usable non-volatile storage.

hamishwillee · 2026-07-02T21:58:30Z

@@ -0,0 +1,109 @@
+# Hardfault Logging on Boards Without Non-Volatile Memory


I'm away for a week or I would subedit this. If you merge next week please prettier the document.

alexcekay · 2026-07-02T15:22:35Z

-	if (up_time && hrt_elapsed_time(&up_time) > 30_s) {
-		up_time = 0;
+	if (_up_time && hrt_elapsed_time(&_up_time) > 30_s) {
+		_up_time = 0;


Question: But doesn't this mean that _up_time will never be more than 30s?

yes: on uacvannode-boards, uavcannode is a watchdogged-application
if the node does run (again after spinOnce) the flag BOARD_RESET_MODE_RTC_BOOT_FWOK is reset
(the idea seams to be, to be able to detect broken flashed-images)

it needs to reset BOARD_RESET_MODE_RTC_BOOT_FWOK once

BUT this is not evaluated anywhere as far as I can tell

// This is done only once to signify the node has run 30 seconds if (_up_time && hrt_elapsed_time(&_up_time) > 30_s) { _up_time = 0; board_configure_reset(BOARD_RESET_MODE_RTC_BOOT_FWOK, 0); } … BOARD_RESET_MODE_RTC_BOOT_FWOK = 4 /* Set by a a watch dogged application after running > 30 Seconds */

alexcekay · 2026-07-07T11:23:54Z

 	sched_lock();

-	for (int i = 0; i < CONFIG_FS_PROCFS_MAX_TASKS; i++) {
+	for (int i = 0; i < MAX_DUMP_TASKS; i++) {


This breaks the whole task watchdog. If the task watchdog has a PID > MAX_DUMP_TASKS it does not find its own PID thus never arms!

yes!
changes it back to CONFIG_FS_PROCFS_MAX_TASKS
now MAX_DUMP_TASKS is only used in capture_and_write_dump()

alexcekay · 2026-07-07T11:26:49Z

+# Input: hex extraction
+# ---------------------------------------------------------------------------
+# Search for chunk-pattern " xxxx <hexdata> xxxx… <hexdata>"
+_CHUNK_RE = re.compile(r"(?<![0-9a-fA-F])([0-9a-fA-F]{4})\s+([0-9a-fA-F]+)\s*$")


What exactly should this regex detect? Does this really only match the hard fault stream and not also other kind of things logged? I am asking because what it searches for is very general.

yes, this was very coarse, (hardfault was filtered later)

now changed for more precise pre-selection:

does now search for beginning hardfault_stream-string
leave space for variations eg
hardfault_stream] xxxx DATE_hardfault_stream] xxxx, hardfault_stream xxxx
_CHUNK_RE = re.compile(r"hardfault_stream\W*?([0-9a-fA-F]{4})\s+([0-9a-fA-F]+)\s*$")
_CRC_RE = re.compile(r"hardfault_stream\W*?crc\s+([0-9a-fA-F]{8})\b")

alexcekay · 2026-07-07T12:00:49Z

+			const uint8_t *ptr   = reinterpret_cast<const uint8_t *>(&g_px4_ram_hardfault.context);
+			const size_t   total = sizeof(fullcontext_s);
+
+			if (_ram_off < total) {


nit: For good measure please init _ram_off to 0 after setting total. Just in case someone restarts the state machine in the future.

(there should be only 1 hardfault per runtime)

is reset on case State::SearchHardFault:

alexcekay · 2026-07-07T12:52:42Z

+]
+
+
+def format_coredump(p: ParsedHardfault, cpuid: int) -> str:


Why does this create a log and a core dump? The normal post mortem debugging uses the fault saved on SD, so one file. Why are both required here?

the hardfault.log - file provide the same output as a console-output on hardfault
(This is also quiet handy to easily get a first overview on registers without starting gdb
or tell if the script has parsing flaws)

Build datetime: unknown unknown Build url: unknown Processor registers: from 0x2001b9ec r0:0x00000000 r1:0x2001bb40 r2:0x00000000 r3:0x00000000 r4:0xffffffef r5:0x200020a4 r6:0x00000003 r7:0x2001b398 … exe return:0xffffffe9 Fault status registers: from NVIC CFSR: 00010000 HFSR: 40000000 DFSR: 00000008 BFAR: e000ed38 AFSR: 00000000 IRQ Stack: sp: 20001550 base: 200012b8 size: 00000300 0x20001540 0x000001b0 0x00000038 0x00000002 0x00000020 0x40000000 0x00000003 0x2001b9ec 0x00010000 … User Stack: …

while the format_coredump is less human-readable and feed easily into gdb

…neCAN logging On boards without a usable persistant (filesystem, BBSRAM, PROGMEM, SSARC, …), a possibility is needed to save the full-hardfault-context (registers + interrupt/user stack). This PR introduces a .noinit SRAM region, that survives warm resets. On the next boot, the hardfault_stream streams the raw hex dump with a CRC32 via PX4_ERR so it is forwarded over DroneCAN LogMessage. The same streaming path is used by task_watchdog: when CPU-starvation is detected the task dump and a cpuload snapshot are emitted via PX4_ERR (MAVLink on FMU boards, DroneCAN on canBoards) instead of writing to SD. Changes: - introduce BOARD_HAS_RAM_HARDFAULT_DUMP to tell the system, that there is no persistent storage (filesystem, BBSRAM, PROGMEM, SSARC, …) available - hardfault_log.h: add BOARD_HAS_RAM_HARDFAULT_DUMP branch with px4_ram_hardfault_dump_s; fix #if HAS_BBSRAM (was #if defined()) so STM32F4 boards without BBSRAM fall through to the RAM dump path - nuttx/CMakeLists.txt: add logic to inject custom cmake-files - board_crashdump.c: save hardfault to SRAM (.noinit) and stamp magic - hardfault_stream: stream hardfault-hex dump + CRC via DroneCAN if configured - task_watchdog: stream dump + cpuload via PX4_ERR if configured - uavcannode: wait for valid id before draining log_msgs

This status is used by hardfault-streamer and task_watchdog. If UAVcan-logging is active, we have to wait till an ID is assigned, otherwise log-msgs get lost

…og-msg-style add: task_watchdog-config now allows setting the amount of logged tasks on starvation fix: linker-script only include linker_preprocessor if linker is linked fix: documentation fix: small pr-reviews fix: publish task_watchdog dump with 200Hz fix: make variable magic in RAM-hardfault-dump volatile

marinauterion requested a review from alexcekay July 1, 2026 15:46

github-actions Bot added the scope:logging ULog, logger, replay, events, or diagnostics. label Jul 1, 2026

marinauterion force-pushed the pr-hardfault-can-logging branch 2 times, most recently from f9755c4 to 0ea677e Compare July 1, 2026 16:58

github-actions Bot added scope:drivers Device drivers and hardware interfaces. scope:middleware DDS, ROS 2, Cyphal/UAVCAN, zenoh, or bridge layers. labels Jul 1, 2026

marinauterion force-pushed the pr-hardfault-can-logging branch 2 times, most recently from c9bbf65 to f0bf1d1 Compare July 1, 2026 18:04

github-actions Bot added the scope:tools Scripts, developer tools, packaging, or setup helpers. label Jul 1, 2026

mrpollo changed the title ~~feat(hardfault_stream,task_watchdog): add SRAM hardfault/watchdog DroneCAN logging~~ feat(hardfault): add SRAM hardfault/watchdog DroneCAN logging Jul 1, 2026

github-actions Bot added the kind:feature Request or change that adds new functionality. label Jul 1, 2026

marinauterion force-pushed the pr-hardfault-can-logging branch from f0bf1d1 to 4ef874c Compare July 2, 2026 09:55

github-actions Bot added the scope:docs label Jul 2, 2026

marinauterion assigned Phil-Engljaehringer and unassigned Phil-Engljaehringer Jul 2, 2026

marinauterion requested a review from Phil-Engljaehringer July 2, 2026 12:18

Phil-Engljaehringer reviewed Jul 2, 2026

View reviewed changes

Comment thread src/modules/task_watchdog/TaskWatchdog.cpp Outdated

Comment thread src/modules/task_watchdog/TaskWatchdog.cpp

alexcekay reviewed Jul 2, 2026

View reviewed changes

Comment thread src/lib/systemlib/hardfault_log.h Outdated

Comment thread src/lib/systemlib/hardfault_log.h

Comment thread src/modules/task_watchdog/TaskWatchdog.hpp Outdated

Comment thread src/modules/task_watchdog/TaskWatchdog.cpp Outdated

alexcekay requested a review from hamishwillee July 2, 2026 15:15

alexcekay reviewed Jul 2, 2026

View reviewed changes

Phil-Engljaehringer reviewed Jul 2, 2026

View reviewed changes

Comment thread src/modules/hardfault_stream/HardfaultStream.cpp Outdated

Comment thread src/modules/task_watchdog/TaskWatchdog.cpp Outdated

github-actions Bot added scope:build-system CMake, Kconfig, board config, or build tooling. scope:boards Board-specific changes or hardware definitions. labels Jul 2, 2026

hamishwillee reviewed Jul 2, 2026

View reviewed changes

Comment thread docs/en/debug/hardfault_no_nvram.md Outdated

hamishwillee reviewed Jul 2, 2026

View reviewed changes

marinauterion force-pushed the pr-hardfault-can-logging branch 9 times, most recently from 55362ee to 0e5d500 Compare July 6, 2026 12:16

marinauterion marked this pull request as ready for review July 6, 2026 12:19

marinauterion requested a review from alexcekay July 6, 2026 12:19

marinauterion force-pushed the pr-hardfault-can-logging branch from 0e5d500 to 42a3f1f Compare July 6, 2026 13:24

alexcekay reviewed Jul 7, 2026

View reviewed changes

marinauterion added 2 commits July 9, 2026 10:37

feat[logging]: uavcannode does now publish it status

25a42ce

This status is used by hardfault-streamer and task_watchdog. If UAVcan-logging is active, we have to wait till an ID is assigned, otherwise log-msgs get lost

marinauterion force-pushed the pr-hardfault-can-logging branch 10 times, most recently from 033d2e2 to feaa826 Compare July 9, 2026 13:15

marinauterion force-pushed the pr-hardfault-can-logging branch from feaa826 to 1b362cb Compare July 9, 2026 14:06

		@@ -0,0 +1,109 @@
		# Hardfault Logging on Boards Without Non-Volatile Memory

		This section introduces what is need by a board-config to persist a hardfault over warm-reset (only), on boards without usable non-volatile storage.

	This section introduces what is need by a board-config to persist a hardfault over warm-reset (only), on boards without usable non-volatile storage.
	This section introduces what is needed by a board-config to persist a hardfault over warm-reset (only), on boards without usable non-volatile storage.

		]


		def format_coredump(p: ParsedHardfault, cpuid: int) -> str:

Uh oh!

Conversation

marinauterion commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Solved Problem

Solution

Changes:

Test:

Uh oh!

github-actions Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔎 FLASH Analysis

Uh oh!

github-actions Bot commented Jul 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexcekay commented Jul 2, 2026

Uh oh!

Uh oh!

Uh oh!

alexcekay Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marinauterion Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marinauterion Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marinauterion commented Jul 1, 2026 •

edited

Loading

github-actions Bot commented Jul 1, 2026 •

edited

Loading

alexcekay Jul 2, 2026 •

edited

Loading

marinauterion Jul 9, 2026 •

edited

Loading

marinauterion Jul 9, 2026 •

edited

Loading