Compile Kernel With KDB_UNATTENDED

[alasdairmuckart]

Please add options KDB_UNATTENDED to the kernel for OPNsense so production systems do not hang indefinitely on panic until someone can phsycially access the console.

In a default OPNsense build options DDB and options KDB are set, but options KDB_UNATTENDED is not:

user@opnsense:~ $ sysctl kern.conftxt | grep DB
options	DDB
options	NETGDB
options	KDB_TRACE
options	KDB
options	DDB_CTF

This means the kernel will drop to debugger in the event of a kernel panic and sit there until someone can access the console to reboot the machine.

This is not desirable in a production system, especially for remote branch office firewalls where it can be difficult and expensive to get an engineer on site to get the machine back online.

The Handbook says:

[...] any panic condition will branch to DDB if the kernel is configured to use it. For this reason, it is not wise to configure a kernel with DDB for a machine running unattended.

To obtain the unattended functionality, add:

options KDB_UNATTENDED

to the kernel configuration file and rebuild/reinstall.

This advice is repeated in the Glossary:

options KDB_UNATTENDED: change the default value of the debug.debugger_on_panic sysctl to 0, which controls whether the debugger is entered on panic. When options KDB is not compiled into the kernel, the behavior is to automatically reboot on panic; when it is compiled into the kernel, the default behavior is to drop into the debugger unless options KDB_UNATTENDED is compiled in. If you want to leave the kernel debugger compiled into the kernel but want the system to come back up unless you’re on-hand to use the debugger for diagnostics, use this option.

Thank you.

[fichtner]

It’s not an option because we lose the ability to gather panics for crashes. What’s your use case, panics before boot completes? And what’s the crash about?

[alasdairmuckart]

All this does is stop the firewall hanging at the debugger. It shouldn't stop the crash dump being written.

The handbook says (emphasis mine):

options KDB_TRACE: change the default value of the debug.trace_on_panic sysctl to 1, which controls whether the debugger automatically prints a stack trace on panic. Especially if running with options KDB_UNATTENDED, this can be helpful to gather basic debugging information on the serial or firewire console while still rebooting to recover.

The specific case I've run into is panics at boot doing upgrades which has taken remote offices offline for extended periods because the firewall sat hung at a debugger until we could get someone physically on site to reboot it from the console. IMO that's unacceptable behaviour for a production system.

I sent the most recent panic info to Deciso, after which OPNsense deleted it from the firewall so I didn't get much chance to analyse it myself.