Several qubes fail to start: Unknown PCI header type '127' for device (and other PCI-related errors)

[deeplow]

Qubes OS release

Qubes 4.2.4

Brief summary

When switching from an old fedora to a new one via SecureDrop's Workstation, qubes fail to start and the entire system needs to be restarted, for the issue to be resolved. The startup failure notification shows: Start failed: internal error: Unknown PCI header type '127' for device ... see /var/log/libvirt/libxl/libxl-driver.log for details.

Both sys-net and sys-usb are affected. But they seem to have different symptoms. Might be worth a separate report. This happens specifically when switching old Fedora templates to a more recent version through this salt state file. The file itself is a bit convoluted a definitely in need of a cleanup / simplification, but I have rendered it in a gist, so it's easier to understand what's going on.

This happens on various systems, quite reliably:

• ThinkPad T14 Gen1
• NovaCustom NV41 (on 2 separate devices)

Steps to reproduce

Note I haven't tried these exact reproduction steps yet, but the issue happens when running the particular sls file. It could be that there is some path dependence and it requires a full SecureDrop Workstation install, but I doubt it.

• Be on a clean Qubes 4.2.4 install
• Install SecureDrop Workstation packages (we only really this mostly self-contained sls file)
  • sudo qubes-dom0-update -y qubes-repo-contrib
  • sudo qubes-dom0-update -y --clean securedrop-workstation-keyring
  • sudo qubes-dom0-update -y --clean securedrop-workstation-dom0-config
• sudo qubesctl --show-output state.sls securedrop_salt.sd-sys-vms

Expected behavior

Completes successfully.

Actual behavior

Soon after Fedora 43 is downloaded and updated, the the system hangs when trying to start various qubes.

sys-firewall:

Specifically:

Start failed: internal error: Unknown PCI header type '127' for device ... see /var/log/libvirt/libxl/libxl-driver.log for details

The /var/log/libvirt/libxl/libxl-driver.log log has several errors about failing to remove devices, as well as an error from libxl_exec.c that a process died with exit status zero.

sys-net

Specifically:

Start failed: internal error: libxenlight failed to create new domain 'sys-net', see /var/log/libvirt/libxl/libxl-driver.log for details

sys-usb

Specifically:

Cannot connect to qrexec agent for 60 seconds, see /var/log/xen/console/guest-sys-usb.log for details

The /var/log/xen/console/guest-sys-usb.log file simply contains this line (for the time period of running sdw --apply):

Logfile Opened

Additional information

Credit to @apyrgio for filing the issue originally in freedomofpress/securedrop-workstation#1668. This issue contains additional debug info. It had also happened to me a few times, but always at an inconvenient time, where a system restart to continue the work was necessary.

Possibly related issues:

• dom0 Kernel update 6.12. 54 -> 59 breaks sys-net with RTL-8822CE #10463
• PCI wifi adapter unrecognized after dom0's kernel update to 6.12 #9689

Logs from Thinkpad T14 (Gen 1)

Dom0 kernel log: kern.log

Output on my system for the sys-net qube:

$ qvm-pci ls sys-net
BACKEND:DEVID  DESCRIPTION                                                                                      USED BY
<id1>                       Network controller: Intel Corporation Comet Lake PCH-LP CNVi Wifi       sys-net (no-strict-reset=True)
<id2>                       Ethernet controller: Intel Corporation Ethernet Connection (10) I219-V     sys-net (no-strict-reset=True)
$ cat /sys/bus/pci/devices/<id1>/reset_method
flr
$ cat /sys/bus/pci/devices/<id1>/power/control
auto
$ cat /sys/bus/pci/devices/<id2>/reset_method
<does not exist>
$ cat /sys/bus/pci/devices/<id2>/power/control
on

sys-usb qube:

$ qvm-pci ls sys-usb
BACKEND:DEVID  DESCRIPTION                                                                                                    USED BY
<id1>                       USB controller: Intel Corporation Comet Lake PCH-LP USB 3.1 xHCI Host Controller                     sys-usb (no-strict-reset=True)
<id2>                       USB controller: Intel Corporation JHL6240-Thunderbolt 3 USB 3.1 Controller (Low Power [Alpine Ridge LP 2016]   sys-usb (no-strict-reset=True)
$ cat /sys/bus/pci/devices/<id1>/reset_method
<does not exist>
$ cat /sys/bus/pci/devices/<id1>/power/control
auto
$ cat /sys/bus/pci/devices/<id2>/reset_method
pm bus
$ cat /sys/bus/pci/devices/<id2>/power/control
on

Originally posted by @apyrgio in #1668

Output on my system for the sys-net qube:

$ qvm-pci ls sys-net
BACKEND:DEVID  DESCRIPTION                                                                                      USED BY
<id1>                       Network controller: Intel Corporation Comet Lake PCH-LP CNVi Wifi       sys-net (no-strict-reset=True)
<id2>                       Ethernet controller: Intel Corporation Ethernet Connection (10) I219-V     sys-net (no-strict-reset=True)
$ cat /sys/bus/pci/devices/<id1>/reset_method
flr
$ cat /sys/bus/pci/devices/<id1>/power/control
auto
$ cat /sys/bus/pci/devices/<id2>/reset_method
<does not exist>
$ cat /sys/bus/pci/devices/<id2>/power/control
on

sys-usb qube:

$ qvm-pci ls sys-usb
BACKEND:DEVID  DESCRIPTION                                                                                                    USED BY
<id1>                       USB controller: Intel Corporation Comet Lake PCH-LP USB 3.1 xHCI Host Controller                     sys-usb (no-strict-reset=True)
<id2>                       USB controller: Intel Corporation JHL6240-Thunderbolt 3 USB 3.1 Controller (Low Power [Alpine Ridge LP 2016]   sys-usb (no-strict-reset=True)
$ cat /sys/bus/pci/devices/<id1>/reset_method
<does not exist>
$ cat /sys/bus/pci/devices/<id1>/power/control
auto
$ cat /sys/bus/pci/devices/<id2>/reset_method
pm bus
$ cat /sys/bus/pci/devices/<id2>/power/control
on

Originally posted by @apyrgio in #1668

[marmarek]

When switching from an old fedora to a new one

It is very unlikely the fedora version matters here. The Unknown PCI header type '127' means the device got into a broken state where it basically stopped responding to any requests from the kernel (in this case - reading config space, which results in 0xff responses).
I'm not sure exactly what could cause such behavior, it's likely device specific. But looking at the salt formula it is likely that killing sys-net/sys-usb (instead of clean shutdown) could lead to such effect, especially if some devices don't advertise reset method (which is the case here).

[deeplow]

Thanks. That makes sense. I'll play around with nicely shutting them down and seeing if we run into this.

[andrewdavidwong]

• NovaCustom NV41 (on 2 separate devices)

Are they Qubes-certified models? If so, I'll add the certfied hardware label and bump up the priority. However, note that only certain configurations are Qubes-certified. See: https://doc.qubes-os.org/en/latest/user/hardware/certified-hardware/novacustom-nv41-series.html

[deeplow]

Yes, both devices are certified

But @marmarek was spot on. Form some limited testing, this was in fact a consequence qubes being killed instead of properly shut down. I'll therefore close this issue and track it only on SecureDrop.