DoS: Qube operations hang dom0 GUI

[GammaSQ]

Qubes OS release

4.3

Brief summary

When starting any operations from the tray icons that takes long enough to notice (such as attaching devices, shutting down a VM, ...) all qubes-specific windows hang. The qube-manager, other tray icons, but not dom0 as a whole. (e.g. terminal in dom0 still works)

Steps to reproduce

Have a slow computer?

Attaching devices from the tray icons takes long enough for me to notice. Shutting down a hanging VM and waiting for the timeout is annoying.

Expected behavior

Ideally, nothing hangs during operation. At the most, the "thing" doing the operations should hang (e.g. device-manager when waiting for device-attachment to finish)

Actual behavior

All of Qubes-GUI (I could find) hangs and don't accept interactions.

Additional information

Ways to freeze:

• qvm-run -u root QUBE -- gdb -p 1
• qvm-run -u root QUBE -- 'echo freeze > /sys/power/state'

Some workarounds used in Qubes' history:

• DoS: Qube operations hang dom0 GUI #10648 (comment)

---

PRs:

• Return appropriate exception when libvirt shutdown times out qubes-core-admin#807
• Add option to wait for domain shutdown qubes-core-admin-client#469
• Don't block main thread on qube hanging shutdown qubes-manager#450
• Schedule synchronous function to separate thread qubes-desktop-linux-manager#312
• Use admin utils wrappers for vm actions qubes-core-admin-linux#212
• Cache device information from hwdata qubes-app-linux-usb-proxy#56
• Cache device information from hwdata qubes-core-admin#812
• Cache libvirt domain state qubes-core-admin#819
• Avoid use of startwith in a large loop qubes-app-linux-usb-proxy#58

[marmarek]

While "qubesadmin" module is still synchronous, qubesd isn't. So, in theory there might be cases where some domains tray action freezes that one menu (it's still a bug!), but definitely it shouldn't freeze other applications. And even within a single application, we try to use threads to execute slower operations in a background - exactly to not freeze the menu. Can you give specific example of an operation that has this effect (you say attaching devices - which type of devices?)?

As for qubesd side, the only explanation I can think of is blocking on libvirt - which does have a global lock for some operations (for example a short time during starting a VM). But those blocking operations should be rather rare.

@ben-grande do you want to take this one?

[todo0123]

@marmarek, I also experience this bug.

For example: I want to connect a USB mic + USB webcam to my "Zoom" qube. I right-click on the corresponding icon in the tray and connect the first device. Then I need to wait for a message about the result (usually takes 10 sec).

This tray icon won't react to my second right-click until the connection operation is finished; it also will not put my click into a queue - so I need to click again and again, waiting for the tray menu to be ready to accept new commands.

P.S.: Unrelated, but I think it would be a good idea to separate drives and USB devices into separate tray menus.

[GammaSQ]

The most reproducible way seems to be attaching a block device to a VM and removing it while data is written to the block device. Not sure this is a 100% method, but I encounter this problem a lot while interacting with / running into problems with block devices.

Other ways to freeze a VM also work (I think running out out of memory?), but I've seen some in-VM freezes that don't cause this problem, e.g. freezing browser allowed shutdown without a problem.

[GammaSQ]

After triggering a shutdown of a hanging VM from the GUI and have the GUI freeze, trying to kill that VM via CLI (using qvm-kill) also hangs the terminal.

[ben-grande]

Related issues

High CPU usage may slow things:

• Expanding xfce notification area causes high cpu usage #7291
• Status Tray Plugin hangs and consumes 50% CPU indefinitely when Properties are opened and changed #9806

Qrexec DoS:

• Too many qrexec requests make the target domain hang - DoS #5343

Device issues:

• Qubesd takes a really long time to query and manage USB devices with multiple USB hubs #10569
• General device issues after installing 4.3 #10636

General hangs:

• Backup cancel freezes qube manager #6537

When starting any operations from the tray icons that takes long enough to notice (such as attaching devices, shutting down a VM, ...) all qubes-specific windows hang. The qube-manager, other tray icons, but not dom0 as a whole. (e.g. terminal in dom0 still works)

How did you trigger shutdown?

1. qui-domains -> -> Shutdown
2. qui-devices -> -> Detach and shut down disp1234

Have a slow computer?

Can you share more info about your device? Is your SSD in good condition?

I tried to reproduce by using slow qubes such as anon-whonix, shutting it down via qui-domain and opened VM Settings of a qube via Qube Manager and opened a terminal of qube via qui-domain. anon-whonix shutdown after these operations, so it didn't hold them.

When you say the window hang, what do you mean? You can make any changes on them? You can't open them? Are you considering qube manager being open?

What do you mean by "other tray icons"? Please be specific and detailed on each thing you noticed, so it makes it easier for me to reproduce.

Attaching devices from the tray icons takes long enough for me to notice. Shutting down a hanging VM and waiting for the timeout is annoying.

I am trying to reproduce this, please give detailed steps so I can follow along.

From other issues I saw you opened, you do have a desktop, you do have some devices attached to different qubes, you do suspend the system, you also use block devices. Can you please share if you are doing any i/o heavy activity in the meantime? A lot of devices being passthroughed?

Can you monitor htop?

Can you share the output on 2 different terminals of:

sudo journalctl -f
journalctl --user -f

Hide the output inside /details tag of github or upload as a file.

Can you make a video of your dom0? Would ease understanding. You can make a video using qubes-video-companion screenshare from a qube and targeting dom0. Might be a time investment, but then you can reuse this knowledge for all future issue sharing. Doesn't reduce the need to add details in text, for search queries and for accessibility, but helps to visually understand things.

Expected behavior

Ideally, nothing hangs during operation. At the most, the "thing" doing the operations should hang (e.g. device-manager when waiting for device-attachment to finish)

Actual behavior

All of Qubes-GUI (I could find) hangs and don't accept interactions.