Add Xen command line parsing

[ArrayBolt3]

Enable GRUB to parse the Xen command line for parameters, and expose those parameters to the GRUB config file (or rescue shell) as environment variables.

The way this works is pretty simple:

• When booting in PVH mode, grab the command line passed by Xen and copy it into the start_info struct pointed to by grub_xen_start_page_addr. (In PV mode, this struct has the command line pre-populated (by the hypervisor?), in PVH mode it has to be manually populated.)
• Before parsing the GRUB configuration (or dropping to a rescue shell), parse the command line:
  • Break apart the command line into words, using spaces as the separator. Quotes (both single and double) can be used to suppress splitting, and backslashes can be used to escape characters.
  • For each word:
    • If the word contains an =, split the word on the first = sign. Export an environment variable with the name set to the word portion on left of the sign, and the value set to the word portion to the right of the sign.
    • Otherwise, export an environment variable with the name set to the word, and the value set to 1.

Submitting this for preliminary review, I still have some doubts and concerns about how this is done, namely:

• I'm assuming that the command line pointed to by pvh_start_info->cmdline_paddr will not be any larger than grub_xen_start_page_addr->cmd_line variable. The latter is hard-coded to 1024 bytes, but the size of the string pointed to by the former doesn't appear to be defined. I should dig into Xen and make sure the command line will never exceed 1024 bytes (including the terminating \0), otherwise this could lead to some "fun" buffer overflows.
• pvh_start_info->cmdline_paddr is a physical address (because of course it is, the hypervisor isn't about to mess with the page tables of the guest). The grub_strcpy call I do to copy its contents into grub_xen_start_page_addr->cmd_line therefore assumes paging is disabled (or the page(s) containing the command line are identity-mapped) at the time of copying. It also assumes that paging will still be disabled (or that the page(s) containing grub_xen_start_page_addr->cmd_line will be identity-mapped) by the time it's used during config parsing, since at best we're copying from a physical address to another physical address here. I tested this in a PVH VM and it seems to work without issues, but seeming to work and actually working are two very different things. I haven't done a deep-dive of when paging is and isn't used in GRUB yet, but it looks to me like some parts use it and some parts don't, so this is playing with fire.
• Every single parameter passed from Xen to GRUB will end up an environment variable used when parsing the config file. This means that if in the future we ever want to pass parameters to GRUB on the Xen command line that do something other than set environment variables, the API will have to be broken. That might not be good. Changing this is pretty trivial given the way that I parse the command line.
• No sanitization of variable names or values is done, arbitrary binary garbage (other than NUL of course) can go in both environment variable names and values. That's probably bad. How exactly should sanitization work though, do we just reject anything other than 7-bit ASCII, or are there scenarios where Unicode might legitimately be used here?
• This hasn't been tested super thoroughly yet.

[marmarek]

• Before parsing the GRUB configuration (or dropping to a rescue shell), parse the command line:

Is it really a good idea? Grub config (script) language is quite extensive, so if one needs it, they can do it themselves. On the other hand, forced parsing has many risks, including:

• finding "variables" that has name not being correct grub variable name (for example with dots - pretty common on kernel cmdline)
• loosing order of options
• loosing quotes (there may be cases when something interprets quoted and unquoted value differently)
• possibly some more

Better to simply set some variable with the whole cmdline as is.

• I'm assuming that the command line pointed to by pvh_start_info->cmdline_paddr will not be any larger than grub_xen_start_page_addr->cmd_line variable.

Theoretically, libxenguest (that is used for constructing start_info) uses the same MAX_GUEST_CMDLINE. But in any case, do not assume it will always fit. If for any reason it's longer (or for some reason simply terminating NUL is missing), it should not crash. And BTW it should be verified even if some spec explicitly spells the limit, unchecked strcpy is usually a bug. Either truncate, or ignore completely too long values (the latter may be safer to do).

[ArrayBolt3]

Is it really a good idea? Grub config (script) language is quite extensive, so if one needs it, they can do it themselves.

I don't particularly like this idea for several reasons:

• GRUB's configuration language doesn't support Bash-style arrays, without which it is impossible to properly handle word splitting when dealing with values that contain spaces (without resorting to extremely tedious and arcane measures that also only allow a static number of words to be handled).
• There also aren't any fancy variable expansion features for string parsing, so doing things like splitting abc=def into abc and def is non-trivial.
• GRUB configuration files are generally autogenerated under Debian (and therefore under Kicksecure), so while it's easy to say "just use the config file for this", in practice that is a very difficult task, worse than just scripting it manually. All of the logic needed for parsing would have to be embedded into a script under /etc/grub.d.
• This also assumes that the only use for the entire Xen command line is only going to be used in a grub.cfg context. What if one day this feature is used to configure something deep inside the guts of GRUB itself, unrelated to config file handling? For instance maybe in the future the Xen command line can be used to enable or disable filesystem drivers in order to reduce attack surface.

finding "variables" that has name not being correct grub variable name (for example with dots - pretty common on kernel cmdline)

This can be solved with string sanitization. Dots are fine in a variable value, but shouldn't be allowed in the name.

loosing order of options

I'm not sure this is too horrible of a limitation. If someone needs to pass in ordered options, they can do so using something like ordered_options="abc def ghi", and then the grub.cfg file can handle getting the individual options out. It'll be no more painful than if it had to parse the whole command line on its own, if this is ever needed.

loosing quotes (there may be cases when something interprets quoted and unquoted value differently)

Backslash escaping is supported, so quotes can be embedded into variable values very easily.

Theoretically, libxenguest (that is used for constructing start_info) uses the same MAX_GUEST_CMDLINE. But in any case, do not assume it will always fit. If for any reason it's longer (or for some reason simply terminating NUL is missing), it should not crash. And BTW it should be verified even if some spec explicitly spells the limit, unchecked strcpy is usually a bug. Either truncate, or ignore completely too long values (the latter may be safer to do).

Got it, that makes perfect sense. I'll implement the latter.

[marmarek]

finding "variables" that has name not being correct grub variable name (for example with dots - pretty common on kernel cmdline)

This can be solved with string sanitization. Dots are fine in a variable value, but shouldn't be allowed in the name.

Think of systemd.unit=sysmaint-boot.target...

Anyway, IIUC your idea is to set boot mode extra args not by appending them to the grub cmdline directly, but as some variable like extra_kernelopts="actual options here", and use that variable in the grub config - correct? The need for extra quoting (if extra kernelopts contain quotes itself) will be inconvenient, but should be doable.

• What if one day this feature is used to configure something deep inside the guts of GRUB itself, unrelated to config file handling?

This is actually one of my concerns. Depending on how this feature is going to be used, it's easy to break something if you conflict with some crucial grub variable (like prefix). But if it's not going to be mixing Linux parameters with grub variables, maybe that's less of a concern.

And one more thing on the process side: I'm okay with merging this patch here, but please do attempt to sent it upstream (before it gets merged here). I don't want to maintain yet another downstream patch indefinitely unless absolutely necessary...

[ArrayBolt3]

Think of systemd.unit=sysmaint-boot.target...

That whole thing will go in a value though. To be clear, what I'm envisioning is that libvirt will tell Xen to pass a command line that looks like boot_mode_opts="systemd.unit=sysmaint-boot.target boot-role=sysmaint". GRUB will then parse that into an environment variable $boot_mode_opts with the value systemd.unit=sysmaint-boot.target boot-role=sysmaint, and then grub.cfg would just append $boot_mode_opts to the kernel command line.

Anyway, IIUC your idea is to set boot mode extra args not by appending them to the grub cmdline directly, but as some variable like extra_kernelopts="actual options here", and use that variable in the grub config - correct?

Yes, that's currently what I'm planning on.

This is actually one of my concerns. Depending on how this feature is going to be used, it's easy to break something if you conflict with some crucial grub variable (like prefix). But if it's not going to be mixing Linux parameters with grub variables, maybe that's less of a concern.

Right, the two won't mix.

And one more thing on the process side: I'm okay with merging this patch here, but please do attempt to sent it upstream (before it gets merged here).

Sure, that makes sense.

On the topic of memory paging, I sent an email to grub-devel asking what is and isn't safe when it comes to dereferencing physical addresses, since I couldn't quite make heads or tails of the assembler code that turns paging on and off: https://lists.gnu.org/archive/html/grub-devel/2025-04/msg00000.html Everything other than that, I'm working on and should hopefully have pushed before I'm done with work tonight.

[ArrayBolt3]

Latest changes fix most issues, however for some reason I can't get PV mode to see the command line anymore and I have no idea why. PVH seems to be working very smoothly at this point though.

[ArrayBolt3]

PV issues seem to have been a case of VM corruption - I don't know what I did but my VM I was using for testing was somehow loading the wrong bootloader. Ended up throwing away the VM and making a new one, issue seems to be resolved now.

[ArrayBolt3]

Did thorough manual testing using the following method:

• Create a testing TemplateVM, and install grub-pc in it.
• Ensure a good, functional grub.cfg file exists for the VM, then move that config file to /boot/grub/grub.cfg.bak.
• Shut down the VM.
• Set the testing VM's kernel to either pvgrub2-pvh or pvgrub2 as appropriate for each test.
• Execute all of the tests in the table below. Each test consists of the following steps:
  • Manually modify /usr/share/qubes/templates/libvirt/xen.xml to set the exact command line to pass to the VM for each test. Set the kernel command line passed by dom0 to the value specified in the test's Input command line column.
  • Boot the VM and immediately launch a debug console for it so you can interact with GRUB.
  • Run the commands listed in the test's Test commands column.
  • Ensure the output of each command matches the expected output listed in the test's Expected output column.
  • Run configfile (xen/xvda,gpt3)/boot/grub/grub.cfg.bak to allow boot to finish.
  • Ensure the boot succeeds without crashing.
  • Log into the VM as root.
  • Run shutdown now.

Here is the test table I used. [x] indicates a test passed, [?] means I was unable to run the test for some reason.

Input command line	Test commands	Expected output	PVH	PV
boot_mode_opts="test1 test2"	echo $boot_mode_opts	test1 test2	[x]	[x]
Boot_Mode_Opts="test1 test2"	echo $Boot_Mode_Opts	test1 test2	[x]	[x]
boot_mode_opts='test1 test2'	echo $boot_mode_opts	test1 test2	[x]	[x]
boot_mode_opts=test1\ test2	echo $boot_mode_opts	test1 test2	[x]	[x]
boot_mode_opts="test1\\ test2"	echo $boot_mode_opts	test1\ test2	[x]	[x]
boot_mode_opts="test1' test2"	echo $boot_mode_opts	test1' test2	[x]	[x]
boot_mode_opts='test1" test2'	echo $boot_mode_opts	test1" test2	[x]	[x]
boot_mode_opts='test1' test'	echo $boot_mode_opts, echo $test	test1, 1	[x]	[x]
boot_mode_opts="test1" test'	echo $boot_mode_opts, echo $test	test1, 1	[x]	[x]
boot_mode_opts="test1" test"	echo $boot_mode_opts, echo $test	test1, 1	[x]	[x]
boot_mode_opts=<1,008 digits here>	echo $boot_mode_opts	<1,008 digits here>	[x]	[x]
boot_mode_opts=<1,009 digits here>	echo $boot_mode_opts	on PVH, <1,008 digits here> on PV	[x]	[x]
a=<10,000 digits here>	echo $a	on PVH, <1,021 digits here> on PV	[x]	[x]
arg_one='arg1' arg_two='arg2'	echo $arg_one, echo $arg_two	arg1, arg2	[x]	[x]
arg1='arg_one' arg2='arg_two'	echo $arg1, echo $arg2	,	[x]	[x]
boot_mode_opts="\x01abc"	echo $boot_mode_opts		[?]	[?]
boot_mode_\x01opts="abc"	echo $boot_mode_opts		[?]	[?]
do_thing	echo $do_thing	1	[x]	[x]
do_\x01thing	echo $do_thing		[?]	[?]
do_\x01thing do_other	echo $do_thing, echo $do_other	, 1	[?]	[?]
""		Boot without crash	[x]	[x]
""=""		Boot without crash	[x]	[x]
=		Boot without crash	[x]	[x]
\		Boot without crash	[x]	[x]
var=\""a"\'	echo $var	"a'	[x]	[x]
var=\''a'\"	echo $var	'a"	[x]	[x]
var_a=\'"a"\" var_b=\"'b'\'	echo $var_a, echo $var_b	'a", "b'	[x]	[x]

Notes:

• The tests involving an \x01 byte could not be completed, as embedding that byte into the kernel command line using an XML entity resulted in Qubes refusing to even try to boot the VM.
• For the tests involving very large numbers of digits in the command line, no counting was actually done, only the presence, absence, or truncation of the value in the command line was verified. I wasn't about to count to 1,021 for obvious reasons.
• All of the above tests have been conducted on Qubes R4.3 using either the code present in the PR, or an extremely close variant that contained a couple extra debugging lines that have no effect on the results of the test. In case someone wonders, this is the only way in which the code used for some of these tests differed from the PR:

--- working/qubes-linux-pvgrub2/0300-Experimental-Xen-hacking-with-GRUB-parameters.patch        2025-04-02 22:26:59.431368520 -0500
+++ linux-pvgrub2/0300-Experimental-Xen-hacking-with-GRUB-parameters.patch      2025-04-02 21:33:46.581643907 -0500
@@ -46,7 +46,7 @@
  
  grub_err_t
 diff --git a/grub-core/kern/main.c b/grub-core/kern/main.c
-index 8e89763..c078075 100644
+index 8e89763..bdbb820 100644
 --- a/grub-core/kern/main.c
 +++ b/grub-core/kern/main.c
 @@ -35,6 +35,10 @@
@@ -60,7 +60,7 @@
  grub_addr_t
  grub_modules_get_end (void)
  {
-@@ -362,6 +366,14 @@ grub_main (void)
+@@ -362,6 +366,16 @@ grub_main (void)
    grub_env_export ("root");
    grub_env_export ("prefix");
  
@@ -71,6 +71,8 @@
 +#if defined (GRUB_MACHINE_XEN) || defined (GRUB_MACHINE_XEN_PVH)
 +  grub_parse_xen_cmdline ();
 +#endif
++  grub_env_set ("test_marker", "marked");
++  grub_env_export ("test_marker");
 +
    /* Reclaim space used for modules.  */
    reclaim_module_space ();

• This has only been tested on Qubes, it has not yet been tested in a more "vanilla" Xen environment, nor has it been submitted to GRUB upstream.
• all by itself does not mean I input a space, or expected a space as output, it means that no input was given or no output was expected.

[ArrayBolt3]

Made some changes in preparation to repost to the grub-devel mailing list (my initial post was entirely ignored). The memory management is now a bit better (I hope, still need to find a robust way to test invalid characters in the Xen cmdline but in theory it should be better), but the main change I made is that Xen parameters are exposed as GRUB environment variables only if they start with xen_grub_env_. This prevents Xen from simply clobbering whatever environment variables it wants, and also makes it easy for a GRUB configuration file to choose to entirely ignore Xen's input if it so chooses. This should make mistakes a bit harder and hopefully improve the patch's upstreamability.

[ArrayBolt3]

Got a very slightly modified variant of the patch to build on vanilla GRUB on Arch Linux, and tested it very thoroughly. (The only modification is the grub_exit call, which should be unreachable code, no longer takes an argument in upstream GRUB.) Each test was done by booting the VM in the appropriate mode and with the appropriate bootloader, passing in the specified input command line, then running the specified test commands. If the expected output was seen, commands were then run to finish booting the VM into an Arch Linux live environment. [x] means a test passed, and that's really the only thing I need to say because all the tests passed :D Testing results:

Input command line	Test commands	Expected output	PVH	PV
xen_grub_env_boot_mode_opts="test1 test2"	echo $xen_grub_env_boot_mode_opts	test1 test2	[x]	[x]
xen_grub_env_Boot_Mode_Opts="test1 test2"	echo $xen_grub_env_Boot_Mode_Opts	test1 test2	[x]	[x]
xen_grub_env_boot_mode_opts='test1 test2'	echo $xen_grub_env_boot_mode_opts	test1 test2	[x]	[x]
xen_grub_env_boot_mode_opts=test1\ test2	echo $xen_grub_env_boot_mode_opts	test1 test2	[x]	[x]
xen_grub_env_boot_mode_opts="test1\\ test2"	echo $xen_grub_env_boot_mode_opts	test1\ test2	[x]	[x]
xen_grub_env_boot_mode_opts="test1' test2"	echo $xen_grub_env_boot_mode_opts	test1' test2	[x]	[x]
xen_grub_env_boot_mode_opts='test1" test2'	echo $xen_grub_env_boot_mode_opts	test1" test2	[x]	[x]
xen_grub_env_boot_mode_opts='test1' test'	echo $xen_grub_env_boot_mode_opts, echo $test	test1,	[x]	[x]
xen_grub_env_boot_mode_opts="test1" test'	echo $xen_grub_env_boot_mode_opts, echo $test	test1,	[x]	[x]
xen_grub_env_boot_mode_opts="test1" test"	echo $xen_grub_env_boot_mode_opts, echo $test	test1,	[x]	[x]
xen_grub_env_boot_mode_opts='test1' xen_grub_env_test'	echo $xen_grub_env_boot_mode_opts, echo $xen_grub_env_test	test1, 1	[x]	[x]
xen_grub_env_boot_mode_opts="test1" xen_grub_env_test'	echo $xen_grub_env_boot_mode_opts, echo $xen_grub_env_test	test1, 1	[x]	[x]
xen_grub_env_boot_mode_opts="test1" xen_grub_env_test"	echo $xen_grub_env_boot_mode_opts, echo $xen_grub_env_test	test1, 1	[x]	[x]
xen_grub_env_boot_mode_opts=<995 digits here>	echo $xen_grub_env_boot_mode_opts	<995 digits here>	[x]	[x]
xen_grub_env_boot_mode_opts=<996 digits here>	echo $xen_grub_env_boot_mode_opts	on PVH, <995 digits here> on PV	[x]	[x]
xen_grub_env_a=<10,000 digits here>	echo $xen_grub_env_a	on PVH, <1008 digits here> on PV	[x]	[x]
xen_grub_env_arg_one='arg1' xen_grub_env_arg_two='arg2'	echo $xen_grub_env_arg_one, echo $xen_grub_env_arg_two	arg1, arg2	[x]	[x]
xen_grub_env_arg1='arg_one' xen_grub_env_arg2='arg_two'	echo $xen_grub_env_arg1, echo $xen_grub_env_arg2	,	[x]	[x]
xen_grub_env_boot_mode_opts="^Aabc"	echo $xen_grub_env_boot_mode_opts		[x]	[x]
xen_grub_env_boot_mode_^Aopts="abc"	echo $xen_grub_env_boot_mode_opts		[x]	[x]
xen_grub_env_do_thing	echo $xen_grub_env_do_thing	1	[x]	[x]
xen_grub_env_do_^Athing	echo $xen_grub_env_do_thing		[x]	[x]
xen_grub_env_do_^Athing xen_grub_env_do_other	echo $xen_grub_env_do_thing, echo $xen_grub_env_do_other	,	[x]	[x]
""		Boot without crash	[x]	[x]
""=""		Boot without crash	[x]	[x]
=		Boot without crash	[x]	[x]
\		Boot without crash	[x]	[x]
xen_grub_env_var=\""a"\'	echo $xen_grub_env_var	"a'	[x]	[x]
xen_grub_env_var=\''a'\"	echo $xen_grub_env_var	'a"	[x]	[x]
xen_grub_env_var_a=\'"a"\" xen_grub_env_var_b=\"'b'\'	echo $xen_grub_env_var_a, echo $xen_grub_env_var_b	'a", "b'	[x]	[x]

[ArrayBolt3]

Current attempt at upstreaming the patch: https://lists.gnu.org/archive/html/grub-devel/2025-04/msg00247.html

[ArrayBolt3]

@marmarek It's been about a month since this was submitted and I have still heard nothing from GRUB upstream. Should I resubmit the patch as it is, or is there something wrong with it?

(Pinging on Github since my Matrix server is near-unusable at the moment due to some infra issues on Canonical's side.)

[marmarek]

Ping on the ML maybe? But also, update this PR in the meantime.

[ArrayBolt3]

I believe I did already update the PR to be the same as the upstream patch, minus small bits needed for compatibility with different GRUB versions. I'll double-check though.

[qubesos-bot]

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025091516-4.3&flavor=pull-requests

Test run included the following:

• Add device info to qrexec call argument qubes-app-linux-input-proxy#40 (QubesOS/qubes-app-linux-input-proxy@e9f8bbf)
• Add Arch packaging qubes-core-admin-client#332 (QubesOS/qubes-core-admin-client@40487bc)
• Skip waiting for session when GUI is disabled qubes-core-admin-client#382 (QubesOS/qubes-core-admin-client@cc9c5f5)
• Redo netvm setup after unpause qubes-core-admin#722 (QubesOS/qubes-core-admin@e0d6609)
• Wait for memory by transfer speed qubes-core-admin#730 (QubesOS/qubes-core-admin@c13626c)
• Prevent backing up dom0 it itself (in home dir) qubes-core-admin#731 (QubesOS/qubes-core-admin@273b5a9)
• tests: rebuild test-pkg rpm qubes-core-admin#732 (QubesOS/qubes-core-admin@f84662d)
• Introduce a service to wait for network uplink qubes-core-agent-linux#603 (QubesOS/qubes-core-agent-linux@78e8b6c)
• minimal-vms: disable GVFS usage for GIO clients qubes-core-agent-linux#606 (QubesOS/qubes-core-agent-linux@cb7cb88)
• Create filesystem on xvda1 on update qubes-core-agent-linux#607 (QubesOS/qubes-core-agent-linux@d4551fc)
• Disable wayland session by default qubes-desktop-linux-common#70 (QubesOS/qubes-desktop-linux-common@2d143cb)
• Show "List USB Devices" option if sys-usb exists qubes-desktop-linux-manager#275 (QubesOS/qubes-desktop-linux-manager@541718f)
• Optimize SVG icons qubes-desktop-linux-menu#57 (QubesOS/qubes-desktop-linux-menu@aa4fdf6)
• Resolve merge conflict for #61 qubes-desktop-linux-menu#65 (QubesOS/qubes-desktop-linux-menu@4f9a86a)
• Avoid using lspci in qubes-gui-agent-pre.sh qubes-gui-agent-linux#245 (QubesOS/qubes-gui-agent-linux@722e32c)
• Add Xen command line parsing #16 (7f31aef)
• qube_manager: support for RemoteVM qubes without crashing qubes-manager#398 (QubesOS/qubes-manager@679eff4)
• Select "sys-usb" as the default backup location qubes-manager#428 (QubesOS/qubes-manager@36e7630)
• Do not install dummy-psu-module for sys-gui template qubes-mgmt-salt-dom0-virtual-machines#83 (QubesOS/qubes-mgmt-salt-dom0-virtual-machines@5733b4a)

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025081011-4.3&flavor=update

• system_tests_network

  • VmNetworking_debian-13-xfce: test_001_simple_networking_paused (failure)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 2 != 0

• system_tests_backup

  • TC_00_Backup: test_100_backup_dom0_no_restore (error)
    TypeError: argument should be a str or an os.PathLike object where ...

  • TC_00_Backup: test_101_backup_dom0_to_homedir (error)
    TypeError: argument should be a str or an os.PathLike object where ...

• system_tests_network_ipv6

  • VmIPv6Networking_debian-13-xfce: test_501_ipv6_simple_networking_paused (error)
    subprocess.CalledProcessError: Command 'qubes.WaitForNetworkUplink'...

• system_tests_audio

  • TC_20_AudioVM_PipeWire_debian-13-xfce: test_260_audio_mic_enabled_switch_audiovm (failure)
    AssertionError: 0 not greater than 0 : Failed to move-source-output

• system_tests_qwt_win10_seamless@hw13

  • windows_install: Failed (test died)
    # Test died: command './install.sh' failed at /usr/lib/os-autoinst/...

• system_tests_qwt_win11@hw13

  • windows_install: wait_serial (wait serial expected)
    # wait_serial expected: qr/qDqV_-\d+-/...

  • windows_install: Failed (test died + timed out)
    # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

• system_tests_qwt_win10@hw13

  • windows_clipboard_and_filecopy: unnamed test (unknown)
  • windows_clipboard_and_filecopy: Failed (test died)
    # Test died: no candidate needle with tag(s) 'windows-Explorer-new-...

Failed tests

11 failures

• system_tests_network

  • VmNetworking_debian-13-xfce: test_001_simple_networking_paused (failure)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 2 != 0

• system_tests_backup

  • TC_00_Backup: test_100_backup_dom0_no_restore (error)
    TypeError: argument should be a str or an os.PathLike object where ...

  • TC_00_Backup: test_101_backup_dom0_to_homedir (error)
    TypeError: argument should be a str or an os.PathLike object where ...

• system_tests_extra

  • TC_00_QVCTest_whonix-workstation-17: test_010_screenshare (failure)
    AssertionError: 1 != 0 : Timeout waiting for /dev/video0 in test-in...

• system_tests_network_ipv6

  • VmIPv6Networking_debian-13-xfce: test_501_ipv6_simple_networking_paused (error)
    subprocess.CalledProcessError: Command 'qubes.WaitForNetworkUplink'...

• system_tests_audio

  • TC_20_AudioVM_PipeWire_debian-13-xfce: test_260_audio_mic_enabled_switch_audiovm (failure)
    AssertionError: 0 not greater than 0 : Failed to move-source-output

• system_tests_qwt_win10_seamless@hw13

  • windows_install: Failed (test died)
    # Test died: command './install.sh' failed at /usr/lib/os-autoinst/...

• system_tests_qwt_win11@hw13

  • windows_install: wait_serial (wait serial expected)
    # wait_serial expected: qr/qDqV_-\d+-/...

  • windows_install: Failed (test died + timed out)
    # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

• system_tests_qwt_win10@hw13

  • windows_clipboard_and_filecopy: unnamed test (unknown)
  • windows_clipboard_and_filecopy: Failed (test died)
    # Test died: no candidate needle with tag(s) 'windows-Explorer-new-...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/149225#dependencies

74 fixed

• system_tests_dispvm

  • TC_20_DispVM_debian-13-xfce: test_012_preload_low_mem (failure)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 1 != 0

  • TC_20_DispVM_debian-13-xfce: test_013_preload_gui (error)
    raise KeyError(key)... KeyError: 'disp3723'

  • TC_20_DispVM_debian-13-xfce: test_014_preload_nogui (error + cleanup)
    raise TimeoutError from exc_val... TimeoutError

  • TC_20_DispVM_debian-13-xfce: test_015_preload_race_more (error + cleanup)
    raise KeyError(key)... KeyError: 'disp1187'

  • TC_20_DispVM_debian-13-xfce: test_016_preload_race_less (failure + cleanup)
    ^^^^^^^^^^^^^^^^^^^^^^... AssertionError

  • TC_20_DispVM_debian-13-xfce: test_017_preload_autostart (error)
    raise KeyError(key)... KeyError: 'disp7317'

  • TC_20_DispVM_debian-13-xfce: test_018_preload_global (error)
    raise KeyError(key)... KeyError: 'disp8572'

  • TC_20_DispVM_debian-13-xfce: test_019_preload_refresh (error)
    raise KeyError(key)... KeyError: 'disp6425'

  • TC_20_DispVM_fedora-42-xfce: test_012_preload_low_mem (failure)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 1 != 0

  • TC_20_DispVM_whonix-workstation-17: test_012_preload_low_mem (failure)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 1 != 0

• system_tests_kde_gui_interactive

  • gui_keyboard_layout: wait_serial (wait serial expected)
    # wait_serial expected: "echo -e '[Layout]\nLayoutList=us,de' | sud...

  • gui_keyboard_layout: Failed (test died)
    # Test died: command 'test "$(cd ~user;ls e1*)" = "$(qvm-run -p wor...

• system_tests_audio

  • TC_20_AudioVM_Pulse_whonix-workstation-17: test_223_audio_play_hvm (error)
    qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • TC_20_AudioVM_Pulse_whonix-workstation-17: test_224_audio_rec_muted_hvm (error)
    qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • TC_20_AudioVM_Pulse_whonix-workstation-17: test_225_audio_rec_unmuted_hvm (error)
    qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • TC_20_AudioVM_Pulse_whonix-workstation-17: test_252_audio_playback_audiovm_switch_hvm (error)
    qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

• system_tests_audio@hw1

  • TC_20_AudioVM_Pulse_whonix-workstation-17: test_223_audio_play_hvm (error)
    qubes.exc.QubesVMError: Cannot connect to qrexec agent for 60 secon...

  • TC_20_AudioVM_Pulse_whonix-workstation-17: test_224_audio_rec_muted_hvm (error)
    qubes.exc.QubesVMError: Cannot connect to qrexec agent for 60 secon...

  • TC_20_AudioVM_Pulse_whonix-workstation-17: test_252_audio_playback_audiovm_switch_hvm (error)
    qubes.exc.QubesVMError: Cannot connect to qrexec agent for 60 secon...

• system_tests_dispvm_perf@hw7

  • TC_00_DispVMPerf_debian-13-xfce: test_000_dispvm (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_002_dispvm_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_006_dispvm_from_dom0 (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_008_dispvm_from_dom0_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_020_dispvm_preload (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_021_dispvm_preload_gui (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_022_dispvm_preload_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_023_dispvm_preload_gui_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_026_dispvm_from_dom0_preload (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_027_dispvm_from_dom0_preload_gui (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_028_dispvm_from_dom0_preload_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_029_dispvm_from_dom0_preload_gui_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_400_dispvm_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_402_dispvm_concurrent_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_404_dispvm_preload_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_404_dispvm_preload_less_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_404_dispvm_preload_more_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_405_dispvm_preload_gui_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_406_dispvm_preload_concurrent_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_407_dispvm_preload_gui_concurrent_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_900_vm (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_902_vm_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_904_vm_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_debian-13-xfce: test_906_vm_concurrent_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_020_dispvm_preload (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_021_dispvm_preload_gui (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_022_dispvm_preload_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_023_dispvm_preload_gui_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_026_dispvm_from_dom0_preload (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_027_dispvm_from_dom0_preload_gui (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_028_dispvm_from_dom0_preload_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_029_dispvm_from_dom0_preload_gui_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_404_dispvm_preload_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_404_dispvm_preload_less_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_404_dispvm_preload_more_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_405_dispvm_preload_gui_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_406_dispvm_preload_concurrent_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_fedora-42-xfce: test_407_dispvm_preload_gui_concurrent_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_020_dispvm_preload (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_021_dispvm_preload_gui (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_022_dispvm_preload_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_023_dispvm_preload_gui_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_026_dispvm_from_dom0_preload (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_027_dispvm_from_dom0_preload_gui (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_028_dispvm_from_dom0_preload_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_029_dispvm_from_dom0_preload_gui_concurrent (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_404_dispvm_preload_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_404_dispvm_preload_less_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_404_dispvm_preload_more_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_405_dispvm_preload_gui_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_406_dispvm_preload_concurrent_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

  • TC_00_DispVMPerf_whonix-workstation-17: test_407_dispvm_preload_gui_concurrent_api (failure)
    AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

• system_tests_guivm_gpu_gui_interactive@hw13

  • guivm_startup: wait_serial (wait serial expected)
    # wait_serial expected: qr/lEcbc-\d+-/...

  • guivm_startup: Failed (test died + timed out)
    # Test died: command '! qvm-check sys-whonix || time qvm-start sys-...

• system_tests_basic_vm_qrexec_gui_ext4

  • TC_20_NonAudio_whonix-gateway-17-pool: test_012_qubes_desktop_run (error + cleanup)
    raise TimeoutError from exc_val... TimeoutError

Unstable tests

Details

Performance Tests

Performance degradation:

15 performance degradations

• dom0_root_seq1m_q8t1_write 3:write_bandwidth_kb: 166306.00 :small_red_triangle: ( previous job: 265260.00, degradation: 62.70%)
• dom0_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 14900.00 :small_red_triangle: ( previous job: 23940.00, degradation: 62.24%)
• dom0_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 1767.00 :small_red_triangle: ( previous job: 2446.00, degradation: 72.24%)
• dom0_varlibqubes_seq1m_q8t1_read 3:read_bandwidth_kb: 168697.00 :small_red_triangle: ( previous job: 292489.00, degradation: 57.68%)
• dom0_varlibqubes_seq1m_q8t1_write 3:write_bandwidth_kb: 85392.00 :small_red_triangle: ( previous job: 110817.00, degradation: 77.06%)
• dom0_varlibqubes_seq1m_q1t1_read 3:read_bandwidth_kb: 114513.00 :small_red_triangle: ( previous job: 137802.00, degradation: 83.10%)
• fedora-42-xfce_root_seq1m_q1t1_write 3:write_bandwidth_kb: 27036.00 :small_red_triangle: ( previous job: 47575.00, degradation: 56.83%)
• fedora-42-xfce_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 2049.00 :small_red_triangle: ( previous job: 3020.00, degradation: 67.85%)
• fedora-42-xfce_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 1020.00 :small_red_triangle: ( previous job: 1368.00, degradation: 74.56%)
• fedora-42-xfce_private_seq1m_q1t1_write 3:write_bandwidth_kb: 43344.00 :small_red_triangle: ( previous job: 79539.00, degradation: 54.49%)
• fedora-42-xfce_private_rnd4k_q32t1_write 3:write_bandwidth_kb: 2154.00 :small_red_triangle: ( previous job: 3765.00, degradation: 57.21%)
• fedora-42-xfce_volatile_seq1m_q8t1_read 3:read_bandwidth_kb: 253156.00 :small_red_triangle: ( previous job: 383531.00, degradation: 66.01%)
• fedora-42-xfce_volatile_seq1m_q8t1_write 3:write_bandwidth_kb: 133538.00 :small_red_triangle: ( previous job: 157382.00, degradation: 84.85%)
• fedora-42-xfce_volatile_seq1m_q1t1_write 3:write_bandwidth_kb: 35901.00 :small_red_triangle: ( previous job: 64217.00, degradation: 55.91%)
• fedora-42-xfce_volatile_rnd4k_q1t1_write 3:write_bandwidth_kb: 1203.00 :small_red_triangle: ( previous job: 2384.00, degradation: 50.46%)

Remaining performance tests:

159 tests

• debian-13-xfce_exec: 8.00 🟢 ( previous job: 8.36, improvement: 95.67%)
• debian-13-xfce_exec-root: 26.88 🟢 ( previous job: 27.36, improvement: 98.25%)
• debian-13-xfce_socket: 8.50 🟢 ( previous job: 8.57, improvement: 99.15%)
• debian-13-xfce_socket-root: 8.62 🔺 ( previous job: 8.26, degradation: 104.41%)
• debian-13-xfce_exec-data-simplex: 61.71 🟢 ( previous job: 72.43, improvement: 85.20%)
• debian-13-xfce_exec-data-duplex: 68.07 🟢 ( previous job: 76.65, improvement: 88.81%)
• debian-13-xfce_exec-data-duplex-root: 86.64 🟢 ( previous job: 91.79, improvement: 94.39%)
• debian-13-xfce_socket-data-duplex: 136.80 🔺 ( previous job: 133.45, degradation: 102.52%)
• fedora-42-xfce_exec: 9.14 🔺 ( previous job: 9.06, degradation: 100.85%)
• fedora-42-xfce_exec-root: 59.17 🔺 ( previous job: 58.19, degradation: 101.70%)
• fedora-42-xfce_socket: 8.36 🟢 ( previous job: 8.48, improvement: 98.59%)
• fedora-42-xfce_socket-root: 8.06 🟢 ( previous job: 8.18, improvement: 98.49%)
• fedora-42-xfce_exec-data-simplex: 68.89 🟢 ( previous job: 78.48, improvement: 87.77%)
• fedora-42-xfce_exec-data-duplex: 73.57 🔺 ( previous job: 67.92, degradation: 108.32%)
• fedora-42-xfce_exec-data-duplex-root: 102.77 🔺 ( previous job: 96.36, degradation: 106.66%)
• fedora-42-xfce_socket-data-duplex: 142.49 🟢 ( previous job: 142.58, improvement: 99.93%)
• whonix-gateway-17_exec: 6.87 🟢 ( previous job: 8.12, improvement: 84.66%)
• whonix-gateway-17_exec-root: 38.36 🟢 ( previous job: 41.05, improvement: 93.44%)
• whonix-gateway-17_socket: 7.28 🟢 ( previous job: 8.52, improvement: 85.36%)
• whonix-gateway-17_socket-root: 6.99 🟢 ( previous job: 8.12, improvement: 86.16%)
• whonix-gateway-17_exec-data-simplex: 70.75 🟢 ( previous job: 83.60, improvement: 84.64%)
• whonix-gateway-17_exec-data-duplex: 74.52 🔺 ( previous job: 68.38, degradation: 108.98%)
• whonix-gateway-17_exec-data-duplex-root: 78.18 🟢 ( previous job: 99.37, improvement: 78.67%)
• whonix-gateway-17_socket-data-duplex: 157.10 🟢 ( previous job: 167.12, improvement: 94.01%)
• whonix-workstation-17_exec: 7.87 🔺 ( previous job: 7.57, degradation: 103.95%)
• whonix-workstation-17_exec-root: 54.81 🟢 ( previous job: 56.76, improvement: 96.57%)
• whonix-workstation-17_socket: 7.91 🟢 ( previous job: 8.59, improvement: 92.12%)
• whonix-workstation-17_socket-root: 8.56 🟢 ( previous job: 8.89, improvement: 96.31%)
• whonix-workstation-17_exec-data-simplex: 67.83 🔺 ( previous job: 66.80, degradation: 101.53%)
• whonix-workstation-17_exec-data-duplex: 71.28 🟢 ( previous job: 74.50, improvement: 95.67%)
• whonix-workstation-17_exec-data-duplex-root: 89.62 🟢 ( previous job: 102.34, improvement: 87.57%)
• whonix-workstation-17_socket-data-duplex: 147.03 🟢 ( previous job: 147.97, improvement: 99.36%)
• dom0_root_seq1m_q8t1_read 3:read_bandwidth_kb: 497663.00 :green_circle: ( previous job: 497426.00, improvement: 100.05%)
• dom0_root_seq1m_q1t1_read 3:read_bandwidth_kb: 446202.00 :green_circle: ( previous job: 431512.00, improvement: 103.40%)
• dom0_root_seq1m_q1t1_write 3:write_bandwidth_kb: 198094.00 :green_circle: ( previous job: 196254.00, improvement: 100.94%)
• dom0_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 11740.00 :green_circle: ( previous job: 5874.00, improvement: 199.86%)
• dom0_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 283.00 :green_circle: ( previous job: 29.00, improvement: 975.86%)
• dom0_varlibqubes_seq1m_q1t1_write 3:write_bandwidth_kb: 110439.00 :small_red_triangle: ( previous job: 121719.00, degradation: 90.73%)
• dom0_varlibqubes_rnd4k_q32t1_read 3:read_bandwidth_kb: 103225.00 :small_red_triangle: ( previous job: 103932.00, degradation: 99.32%)
• dom0_varlibqubes_rnd4k_q32t1_write 3:write_bandwidth_kb: 8005.00 :green_circle: ( previous job: 6356.00, improvement: 125.94%)
• dom0_varlibqubes_rnd4k_q1t1_read 3:read_bandwidth_kb: 8188.00 :green_circle: ( previous job: 7695.00, improvement: 106.41%)
• dom0_varlibqubes_rnd4k_q1t1_write 3:write_bandwidth_kb: 5191.00 :green_circle: ( previous job: 3925.00, improvement: 132.25%)
• fedora-42-xfce_root_seq1m_q8t1_read 3:read_bandwidth_kb: 368179.00 :green_circle: ( previous job: 366891.00, improvement: 100.35%)
• fedora-42-xfce_root_seq1m_q8t1_write 3:write_bandwidth_kb: 149354.00 :green_circle: ( previous job: 140215.00, improvement: 106.52%)
• fedora-42-xfce_root_seq1m_q1t1_read 3:read_bandwidth_kb: 326760.00 :green_circle: ( previous job: 299764.00, improvement: 109.01%)
• fedora-42-xfce_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 90927.00 :green_circle: ( previous job: 86001.00, improvement: 105.73%)
• fedora-42-xfce_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 8499.00 :small_red_triangle: ( previous job: 9042.00, degradation: 93.99%)
• fedora-42-xfce_private_seq1m_q8t1_read 3:read_bandwidth_kb: 393609.00 :green_circle: ( previous job: 387500.00, improvement: 101.58%)
• fedora-42-xfce_private_seq1m_q8t1_write 3:write_bandwidth_kb: 224726.00 :green_circle: ( previous job: 136640.00, improvement: 164.47%)
• fedora-42-xfce_private_seq1m_q1t1_read 3:read_bandwidth_kb: 349176.00 :green_circle: ( previous job: 325139.00, improvement: 107.39%)
• fedora-42-xfce_private_rnd4k_q32t1_read 3:read_bandwidth_kb: 88318.00 :green_circle: ( previous job: 87396.00, improvement: 101.05%)
• fedora-42-xfce_private_rnd4k_q1t1_read 3:read_bandwidth_kb: 8841.00 :small_red_triangle: ( previous job: 8992.00, degradation: 98.32%)
• fedora-42-xfce_private_rnd4k_q1t1_write 3:write_bandwidth_kb: 1380.00 :green_circle: ( previous job: 1251.00, improvement: 110.31%)
• fedora-42-xfce_volatile_seq1m_q1t1_read 3:read_bandwidth_kb: 305707.00 :green_circle: ( previous job: 293225.00, improvement: 104.26%)
• fedora-42-xfce_volatile_rnd4k_q32t1_read 3:read_bandwidth_kb: 79556.00 :small_red_triangle: ( previous job: 87141.00, degradation: 91.30%)
• fedora-42-xfce_volatile_rnd4k_q32t1_write 3:write_bandwidth_kb: 4011.00 :small_red_triangle: ( previous job: 4098.00, degradation: 97.88%)
• fedora-42-xfce_volatile_rnd4k_q1t1_read 3:read_bandwidth_kb: 8147.00 :small_red_triangle: ( previous job: 8804.00, degradation: 92.54%)
• debian-13-xfce_dispvm (mean:6.595): 79.14
• debian-13-xfce_dispvm-gui (mean:7.23): 86.76 🟢 ( previous job: 119.40, improvement: 72.66%)
• debian-13-xfce_dispvm-concurrent (mean:3.215): 38.58
• debian-13-xfce_dispvm-gui-concurrent (mean:3.772): 45.27 🟢 ( previous job: 64.59, improvement: 70.08%)
• debian-13-xfce_dispvm-dom0 (mean:6.899): 82.79
• debian-13-xfce_dispvm-dom0-gui (mean:7.904): 94.84 🟢 ( previous job: 127.44, improvement: 74.43%)
• debian-13-xfce_dispvm-dom0-concurrent (mean:3.688): 44.26
• debian-13-xfce_dispvm-dom0-gui-concurrent (mean:4.274): 51.28 🟢 ( previous job: 65.60, improvement: 78.18%)
• debian-13-xfce_dispvm-preload (mean:3.607): 43.29
• debian-13-xfce_dispvm-preload-gui (mean:4.328): 51.93
• debian-13-xfce_dispvm-preload-concurrent (mean:2.631): 31.57
• debian-13-xfce_dispvm-preload-gui-concurrent (mean:3.366): 40.39
• debian-13-xfce_dispvm-preload-dom0 (mean:3.926): 47.12
• debian-13-xfce_dispvm-preload-dom0-gui (mean:5.402): 64.83
• debian-13-xfce_dispvm-preload-dom0-concurrent (mean:3.282): 39.39
• debian-13-xfce_dispvm-preload-dom0-gui-concurrent (mean:3.997): 47.96
• debian-13-xfce_dispvm-api (mean:7.113): 85.36
• debian-13-xfce_dispvm-gui-api (mean:8.236): 98.84 🟢 ( previous job: 127.48, improvement: 77.53%)
• debian-13-xfce_dispvm-concurrent-api (mean:3.442): 41.30
• debian-13-xfce_dispvm-gui-concurrent-api (mean:4.067): 48.80 🟢 ( previous job: 65.39, improvement: 74.62%)
• debian-13-xfce_dispvm-preload-api (mean:3.895): 46.74
• debian-13-xfce_dispvm-preload-less-api (mean:6.255): 75.06
• debian-13-xfce_dispvm-preload-more-api (mean:3.797): 45.56
• debian-13-xfce_dispvm-preload-gui-api (mean:4.99): 59.88
• debian-13-xfce_dispvm-preload-concurrent-api (mean:3.26): 39.12
• debian-13-xfce_dispvm-preload-gui-concurrent-api (mean:3.945): 47.34
• debian-13-xfce_vm (mean:0.042): 0.51
• debian-13-xfce_vm-gui (mean:0.048): 0.58 🟢 ( previous job: 7.40, improvement: 7.82%)
• debian-13-xfce_vm-concurrent (mean:0.019): 0.22
• debian-13-xfce_vm-gui-concurrent (mean:0.033): 0.40 🟢 ( previous job: 7.33, improvement: 5.47%)
• debian-13-xfce_vm-api (mean:0.041): 0.49
• debian-13-xfce_vm-gui-api (mean:0.046): 0.55 🟢 ( previous job: 2.17, improvement: 25.20%)
• debian-13-xfce_vm-concurrent-api (mean:0.033): 0.40
• debian-13-xfce_vm-gui-concurrent-api (mean:0.038): 0.45 🟢 ( previous job: 1.82, improvement: 24.84%)
• fedora-42-xfce_dispvm (mean:6.883): 82.60 🟢 ( previous job: 111.99, improvement: 73.76%)
• fedora-42-xfce_dispvm-gui (mean:8.566): 102.79 🟢 ( previous job: 131.63, improvement: 78.09%)
• fedora-42-xfce_dispvm-concurrent (mean:3.787): 45.44 🟢 ( previous job: 57.25, improvement: 79.37%)
• fedora-42-xfce_dispvm-gui-concurrent (mean:4.461): 53.54 🟢 ( previous job: 74.72, improvement: 71.65%)
• fedora-42-xfce_dispvm-dom0 (mean:7.83): 93.96 🟢 ( previous job: 124.92, improvement: 75.22%)
• fedora-42-xfce_dispvm-dom0-gui (mean:9.252): 111.02 🟢 ( previous job: 147.17, improvement: 75.44%)
• fedora-42-xfce_dispvm-dom0-concurrent (mean:4.09): 49.08 🟢 ( previous job: 64.09, improvement: 76.57%)
• fedora-42-xfce_dispvm-dom0-gui-concurrent (mean:5.036): 60.44 🟢 ( previous job: 75.59, improvement: 79.95%)
• fedora-42-xfce_dispvm-preload (mean:4.099): 49.19 🟢 ( previous job: 69.72, improvement: 70.54%)
• fedora-42-xfce_dispvm-preload-gui (mean:4.793): 57.52 🟢 ( previous job: 79.21, improvement: 72.61%)
• fedora-42-xfce_dispvm-preload-concurrent (mean:2.975): 35.70 🟢 ( previous job: 49.89, improvement: 71.56%)
• fedora-42-xfce_dispvm-preload-gui-concurrent (mean:3.852): 46.22 🟢 ( previous job: 69.08, improvement: 66.91%)
• fedora-42-xfce_dispvm-preload-dom0 (mean:4.305): 51.66 🟢 ( previous job: 72.35, improvement: 71.40%)
• fedora-42-xfce_dispvm-preload-dom0-gui (mean:5.611): 67.34 🟢 ( previous job: 91.39, improvement: 73.68%)
• fedora-42-xfce_dispvm-preload-dom0-concurrent (mean:3.613): 43.35 🟢 ( previous job: 57.13, improvement: 75.89%)
• fedora-42-xfce_dispvm-preload-dom0-gui-concurrent (mean:4.128): 49.53 🟢 ( previous job: 67.89, improvement: 72.96%)
• fedora-42-xfce_dispvm-api (mean:7.794): 93.53 🟢 ( previous job: 128.15, improvement: 72.98%)
• fedora-42-xfce_dispvm-gui-api (mean:9.18): 110.16 🟢 ( previous job: 149.03, improvement: 73.92%)
• fedora-42-xfce_dispvm-concurrent-api (mean:3.804): 45.65 🟢 ( previous job: 66.32, improvement: 68.83%)
• fedora-42-xfce_dispvm-gui-concurrent-api (mean:4.613): 55.36 🟢 ( previous job: 77.33, improvement: 71.59%)
• fedora-42-xfce_dispvm-preload-api (mean:4.376): 52.51 🟢 ( previous job: 73.95, improvement: 71.01%)
• fedora-42-xfce_dispvm-preload-less-api (mean:7.248): 86.97 🟢 ( previous job: 116.39, improvement: 74.72%)
• fedora-42-xfce_dispvm-preload-more-api (mean:4.465): 53.58 🟢 ( previous job: 71.33, improvement: 75.12%)
• fedora-42-xfce_dispvm-preload-gui-api (mean:5.735): 68.83 🟢 ( previous job: 92.06, improvement: 74.76%)
• fedora-42-xfce_dispvm-preload-concurrent-api (mean:3.614): 43.37 🟢 ( previous job: 61.60, improvement: 70.40%)
• fedora-42-xfce_dispvm-preload-gui-concurrent-api (mean:4.111): 49.33 🟢 ( previous job: 77.66, improvement: 63.53%)
• fedora-42-xfce_vm (mean:0.031): 0.37 🟢 ( previous job: 9.19, improvement: 4.04%)
• fedora-42-xfce_vm-gui (mean:0.036): 0.43 🟢 ( previous job: 9.01, improvement: 4.83%)
• fedora-42-xfce_vm-concurrent (mean:0.019): 0.23 🟢 ( previous job: 8.88, improvement: 2.57%)
• fedora-42-xfce_vm-gui-concurrent (mean:0.031): 0.37 🟢 ( previous job: 9.15, improvement: 4.04%)
• fedora-42-xfce_vm-api (mean:0.034): 0.41 🟢 ( previous job: 2.24, improvement: 18.32%)
• fedora-42-xfce_vm-gui-api (mean:0.04): 0.47 🟢 ( previous job: 2.33, improvement: 20.36%)
• fedora-42-xfce_vm-concurrent-api (mean:0.027): 0.32 🟢 ( previous job: 1.62, improvement: 19.85%)
• fedora-42-xfce_vm-gui-concurrent-api (mean:0.032): 0.38 🟢 ( previous job: 2.20, improvement: 17.40%)
• whonix-workstation-17_dispvm (mean:7.549): 90.59 🟢 ( previous job: 123.87, improvement: 73.13%)
• whonix-workstation-17_dispvm-gui (mean:8.644): 103.73 🟢 ( previous job: 148.68, improvement: 69.77%)
• whonix-workstation-17_dispvm-concurrent (mean:4.03): 48.35 🟢 ( previous job: 77.00, improvement: 62.80%)
• whonix-workstation-17_dispvm-gui-concurrent (mean:4.875): 58.50 🟢 ( previous job: 89.18, improvement: 65.59%)
• whonix-workstation-17_dispvm-dom0 (mean:8.457): 101.48 🟢 ( previous job: 135.24, improvement: 75.04%)
• whonix-workstation-17_dispvm-dom0-gui (mean:9.596): 115.15 🟢 ( previous job: 159.23, improvement: 72.32%)
• whonix-workstation-17_dispvm-dom0-concurrent (mean:4.375): 52.51 🟢 ( previous job: 76.91, improvement: 68.27%)
• whonix-workstation-17_dispvm-dom0-gui-concurrent (mean:5.221): 62.65 🟢 ( previous job: 87.45, improvement: 71.65%)
• whonix-workstation-17_dispvm-preload (mean:7.669): 92.03 🟢 ( previous job: 124.31, improvement: 74.03%)
• whonix-workstation-17_dispvm-preload-gui (mean:8.645): 103.74 🟢 ( previous job: 138.71, improvement: 74.79%)
• whonix-workstation-17_dispvm-preload-concurrent (mean:4.004): 48.05 🟢 ( previous job: 66.00, improvement: 72.80%)
• whonix-workstation-17_dispvm-preload-gui-concurrent (mean:4.767): 57.20 🟢 ( previous job: 77.46, improvement: 73.85%)
• whonix-workstation-17_dispvm-preload-dom0 (mean:4.701): 56.41 🟢 ( previous job: 81.24, improvement: 69.43%)
• whonix-workstation-17_dispvm-preload-dom0-gui (mean:6.269): 75.23 🟢 ( previous job: 102.71, improvement: 73.24%)
• whonix-workstation-17_dispvm-preload-dom0-concurrent (mean:4.006): 48.07 🟢 ( previous job: 72.84, improvement: 65.99%)
• whonix-workstation-17_dispvm-preload-dom0-gui-concurrent (mean:4.761): 57.14 🟢 ( previous job: 87.14, improvement: 65.57%)
• whonix-workstation-17_dispvm-api (mean:8.427): 101.13 🟢 ( previous job: 140.50, improvement: 71.98%)
• whonix-workstation-17_dispvm-gui-api (mean:9.671): 116.05 🟢 ( previous job: 157.40, improvement: 73.73%)
• whonix-workstation-17_dispvm-concurrent-api (mean:4.395): 52.74 🟢 ( previous job: 76.06, improvement: 69.34%)
• whonix-workstation-17_dispvm-gui-concurrent-api (mean:5.015): 60.18 🟢 ( previous job: 87.18, improvement: 69.03%)
• whonix-workstation-17_dispvm-preload-api (mean:4.826): 57.91 🟢 ( previous job: 84.09, improvement: 68.87%)
• whonix-workstation-17_dispvm-preload-less-api (mean:7.738): 92.86 🟢 ( previous job: 126.04, improvement: 73.68%)
• whonix-workstation-17_dispvm-preload-more-api (mean:5.065): 60.78 🟢 ( previous job: 89.12, improvement: 68.20%)
• whonix-workstation-17_dispvm-preload-gui-api (mean:6.086): 73.04 🟢 ( previous job: 101.72, improvement: 71.80%)
• whonix-workstation-17_dispvm-preload-concurrent-api (mean:4.021): 48.25 🟢 ( previous job: 71.58, improvement: 67.41%)
• whonix-workstation-17_dispvm-preload-gui-concurrent-api (mean:5.037): 60.44 🟢 ( previous job: 88.24, improvement: 68.49%)
• whonix-workstation-17_vm (mean:0.036): 0.43 🟢 ( previous job: 9.27, improvement: 4.67%)
• whonix-workstation-17_vm-gui (mean:0.054): 0.65 🟢 ( previous job: 9.82, improvement: 6.63%)
• whonix-workstation-17_vm-concurrent (mean:0.03): 0.36 🟢 ( previous job: 8.93, improvement: 4.01%)
• whonix-workstation-17_vm-gui-concurrent (mean:0.029): 0.35 🟢 ( previous job: 9.38, improvement: 3.75%)
• whonix-workstation-17_vm-api (mean:0.044): 0.53 🟢 ( previous job: 2.56, improvement: 20.79%)
• whonix-workstation-17_vm-gui-api (mean:0.044): 0.53 🟢 ( previous job: 2.53, improvement: 20.96%)
• whonix-workstation-17_vm-concurrent-api (mean:0.027): 0.33 🟢 ( previous job: 1.81, improvement: 18.11%)
• whonix-workstation-17_vm-gui-concurrent-api (mean:0.027): 0.32 🟢 ( previous job: 2.57, improvement: 12.59%)

[marmarek]

Note to self: kernel property can be empty for in-vm kernel in PV/PVH too, but kernelopts is ignored then.

[ArrayBolt3]

Took the patch submitted upstream at https://lists.gnu.org/archive/html/grub-devel/2025-06/msg00012.html, and backported it to Qubes OS. The patch built successfully, and I was able to install it on my Qubes R4.3 system and run it through a full suite of tests. The same testing methodology listed at #16 (comment) was used, although I logged into the VM as user rather than root each time.

Test results:

Input command line	Test commands	Expected output	PVH	PV
xen_grub_env_boot_mode_opts="test1 test2"	echo $xen_grub_env_boot_mode_opts	test1 test2	[x]	[x]
xen_grub_env_Boot_Mode_Opts="test1 test2"	echo $xen_grub_env_Boot_Mode_Opts	test1 test2	[x]	[x]
xen_grub_env_boot_mode_opts='test1 test2'	echo $xen_grub_env_boot_mode_opts	test1 test2	[x]	[x]
xen_grub_env_boot_mode_opts=test1\ test2	echo $xen_grub_env_boot_mode_opts	test1 test2	[x]	[x]
xen_grub_env_boot_mode_opts="test1\\ test2"	echo $xen_grub_env_boot_mode_opts	test1\ test2	[x]	[x]
xen_grub_env_boot_mode_opts="test1' test2"	echo $xen_grub_env_boot_mode_opts	test1' test2	[x]	[x]
xen_grub_env_boot_mode_opts='test1" test2'	echo $xen_grub_env_boot_mode_opts	test1" test2	[x]	[x]
xen_grub_env_boot_mode_opts='test1' test'	echo $xen_grub_env_boot_mode_opts, echo $test	test1,	[x]	[x]
xen_grub_env_boot_mode_opts="test1" test'	echo $xen_grub_env_boot_mode_opts, echo $test	test1,	[x]	[x]
xen_grub_env_boot_mode_opts="test1" test"	echo $xen_grub_env_boot_mode_opts, echo $test	test1,	[x]	[x]
xen_grub_env_boot_mode_opts='test1' xen_grub_env_test'	echo $xen_grub_env_boot_mode_opts, echo $xen_grub_env_test	test1, 1	[x]	[x]
xen_grub_env_boot_mode_opts="test1" xen_grub_env_test'	echo $xen_grub_env_boot_mode_opts, echo $xen_grub_env_test	test1, 1	[x]	[x]
xen_grub_env_boot_mode_opts="test1" xen_grub_env_test"	echo $xen_grub_env_boot_mode_opts, echo $xen_grub_env_test	test1, 1	[x]	[x]
xen_grub_env_boot_mode_opts=<995 digits here>	echo $xen_grub_env_boot_mode_opts	<995 digits here>	[x]	[x]
xen_grub_env_boot_mode_opts=<996 digits here>	echo $xen_grub_env_boot_mode_opts	on PVH, <995 digits here> on PV	[x]	[x]
xen_grub_env_a=<10,000 digits here>	echo $xen_grub_env_a	on PVH, <1008 digits here> on PV	[x]	[x]
xen_grub_env_arg_one='arg1' xen_grub_env_arg_two='arg2'	echo $xen_grub_env_arg_one, echo $xen_grub_env_arg_two	arg1, arg2	[x]	[x]
xen_grub_env_arg1='arg_one' xen_grub_env_arg2='arg_two'	echo $xen_grub_env_arg1, echo $xen_grub_env_arg2	,	[x]	[x]
xen_grub_env_boot_mode_opts=	`echo $xen_grub_env_boot_mode_opts		[x]	[x]
xen_grub_env_arg1='arg_one' xen_grub_env_arg_two='arg2'	echo $xen_grub_env_arg1, echo $xen_grub_env_arg_two	, arg2	[x]	[x]
xen_grub_env_boot_mode_opts="^Aabc"	echo $xen_grub_env_boot_mode_opts		[?]	[?]
xen_grub_env_boot_mode_^Aopts="abc"	echo $xen_grub_env_boot_mode_opts		[?]	[?]
xen_grub_env_do_thing	echo $xen_grub_env_do_thing	1	[x]	[x]
xen_grub_env_do_^Athing	echo $xen_grub_env_do_thing		[?]	[?]
xen_grub_env_do_^Athing xen_grub_env_do_other	echo $xen_grub_env_do_thing, echo $xen_grub_env_do_other	,	[?]	[?]
""		Boot without crash	[x]	[x]
""=""		Boot without crash	[x]	[x]
=		Boot without crash	[x]	[x]
\		Boot without crash	[x]	[x]
xen_grub_env_var=\""a"\'	echo $xen_grub_env_var	"a'	[x]	[x]
xen_grub_env_var=\''a'\"	echo $xen_grub_env_var	'a"	[x]	[x]
xen_grub_env_var_a=\'"a"\" xen_grub_env_var_b=\"'b'\'	echo $xen_grub_env_var_a, echo $xen_grub_env_var_b	'a", "b'	[x]	[x]

As all of the tests appear to pass to me, and all review notes both from the grub-devel mailing list and from this PR have been integrated into the patch, I believe this is ready for review.

[ArrayBolt3]

@marmarek Updated code to match the latest submission to the grub-devel ML.

I re-ran the full suite of tests from above again, but I did NOT test PV this time, since I tried to use a Fedora-based VM for testing this time and getting Fedora + pvgrub + PV virtualization to work is just about impossible (and is part of the inspiration for the "let's remove PV support entirely" feature request I filed). Usually I'd use the extract-vmlinux script from kernel-devel to work around PV's shortcomings, but that script apparently doesn't recognize Fedora's kernels anymore, so that wasn't possible. All of the PVH tests passed though.

[marmarek]

PipelineRetry