Skip to content

tests: log more info on failure#736

Draft
marmarek wants to merge 1 commit intoQubesOS:mainfrom
marmarek:debug-net-reconnect
Draft

tests: log more info on failure#736
marmarek wants to merge 1 commit intoQubesOS:mainfrom
marmarek:debug-net-reconnect

Conversation

@marmarek
Copy link
Copy Markdown
Member

Log frontend/backend state from xenstore.

@marmarek marmarek marked this pull request as draft October 14, 2025 12:43
Log frontend/backend state from xenstore.
@qubesos-bot
Copy link
Copy Markdown

qubesos-bot commented Oct 14, 2025

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2026042817-devel&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2026032404-devel&flavor=update

  • system_tests_basic_vm_qrexec_gui

    • TC_20_NonAudio_debian-13-xfce: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_fedora-43-xfce: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-gateway-18: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-workstation-18: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

  • system_tests_extra

    • TC_00_PDFConverter_debian-13-xfce: test_004_cancel_stops_conversion (failure)
      AssertionError: DispVM not cleaned up 10s after cancel: {<DispVM at...

    • TC_00_PDFConverter_fedora-43-xfce: test_004_cancel_stops_conversion (failure)
      AssertionError: DispVM not cleaned up 10s after cancel: {<DispVM at...

    • TC_01_InputProxyExclude_fedora-43-xfce: test_000_qemu_tablet (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • system_tests_usbproxy

    • TC_20_USBProxy_core3_debian-13-xfce: test_090_attach_stubdom (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_20_USBProxy_core3_fedora-43-xfce: test_090_attach_stubdom (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • system_tests_network_ipv6

  • install_default_upload@hw1

    • firstboot: unnamed test (unknown)

    • firstboot: Stall detected (unknown)
      Stall was detected during assert_screen fail...

    • firstboot: Failed (test died)
      # Test died: no candidate needle with tag(s) 'luks-prompt, firstboo...

  • system_tests_basic_vm_qrexec_gui_zfs

    • TC_20_NonAudio_debian-13-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_fedora-43-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-gateway-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-workstation-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

  • system_tests_qwt_win10_seamless@hw13

    • windows_clipboard_and_filecopy: unnamed test (unknown)
    • windows_clipboard_and_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'windows-Explorer-new-...
  • system_tests_qwt_win11@hw13

    • windows_clipboard_and_filecopy: unnamed test (unknown)
    • windows_clipboard_and_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'windows-Notepad' matc...
  • system_tests_guivm_gpu_gui_interactive@hw13

    • shutdown: unnamed test (unknown)
    • shutdown: Failed (test died)
      # Test died: no candidate needle with tag(s) 'text-logged-in-root' ...
  • system_tests_gpu_passthrough@hw13

    • gpu_passthrough: Failed (test died)
      # Test died: command 'ansible-playbook -i inventory -e pci_devices=...
  • system_tests_basic_vm_qrexec_gui_btrfs

    • TC_20_NonAudio_debian-13-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_fedora-43-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-gateway-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-workstation-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

  • system_tests_basic_vm_qrexec_gui_ext4

    • TC_20_NonAudio_debian-13-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_fedora-43-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-gateway-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-workstation-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

  • system_tests_basic_vm_qrexec_gui_xfs

    • TC_20_NonAudio_debian-13-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_fedora-43-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-gateway-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-workstation-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

Failed tests

43 failures
  • system_tests_whonix

    • whonixcheck: fail (unknown)
      Whonixcheck for sys-whonix failed...

    • whonixcheck: Failed (test died)
      # Test died: systemcheck failed at qubesos/tests/whonixcheck.pm lin...

  • system_tests_basic_vm_qrexec_gui

    • TC_20_NonAudio_debian-13-xfce: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_fedora-43-xfce: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-gateway-18: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-workstation-18: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

  • system_tests_pvgrub_salt_storage

  • system_tests_extra

    • TC_00_PDFConverter_debian-13-xfce: test_004_cancel_stops_conversion (failure)
      AssertionError: DispVM not cleaned up 10s after cancel: {<DispVM at...

    • TC_00_PDFConverter_fedora-43-xfce: test_004_cancel_stops_conversion (failure)
      AssertionError: DispVM not cleaned up 10s after cancel: {<DispVM at...

    • TC_01_InputProxyExclude_debian-13-xfce: test_000_qemu_tablet (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_01_InputProxyExclude_fedora-43-xfce: test_000_qemu_tablet (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • system_tests_usbproxy

    • TC_20_USBProxy_core3_debian-13-xfce: test_090_attach_stubdom (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_20_USBProxy_core3_fedora-43-xfce: test_090_attach_stubdom (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • system_tests_network_ipv6

  • install_default_upload@hw1

    • firstboot: unnamed test (unknown)

    • firstboot: Stall detected (unknown)
      Stall was detected during assert_screen fail...

    • firstboot: Failed (test died)
      # Test died: no candidate needle with tag(s) 'luks-prompt, firstboo...

  • system_tests_basic_vm_qrexec_gui_zfs

    • TC_20_NonAudio_debian-13-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_fedora-43-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-gateway-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-workstation-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

  • system_tests_guivm_vnc_gui_interactive

    • [unstable] collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/dEXuX-\d+-/...

    • [unstable] collect_logs: Failed (test died + timed out)
      # Test died: command 'curl --form upload=@journalctl.log --form upn...

    • [unstable] collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/JtfXt-\d+-/...

  • system_tests_qwt_win10_seamless@hw13

    • windows_clipboard_and_filecopy: unnamed test (unknown)
    • windows_clipboard_and_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'windows-Explorer-new-...
  • system_tests_qwt_win11@hw13

    • windows_clipboard_and_filecopy: unnamed test (unknown)
    • windows_clipboard_and_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'windows-Notepad' matc...
  • system_tests_guivm_gpu_gui_interactive@hw13

    • shutdown: unnamed test (unknown)
    • shutdown: Failed (test died)
      # Test died: no candidate needle with tag(s) 'text-logged-in-root' ...
  • system_tests_gpu_passthrough@hw13

    • gpu_passthrough: Failed (test died)
      # Test died: command 'ansible-playbook -i inventory -e pci_devices=...
  • system_tests_basic_vm_qrexec_gui_btrfs

    • TC_20_NonAudio_debian-13-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_fedora-43-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-gateway-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-workstation-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

  • system_tests_basic_vm_qrexec_gui_ext4

    • TC_20_NonAudio_debian-13-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_fedora-43-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-gateway-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-workstation-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

  • system_tests_basic_vm_qrexec_gui_xfs

    • TC_20_NonAudio_debian-13-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_fedora-43-xfce-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-gateway-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

    • TC_20_NonAudio_whonix-workstation-18-pool: test_500_gui_agent_env_sync (failure)
      AssertionError: unexpected QUBES_ENV_TEST value from session, got '...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/170766#dependencies

27 fixed
  • system_tests_network

    • system_tests: Fail (unknown)
      Tests qubes.tests.integ.network failed (exit code 1), details repor...

    • system_tests: Failed (test died)
      # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...

    • VmNetworking_debian-13-xfce: test_203_fake_ip_inter_vm_allow (failure)
      ^... AssertionError: 1 != 0

  • system_tests_pvgrub_salt_storage

    • TC_41_HVMGrub_debian-13-xfce: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_41_HVMGrub_fedora-42-xfce: test_000_standalone_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_41_HVMGrub_fedora-42-xfce: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • system_tests_gui_interactive

    • collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/Dhelp-\d+-/...

    • collect_logs: Failed (test died + timed out)
      # Test died: command 'curl --form upload=@journalctl.log --form upn...

  • system_tests_network_ipv6

  • system_tests_network_updates

    • system_tests: Fail (unknown)
      Tests qubes.tests.integ.dom0_update failed (exit code 1), details r...

    • system_tests: Failed (test died)
      # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...

    • TC_00_Dom0Upgrade_whonix-gateway-18: test_020_install_wrong_sign (error)
      subprocess.CalledProcessError: Command 'timeout=120; while ! tor-ci...

    • TC_11_QvmTemplateMgmtVM_debian-13-xfce: test_000_template_list (failure)
      qvm-template: error: No matching templates to list

    • TC_11_QvmTemplateMgmtVM_debian-13-xfce: test_010_template_install (failure)
      qvm-template: error: Template 'debian-12-minimal' not found.

    • TC_11_QvmTemplateMgmtVM_fedora-42-xfce: test_000_template_list (failure)
      qvm-template: error: No matching templates to list

    • TC_11_QvmTemplateMgmtVM_fedora-42-xfce: test_010_template_install (failure)
      qvm-template: error: Template 'debian-12-minimal' not found.

    • TC_11_QvmTemplateMgmtVM_whonix-gateway-18: test_000_template_list (failure)
      qvm-template: error: No matching templates to list

    • TC_11_QvmTemplateMgmtVM_whonix-gateway-18: test_010_template_install (failure)
      qvm-template: error: Template 'debian-12-minimal' not found.

  • system_tests_kde_gui_interactive

    • collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/bMse8-\d+-/...

    • collect_logs: Failed (test died + timed out)
      # Test died: command 'curl --form upload=@journalctl.log --form upn...

  • system_tests_audio

    • system_tests: Fail (unknown)
      Tests qubes.tests.integ.audio failed (exit code 1), details reporte...

    • system_tests: Failed (test died)
      # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...

    • TC_20_AudioVM_Pulse_whonix-workstation-18: test_225_audio_rec_unmuted_hvm (failure)
      AssertionError: too short audio, expected 10s, got 7.59433106575963...

    • TC_20_AudioVM_PipeWire_debian-13-xfce: test_251_audio_playback_audiovm_pipewire_late_start (failure)
      AssertionError: pacat for test-inst-vm1 (xid 48) running(False) in ...

  • system_tests_qwt_win10@hw13

    • windows_install: Failed (test died)
      # Test died: Install failed with code 1 at qubesos/tests/windows_in...
  • system_tests_qwt_win10_seamless@hw13

    • windows_install: Failed (test died)
      # Test died: Install failed with code 1 at qubesos/tests/windows_in...
  • system_tests_qwt_win11@hw13

    • windows_install: Failed (test died)
      # Test died: Install failed with code 1 at qubesos/tests/windows_in...

Unstable tests

Details
  • system_tests_guivm_vnc_gui_interactive

    collect_logs/Failed (1/5 times with errors)
    • job 172242 # Test died: command 'curl --form upload=@journalctl.log --form upn...
    collect_logs/wait_serial (1/5 times with errors)
    • job 172242 # wait_serial expected: qr/73DgV-\d+-/...
    collect_logs/wait_serial (1/5 times with errors)
    • job 172242 # wait_serial expected: qr/uOleL-\d+-/...

Performance Tests

Performance degradation:

No issues

Remaining performance tests:

39 tests
  • debian-13-xfce_dom0-dispvm-api (mean:6.0): 72.01 🟢 ( previous job: 81.47, improvement: 88.38%)
  • debian-13-xfce_dom0-dispvm-gui-api (mean:7.771): 93.26 🔻 ( previous job: 92.38, degradation: 100.95%)
  • debian-13-xfce_dom0-dispvm-preload-2-api (mean:3.192): 38.31 🟢 ( previous job: 48.28, improvement: 79.34%)
  • debian-13-xfce_dom0-dispvm-preload-2-delay-0-api (mean:2.984): 35.81 🟢 ( previous job: 44.34, improvement: 80.76%)
  • debian-13-xfce_dom0-dispvm-preload-2-delay-minus-1d2-api (mean:3.281): 39.38 🟢 ( previous job: 54.23, improvement: 72.61%)
  • debian-13-xfce_dom0-dispvm-preload-4-api (mean:2.128): 25.53 🟢 ( previous job: 40.37, improvement: 63.24%)
  • debian-13-xfce_dom0-dispvm-preload-4-delay-0-api (mean:2.609): 31.31 🟢 ( previous job: 44.04, improvement: 71.09%)
  • debian-13-xfce_dom0-dispvm-preload-4-delay-minus-1d2-api (mean:2.392): 28.71 🟢 ( previous job: 45.36, improvement: 63.29%)
  • debian-13-xfce_dom0-dispvm-preload-2-gui-api (mean:4.605): 55.26 🟢 ( previous job: 58.18, improvement: 94.98%)
  • debian-13-xfce_dom0-dispvm-preload-4-gui-api (mean:3.654): 43.84 🔻 ( previous job: 43.54, degradation: 100.69%)
  • debian-13-xfce_dom0-dispvm-preload-6-gui-api (mean:3.262): 39.14 🟢 ( previous job: 47.37, improvement: 82.62%)
  • debian-13-xfce_dom0-vm-api (mean:0.037): 0.44 🟢 ( previous job: 0.46, improvement: 96.73%)
  • debian-13-xfce_dom0-vm-gui-api (mean:0.04): 0.48 🟢 ( previous job: 0.51, improvement: 94.85%)
  • fedora-43-xfce_dom0-dispvm-api (mean:7.536): 90.43
  • fedora-43-xfce_dom0-dispvm-gui-api (mean:9.498): 113.97
  • fedora-43-xfce_dom0-dispvm-preload-2-api (mean:3.549): 42.59
  • fedora-43-xfce_dom0-dispvm-preload-2-delay-0-api (mean:3.608): 43.30
  • fedora-43-xfce_dom0-dispvm-preload-2-delay-minus-1d2-api (mean:3.937): 47.24
  • fedora-43-xfce_dom0-dispvm-preload-4-api (mean:2.896): 34.75
  • fedora-43-xfce_dom0-dispvm-preload-4-delay-0-api (mean:2.867): 34.40
  • fedora-43-xfce_dom0-dispvm-preload-4-delay-minus-1d2-api (mean:2.964): 35.56
  • fedora-43-xfce_dom0-dispvm-preload-2-gui-api (mean:5.561): 66.73
  • fedora-43-xfce_dom0-dispvm-preload-4-gui-api (mean:4.237): 50.84
  • fedora-43-xfce_dom0-dispvm-preload-6-gui-api (mean:3.648): 43.77
  • fedora-43-xfce_dom0-vm-api (mean:0.033): 0.39
  • fedora-43-xfce_dom0-vm-gui-api (mean:0.044): 0.53
  • whonix-workstation-18_dom0-dispvm-api (mean:7.871): 94.45 🟢 ( previous job: 114.77, improvement: 82.30%)
  • whonix-workstation-18_dom0-dispvm-gui-api (mean:10.342): 124.10 🟢 ( previous job: 127.27, improvement: 97.51%)
  • whonix-workstation-18_dom0-dispvm-preload-2-api (mean:4.232): 50.78 🟢 ( previous job: 70.96, improvement: 71.56%)
  • whonix-workstation-18_dom0-dispvm-preload-2-delay-0-api (mean:3.949): 47.39 🟢 ( previous job: 65.29, improvement: 72.59%)
  • whonix-workstation-18_dom0-dispvm-preload-2-delay-minus-1d2-api (mean:5.013): 60.16 🟢 ( previous job: 74.32, improvement: 80.94%)
  • whonix-workstation-18_dom0-dispvm-preload-4-api (mean:3.421): 41.05 🟢 ( previous job: 57.74, improvement: 71.09%)
  • whonix-workstation-18_dom0-dispvm-preload-4-delay-0-api (mean:3.544): 42.52 🟢 ( previous job: 65.76, improvement: 64.67%)
  • whonix-workstation-18_dom0-dispvm-preload-4-delay-minus-1d2-api (mean:3.399): 40.79 🟢 ( previous job: 59.80, improvement: 68.21%)
  • whonix-workstation-18_dom0-dispvm-preload-2-gui-api (mean:6.036): 72.43 🟢 ( previous job: 78.19, improvement: 92.64%)
  • whonix-workstation-18_dom0-dispvm-preload-4-gui-api (mean:4.767): 57.20 🟢 ( previous job: 65.73, improvement: 87.03%)
  • whonix-workstation-18_dom0-dispvm-preload-6-gui-api (mean:4.054): 48.65 🟢 ( previous job: 61.35, improvement: 79.29%)
  • whonix-workstation-18_dom0-vm-api (mean:0.043): 0.52 🟢 ( previous job: 0.58, improvement: 89.58%)
  • whonix-workstation-18_dom0-vm-gui-api (mean:0.039): 0.47 🟢 ( previous job: 0.62, improvement: 74.80%)

@ben-grande
Copy link
Copy Markdown
Contributor

  • system_tests: Failed (test died)
    # Test died: command 'curl --form upload=@tests-qubes.tests.integ.n...

Test results were not uploaded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants