fix(tests): avoid soft lockup in test_serial_active_tx_snapshot

The test was hanging because `cat /dev/zero > /dev/ttyS0` on a multi-vCPU
guest triggers a soft lockup in the 8250 IRQ handler. The root cause is
that FC's serial emulation re-asserts IIR (THRE pending) on every THR
write, causing the guest's serial8250_interrupt() loop to run up to
PASS_LIMIT=512 iterations per interrupt with IRQs disabled (4-12ms).
When writer and IRQ handler are on different CPUs, the writer continuously
re-asserts IIR between the handler's reads, so the loop never exits
naturally.

Fix by:
- Using SSH as the control channel instead of serial (which is unusable
  during the flood)
- Using 4 vCPUs with explicit IRQ affinity pinning: all IRQs to CPU0
  (keeping SSH/virtio-net responsive), serial IRQ to CPU1 (isolating the
  lockup), writer pinned to CPU3 (cross-CPU to exercise the worst case)
- After snapshot/restore, killing the writer via SSH then verifying
  serial console functionality

This preserves the original test intent (snapshot with active serial TX
on a multi-vCPU guest) while working around the known device emulation
bug.

Signed-off-by: Jack Thomson <jackabt@amazon.com>

Author: JackThomson2

tests/integration_tests/functional/test_serial_io.py

@@ -66,25 +66,33 @@ def test_serial_active_tx_snapshot(uvm_plain, microvm_factory):
     microvm.help.enable_console()
     microvm.spawn(serial_out_path=None)
     microvm.basic_config(
-        vcpu_count=2,
+        vcpu_count=4,
         mem_size_mib=256,
     )
+    microvm.add_net_iface()
     serial = Serial(microvm)
     serial.open()
     microvm.start()
 
-    # looking for the # prompt at the end
-    serial.rx(microvm.distro.shell_prompt)
-
-    # Start an unbounded serial transmission from inside the guest such that
-    # there will be an active transmission at the point of pausing the VM to
-    # take the snapshot. This will saturate the TX buffer of the UART and it
-    # might make the guest driver enable TX interrupts.
-    serial.tx("cat /dev/zero")
-    # Give the guest time to start the transmission
-    time.sleep(1)
+    # Pin all IRQs to CPU0 first, then move serial to CPU1. This ensures the
+    # serial flood lockup (which starves the IRQ-handling CPU) is isolated to
+    # CPU1, while SSH (virtio-net) stays on CPU0 and remains reachable.
+    # Pin the writer (cat) to CPU3 so it's on a different CPU than the serial
+    # IRQ handler — this is the cross-CPU configuration that triggers the
+    # lockup (writer re-asserts IIR from CPU3 while handler loops on CPU1).
+    microvm.ssh.check_output(
+        "for irq in $(ls /proc/irq/ | grep -E '^[0-9]+$'); do"
+        "  echo 1 > /proc/irq/$irq/smp_affinity 2>/dev/null || true;"
+        " done;"
+        " SER_IRQ=$(awk '/ttyS0/{print $1}' /proc/interrupts | tr -d :);"
+        " echo 2 > /proc/irq/$SER_IRQ/smp_affinity;"
+        " nohup taskset -c 3 cat /dev/zero > /dev/ttyS0 2>/dev/null &"
+    )
+    # Give the guest time to saturate the TX buffer
+    time.sleep(2)
 
-    # Create snapshot.
+    # Create snapshot — FC pause works even during the serial flood because
+    # vCPU threads exit KVM_RUN between IO exits.
     snapshot = microvm.snapshot_full()
     # Kill base microVM.
     microvm.kill()
@@ -94,12 +102,17 @@ def test_serial_active_tx_snapshot(uvm_plain, microvm_factory):
     vm.help.enable_console()
     vm.spawn(serial_out_path=None)
     vm.restore_from_snapshot(snapshot, resume=True)
+
+    # The restored VM resumes the cat flood. Kill it via SSH — the serial IRQ
+    # affinity persists from the snapshot so only CPU1 is affected and SSH on
+    # CPU0 remains reachable.
+    vm.ssh.check_output("pkill -9 cat || true")
+    time.sleep(0.5)
+
+    # Verify the serial console is functional after stopping the flood
     serial = Serial(vm)
     serial.open()
-
-    # Send Ctrl-C to the guest to stop the ongoing transmission and regain the shell
-    serial.tx("\x03", end="")
-    # looking for the # prompt at the end
+    serial.tx("")
     serial.rx(vm.distro.shell_prompt)
     serial.tx("pwd")
     res = serial.rx("#")