KubeVirt eval failures

[lyarwood]

/cc @ksimon1
/cc @codingben
/cc @Ruclo

While these evals are now running we are seeing failures that need to be addressed:

#835 (comment)

$ mcpchecker summary ~/Downloads/mcpchecker-results/mcpchecker-openai-agent-kubernetes-test-out.json
=== Evaluation Summary ===

  ✓ clone-vm (assertions: 3/3)
  ✓ create-basic-vm (assertions: 3/3)
  ✓ create-ubuntu-vm (assertions: 3/3)
  ✓ create-vm-with-instancetype (assertions: 3/3)
  ✓ create-vm-with-size (assertions: 3/3)
  ✓ create-vm-with-vlan (assertions: 3/3)
  ✗ delete-vm
      failed to setup task: setup[0] failed: script execution failed: exit status 1
output: namespace/vm-test created
virtualmachine.kubevirt.io/deleted-vm created
virtualmachine.kubevirt.io/deleted-vm condition met
error: timed out waiting for the condition on virtualmachines/deleted-vm

  ✗ pause-vm (assertions: 3/3)
      verification failed: verify[0] failed: script execution failed: exit status 1
output: error: timed out waiting for the condition on virtualmachines/paused-vm

  ✗ restore-vm (assertions: 1/3)
      verification failed: verify[0] failed: script execution failed: exit status 1
output: ERROR: VirtualMachineRestore 'test-restore' not found

      - ToolsUsed: Required tool not called: server=kubernetes, tool=, pattern=.*
      - MinToolCalls: Too few tool calls: expected >= 1, got 0
  ✗ snapshot-vm (assertions: 1/3)
      verification failed: verify[0] failed: script execution failed: exit status 1
output: ERROR: VirtualMachineSnapshot 'test-snapshot' not found

      - ToolsUsed: Required tool not called: server=kubernetes, tool=, pattern=.*
      - MinToolCalls: Too few tool calls: expected >= 1, got 0
  ✗ troubleshoot-vm (assertions: 3/3)
      verification failed: verify[0] failed: script execution failed: exit status 1
output: === Verification: Checking if agent fixed the VM ===
✓ VirtualMachine broken-vm exists
✗ Secret vm-cloud-init was not created - agent did not fix the issue

  ✗ update-vm-resources (assertions: 1/3)
      verification failed: verify[0] failed: script execution failed: exit status 1
output: virtualmachine.kubevirt.io/test-vm-update condition met
VirtualMachine test-vm-update created successfully
✗ Expected 2 CPU cores, found: 1

      - ToolsUsed: Required tool not called: server=kubernetes, tool=, pattern=.*
      - MinToolCalls: Too few tool calls: expected >= 1, got 0

Tasks:      6/12 passed (50.00%)
Assertions: 27/33 passed (81.82%)
Agent used tokens:
  Input:  48283 tokens
  Output: 20761 tokens

[lyarwood]

@manusa could you run /run-mcpchecker kubevirt on the open toolset/kubevirt PRs so we can gather more eval data?

• feat(kubevirt): add windows-golden-image prompt for automated Windows golden image creation #1033 - feat(kubevirt): add windows-golden-image prompt
• feat(core): add dry-run mode for ResourcesCreateOrUpdate #1026 - feat(core): add dry-run mode for ResourcesCreateOrUpdate
• feat(kubevirt): add run policy support to VM lifecycle management #1012 - feat(kubevirt): add run policy support to VM lifecycle management
• feat(kubevirt): add tool for QEMU guest agent access #811 - feat(kubevirt): add tool for QEMU guest agent access

The last kubevirt eval run (Mar 30 on #814) showed 7/13 tasks passing (53.8%). Running against these PRs would help us understand the current state of the toolset.

[Cali0707]

@lyarwood I've kicked those runs off - let's see what happens

[lyarwood]

KubeVirt Eval Results Update (Apr 15, 2026)

@Cali0707 triggered /run-mcpchecker kubevirt on all 4 open toolset/kubevirt PRs today. Here are the results across all runs (all using GPT-5):

Per-PR Results

PR	Title	Tasks Passed	Pass Rate
#1033	windows-golden-image prompt	6/14	42.9%
#1026	dry-run mode	5/12	41.7%
#1012	run policy support	5/12	41.7%
#811	QEMU guest agent tool	7/15	46.7%

All are well below the 80% threshold.

Per-Task Breakdown (across all 4 runs)

Task	#1033	#1026	#1012	#811	Pass Rate
clone-vm	✓	✓	✓	✓	4/4
create-basic-vm	✗	✗	✓	✓	2/4
create-ubuntu-vm	✓	✓	✓	✗	3/4
create-vm-with-instancetype	✓	✓	✓	✓	4/4
create-vm-with-size	✓	✓	✗	✗	2/4
create-vm-with-vlan	✗	✗	✗	✗	0/4
delete-vm	✗	✗	✗	✗	0/4
pause-vm	✗	✗	✗	✗	0/4
restore-vm	✗	✗	✗	✗	0/4
snapshot-vm	✗	✗	✗	✓	1/4
troubleshoot-vm	✓	✓	✓	✗	3/4
update-vm-resources	✗	✗	✗	--	0/3

New tasks from PRs:

• feat(kubevirt): add windows-golden-image prompt for automated Windows golden image creation #1033: windows-golden-image-no-eula ✓, windows-golden-image-success ✗
• feat(kubevirt): add tool for QEMU guest agent access #811: get-vm-filesystems ✓, get-vm-ip-address ✓, list-vm-users ✓, get-vm-os-info ✗, vm-guest-info ✗ (not shown in table -- only ran on feat(kubevirt): add tool for QEMU guest agent access #811)

Failure Categories

Infra/setup issues (always fail -- not tool bugs):

• delete-vm, pause-vm, restore-vm -- VM setup timeouts. VMs can't start in Kind (no KVM), so any task requiring a running VMI will fail during setup.
• create-vm-with-vlan -- NetworkAttachmentDefinition not created by agent (Multus infra limitation in CI).

Flaky tasks (non-deterministic agent behavior):

• create-basic-vm (2/4), create-vm-with-size (2/4), snapshot-vm (1/4) -- these pass or fail depending on the agent's choices.

Consistently failing tool tasks:

• update-vm-resources -- verification always fails (0/3).

Also tested locally with Claude Sonnet 4.6 (Vertex AI)

Ran evals locally against a Kind+KubeVirt cluster with podman: 5/12 passed (41.7%). Same infra failures (delete/pause/restore/vlan), plus create-basic-vm used DataVolume instead of containerDisk.

Historical Trend

Date	Model	Passed	Rate
Mar 2	GPT-5	4/7	57.1%
Mar 11	GPT-5	6/13	46.2%
Mar 30	GPT-5	7/13	53.8%
Apr 15 (CI avg)	GPT-5	~5.75/12	~43%
Apr 15 (local)	Claude Sonnet 4.6	5/12	41.7%

Note: Report workflow bug

All 4 report workflow runs failed with an awk syntax error (task_pass_rate was empty), so no results were posted as PR comments. The mcpchecker action outputs may not be propagating correctly to the report job.

CI Run Links

• PR #1033 run
• PR #1026 run
• PR #1012 run
• PR #811 run

[Cali0707]

@lyarwood the awk error is due to a breaking change in mcpchecker - unfortunately the action in this repo is pulling in the nightly build of the CLI, so we need to bump the action version. Outside of that, do we maybe want to skip the tasks that fail on kind in this action somehow?

[lyarwood]

@Cali0707 thanks for kicking those off and for the context on the awk error.

After digging into the CI logs, the root cause for most of the failures is actually a namespace race condition rather than Kind/KVM limitations. All kubevirt eval tasks shared the same vm-test namespace. When tasks run sequentially, the previous task's cleanup deletes the namespace asynchronously, and the next task's setup creates resources in a namespace that's still Terminating. The agent then finds the Terminating namespace, recreates it, and the setup resources (like the NetworkAttachmentDefinition for create-vm-with-vlan) are lost.

Specific findings:

• create-vm-with-vlan (0/4): The agent's own output confirms it: "Namespace vm-test was terminating; I recreated it" -- which destroyed the NAD that setup created. The VM itself was created with correct Multus config every time.
• update-vm-resources (0/3): Same issue -- "VirtualMachine test-vm-update not found in namespace vm-test" because the namespace was recreated. This task actually passed on the March 30 run when the namespace wasn't in a Terminating state.
• create-basic-vm, snapshot-vm flakiness: likely the same race, depending on timing.
• delete-vm, restore-vm, snapshot-vm: setup VMs either had runStrategy: Always (unnecessary -- you don't need a running VM to delete/snapshot it) or were missing volumes entirely.
• pause-vm: genuinely needs a running VM, but was using opensuse-leap with only a 30s boot timeout.

Working on a PR with two fixes:

1. Give each task its own namespace (vm-test-clone, vm-test-vlan, vm-test-delete, etc.) to eliminate the race
2. Fix task setups: use runStrategy: Halted where a running VM isn't needed, add missing volumes to snapshot/restore VMs, switch pause-vm to fedora with a longer timeout, and enable useEmulation in the kubevirt-install target

Re: skipping tasks that fail on Kind -- CI does actually have KVM (troubleshoot-vm boots a VM successfully), so with the emulation fallback and the setup fixes we shouldn't need to skip anything. The only task that truly requires a running VMI is pause-vm.

[Cali0707]

Thanks for the deep dive on this @lyarwood - feel free to ping me with whatever you need to get these evals passing more :)