eval(kubevirt): add vm_troubleshoot tool evals for

missing StorageClass and cloud-init crashloop

Adds two eval tasks that exercise the vm_troubleshoot
tool's heuristic issue detection:

1. troubleshoot-vm-missing-storageclass: Creates a VM
   referencing a non-existent StorageClass. Verifies
   the agent calls vm_troubleshoot and reports the
   CRITICAL missing SC finding with alternatives.

2. troubleshoot-vm-cloudinit-shutdown: Creates a VM
   with "shutdown -h now" in cloud-init runcmd that
   causes CrashLoopBackOff. Verifies the agent
   identifies the dangerous command as root cause.

Both scenarios are deterministic and non-flaky on any
cluster with KubeVirt installed.

Signed-off-by: Shirly Radco <sradco@redhat.com>
Co-authored-by: AI Assistant <noreply@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

Author: 3 people

evals/tasks/kubevirt/troubleshoot-vm-cloudinit-shutdown/task.yaml

@@ -0,0 +1,100 @@
+kind: Task
+apiVersion: mcpchecker/v1alpha2
+metadata:
+  labels:
+    suite: kubevirt
+    requires: kubevirt
+  name: "troubleshoot-vm-cloudinit-shutdown"
+  difficulty: medium
+  description: "Use the vm_troubleshoot tool to diagnose a VM crashlooping due to a shutdown command in cloud-init"
+spec:
+  requires:
+    - extension: kubernetes
+      as: k8s
+  setup:
+    - k8s.delete:
+        apiVersion: v1
+        kind: Namespace
+        metadata:
+          name: vm-test-troubleshoot-ci
+        ignoreNotFound: true
+    - k8s.create:
+        apiVersion: v1
+        kind: Namespace
+        metadata:
+          name: vm-test-troubleshoot-ci
+    - k8s.create:
+        apiVersion: kubevirt.io/v1
+        kind: VirtualMachine
+        metadata:
+          name: crashloop-vm
+          namespace: vm-test-troubleshoot-ci
+        spec:
+          runStrategy: Always
+          template:
+            spec:
+              domain:
+                devices:
+                  disks:
+                    - name: containerdisk
+                      disk:
+                        bus: virtio
+                    - name: cloudinit
+                      disk:
+                        bus: virtio
+                resources:
+                  requests:
+                    memory: 2Gi
+              terminationGracePeriodSeconds: 0
+              volumes:
+                - name: containerdisk
+                  containerDisk:
+                    image: quay.io/containerdisks/fedora:latest
+                - name: cloudinit
+                  cloudInitNoCloud:
+                    userData: |
+                      #cloud-config
+                      runcmd:
+                        - shutdown -h now
+    - script:
+        inline: |-
+          #!/usr/bin/env bash
+          echo "Waiting for VM to enter CrashLoopBackOff state..."
+          sleep 90
+  verify:
+    - script:
+        inline: |-
+          #!/usr/bin/env bash
+          NS="vm-test-troubleshoot-ci"
+
+          echo "=== Verification: Agent used vm_troubleshoot and identified cloud-init issue ==="
+
+          if ! kubectl get virtualmachine crashloop-vm -n "$NS" > /dev/null 2>&1; then
+              echo "✗ VirtualMachine crashloop-vm no longer exists"
+              exit 1
+          fi
+          echo "✓ VirtualMachine crashloop-vm still exists"
+          exit 0
+    - llmJudge:
+        contains: "shutdown"
+  cleanup:
+    - k8s.delete:
+        apiVersion: kubevirt.io/v1
+        kind: VirtualMachine
+        metadata:
+          name: crashloop-vm
+          namespace: vm-test-troubleshoot-ci
+        ignoreNotFound: true
+    - k8s.delete:
+        apiVersion: v1
+        kind: Namespace
+        metadata:
+          name: vm-test-troubleshoot-ci
+        ignoreNotFound: true
+  prompt:
+    inline: |-
+      A VirtualMachine named "crashloop-vm" in the vm-test-troubleshoot-ci namespace keeps restarting.
+      Use the vm_troubleshoot tool to diagnose why this VM is crashlooping. Report:
+      - The root cause of the crashloop
+      - Which cloud-init command is causing the problem
+      - How to fix it

evals/tasks/kubevirt/troubleshoot-vm-missing-storageclass/task.yaml

@@ -0,0 +1,105 @@
+kind: Task
+apiVersion: mcpchecker/v1alpha2
+metadata:
+  labels:
+    suite: kubevirt
+    requires: kubevirt
+  name: "troubleshoot-vm-missing-storageclass"
+  difficulty: medium
+  description: "Use the vm_troubleshoot tool to diagnose a VM stuck in Provisioning due to a non-existent StorageClass"
+spec:
+  requires:
+    - extension: kubernetes
+      as: k8s
+  setup:
+    - k8s.delete:
+        apiVersion: v1
+        kind: Namespace
+        metadata:
+          name: vm-test-troubleshoot-sc
+        ignoreNotFound: true
+    - k8s.create:
+        apiVersion: v1
+        kind: Namespace
+        metadata:
+          name: vm-test-troubleshoot-sc
+    - k8s.create:
+        apiVersion: kubevirt.io/v1
+        kind: VirtualMachine
+        metadata:
+          name: broken-sc-vm
+          namespace: vm-test-troubleshoot-sc
+        spec:
+          runStrategy: Always
+          dataVolumeTemplates:
+            - metadata:
+                name: broken-sc-vm-rootdisk
+              spec:
+                storage:
+                  storageClassName: non-existent-sc-xyz
+                  accessModes:
+                    - ReadWriteOnce
+                  resources:
+                    requests:
+                      storage: 10Gi
+                source:
+                  registry:
+                    url: docker://quay.io/containerdisks/fedora:latest
+          template:
+            spec:
+              domain:
+                devices:
+                  disks:
+                    - name: rootdisk
+                      disk:
+                        bus: virtio
+                resources:
+                  requests:
+                    memory: 2Gi
+              terminationGracePeriodSeconds: 0
+              volumes:
+                - name: rootdisk
+                  dataVolume:
+                    name: broken-sc-vm-rootdisk
+    - script:
+        inline: |-
+          #!/usr/bin/env bash
+          echo "Waiting for VM to enter Provisioning state..."
+          sleep 15
+  verify:
+    - script:
+        inline: |-
+          #!/usr/bin/env bash
+          NS="vm-test-troubleshoot-sc"
+
+          echo "=== Verification: Agent used vm_troubleshoot and identified missing StorageClass ==="
+
+          if ! kubectl get virtualmachine broken-sc-vm -n "$NS" > /dev/null 2>&1; then
+              echo "✗ VirtualMachine broken-sc-vm no longer exists"
+              exit 1
+          fi
+          echo "✓ VirtualMachine broken-sc-vm still exists"
+          exit 0
+    - llmJudge:
+        contains: "non-existent-sc-xyz"
+  cleanup:
+    - k8s.delete:
+        apiVersion: kubevirt.io/v1
+        kind: VirtualMachine
+        metadata:
+          name: broken-sc-vm
+          namespace: vm-test-troubleshoot-sc
+        ignoreNotFound: true
+    - k8s.delete:
+        apiVersion: v1
+        kind: Namespace
+        metadata:
+          name: vm-test-troubleshoot-sc
+        ignoreNotFound: true
+  prompt:
+    inline: |-
+      A VirtualMachine named "broken-sc-vm" in the vm-test-troubleshoot-sc namespace is not starting.
+      Use the vm_troubleshoot tool to diagnose why this VM is stuck. Report:
+      - The root cause of the issue
+      - What StorageClass is missing
+      - What alternative StorageClasses are available on the cluster