test(evals): add update-vm-instancetype eval task (#814)

* feat(evals): add update-vm-instancetype eval task

Add a new kubevirt eval that tests doubling VM memory by changing
instance types. Includes a verify_instancetype_changed helper function.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Lee Yarwood <lyarwood@redhat.com>

* fix(evals): isolate update-vm-instancetype namespace and reuse helpers

Bring the task in line with the suite-wide namespace-race fix from #1054:
every kubevirt eval task uses a unique namespace so a previous task's
asynchronous namespace teardown can't delete the next task's setup
resources. This task still used the shared `vm-test` namespace and failed
in CI with vm-test stuck Terminating and fedora-vm absent when the agent
ran.

- Use a dedicated vm-test-instancetype-update namespace in setup, the VM
  spec, verify, cleanup, and the prompt.
- Replace the hand-rolled verify script with the shared verify-vm.sh
  helpers (verify_vm_exists, verify_instancetype, verify_no_direct_resources),
  matching update-vm-resources and create-vm-with-instancetype.
- Add the task to the kubevirt task catalog README.

Signed-off-by: Marc Nuri <marc@marcnuri.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Signed-off-by: Lee Yarwood <lyarwood@redhat.com>
Signed-off-by: Marc Nuri <marc@marcnuri.com>
Co-authored-by: Marc Nuri <marc@marcnuri.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: 3 people

evals/tasks/kubevirt/README.md

@@ -49,6 +49,9 @@ KubeVirt-focused MCP tasks live here. Each folder under this directory represent
 - **[hard] update-vm-resources** - Update VM CPU and memory resources
   - **Prompt:** *A VirtualMachine named test-vm-update exists in the vm-test namespace. It currently has 1 vCPU and 2Gi of memory. Please update the VirtualMachine to add an additional vCPU (making it 2 vCPUs total) and increase the memory to at least 3Gi.*
 
+- **[hard] update-vm-instancetype** - Double a VM's memory by switching its instancetype
+  - **Prompt:** *A Fedora VirtualMachine named fedora-vm exists in the vm-test-instancetype-update namespace using the u1.small instancetype. Double its memory by updating the instance type.*
+
 ### VM Troubleshooting
 
 - **[hard] troubleshoot-vm** - Use the vm-troubleshoot prompt to diagnose VirtualMachine issues

evals/tasks/kubevirt/update-vm-instancetype/task.yaml

@@ -0,0 +1,77 @@
+kind: Task
+apiVersion: mcpchecker/v1alpha2
+metadata:
+  labels:
+    suite: kubevirt
+    requires: kubevirt
+  name: "update-vm-instancetype"
+  difficulty: hard
+spec:
+  requires:
+    - extension: kubernetes
+      as: k8s
+  setup:
+    - k8s.delete:
+        apiVersion: v1
+        kind: Namespace
+        metadata:
+          name: vm-test-instancetype-update
+        ignoreNotFound: true
+    - k8s.create:
+        apiVersion: v1
+        kind: Namespace
+        metadata:
+          name: vm-test-instancetype-update
+    - k8s.create:
+        apiVersion: kubevirt.io/v1
+        kind: VirtualMachine
+        metadata:
+          name: fedora-vm
+          namespace: vm-test-instancetype-update
+        spec:
+          runStrategy: Halted
+          instancetype:
+            name: u1.small
+            kind: VirtualMachineClusterInstancetype
+          template:
+            spec:
+              domain:
+                devices:
+                  disks:
+                  - name: containerdisk
+                    disk:
+                      bus: virtio
+              volumes:
+              - name: containerdisk
+                containerDisk:
+                  image: quay.io/containerdisks/fedora:latest
+  verify:
+    - script:
+        inline: |-
+          #!/usr/bin/env bash
+          source ../helpers/verify-vm.sh
+          NS="vm-test-instancetype-update"
+
+          verify_vm_exists "fedora-vm" "$NS" || exit 1
+          verify_instancetype "fedora-vm" "$NS" "u1.medium" || exit 1
+          verify_no_direct_resources "fedora-vm" "$NS"
+
+          echo "All validations passed"
+          exit 0
+  cleanup:
+    - k8s.delete:
+        apiVersion: kubevirt.io/v1
+        kind: VirtualMachine
+        metadata:
+          name: fedora-vm
+          namespace: vm-test-instancetype-update
+        ignoreNotFound: true
+    - k8s.delete:
+        apiVersion: v1
+        kind: Namespace
+        metadata:
+          name: vm-test-instancetype-update
+        ignoreNotFound: true
+  prompt:
+    inline: |
+      A Fedora VirtualMachine named fedora-vm exists in the vm-test-instancetype-update namespace using the u1.small instancetype. Double its memory by updating the instance type.