-
Notifications
You must be signed in to change notification settings - Fork 390
feat(kubevirt): add vm_troubleshoot tool for automated VM diagnostics #1239
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| kind: Task | ||
| apiVersion: mcpchecker/v1alpha2 | ||
| metadata: | ||
| labels: | ||
| suite: kubevirt | ||
| requires: kubevirt | ||
| name: "troubleshoot-vm-cloudinit-shutdown" | ||
| difficulty: medium | ||
| description: "Diagnose a VM crashlooping due to a shutdown command in cloud-init" | ||
| spec: | ||
| requires: | ||
| - extension: kubernetes | ||
| as: k8s | ||
| setup: | ||
| - k8s.delete: | ||
| apiVersion: v1 | ||
| kind: Namespace | ||
| metadata: | ||
| name: vm-test-troubleshoot-ci | ||
| ignoreNotFound: true | ||
| - k8s.create: | ||
| apiVersion: v1 | ||
| kind: Namespace | ||
| metadata: | ||
| name: vm-test-troubleshoot-ci | ||
| - k8s.create: | ||
| apiVersion: kubevirt.io/v1 | ||
| kind: VirtualMachine | ||
| metadata: | ||
| name: crashloop-vm | ||
| namespace: vm-test-troubleshoot-ci | ||
| spec: | ||
| runStrategy: Always | ||
| template: | ||
| spec: | ||
| domain: | ||
| devices: | ||
| disks: | ||
| - name: containerdisk | ||
| disk: | ||
| bus: virtio | ||
| - name: cloudinit | ||
| disk: | ||
| bus: virtio | ||
| resources: | ||
| requests: | ||
| memory: 2Gi | ||
| terminationGracePeriodSeconds: 0 | ||
| volumes: | ||
| - name: containerdisk | ||
| containerDisk: | ||
| image: quay.io/containerdisks/fedora:latest | ||
| - name: cloudinit | ||
| cloudInitNoCloud: | ||
| userData: | | ||
| #cloud-config | ||
| runcmd: | ||
| - shutdown -h now | ||
| - script: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| echo "Waiting for VM to attempt boot and crash..." | ||
| sleep 60 | ||
| verify: | ||
| - script: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| NS="vm-test-troubleshoot-ci" | ||
|
|
||
| echo "=== Verification: Agent identified cloud-init issue ===" | ||
|
|
||
| if ! kubectl get virtualmachine crashloop-vm -n "$NS" > /dev/null 2>&1; then | ||
| echo "✗ VirtualMachine crashloop-vm no longer exists" | ||
| exit 1 | ||
| fi | ||
| echo "✓ VirtualMachine crashloop-vm still exists" | ||
| exit 0 | ||
| - llmJudge: | ||
| contains: "shutdown -h now" | ||
| cleanup: | ||
| - k8s.delete: | ||
| apiVersion: kubevirt.io/v1 | ||
| kind: VirtualMachine | ||
| metadata: | ||
| name: crashloop-vm | ||
| namespace: vm-test-troubleshoot-ci | ||
| ignoreNotFound: true | ||
| - k8s.delete: | ||
| apiVersion: v1 | ||
| kind: Namespace | ||
| metadata: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The prompt explicitly names the tool to use ( Consider rephrasing to present the problem and let the agent figure out which tools to use, e.g.:
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fixed |
||
| name: vm-test-troubleshoot-ci | ||
| ignoreNotFound: true | ||
| prompt: | ||
| inline: |- | ||
| A VirtualMachine named "crashloop-vm" in the vm-test-troubleshoot-ci namespace keeps restarting. | ||
| Diagnose why this VM is crashlooping. Report: | ||
| - The root cause of the crashloop | ||
| - Which cloud-init command is causing the problem | ||
| - How to fix it | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| kind: Task | ||
| apiVersion: mcpchecker/v1alpha2 | ||
| metadata: | ||
| labels: | ||
| suite: kubevirt | ||
| requires: kubevirt | ||
| name: "troubleshoot-vm-missing-storageclass" | ||
| difficulty: medium | ||
| description: "Diagnose a VM stuck in Provisioning due to a non-existent StorageClass" | ||
| spec: | ||
| requires: | ||
| - extension: kubernetes | ||
| as: k8s | ||
| setup: | ||
| - k8s.delete: | ||
| apiVersion: v1 | ||
| kind: Namespace | ||
| metadata: | ||
| name: vm-test-troubleshoot-sc | ||
| ignoreNotFound: true | ||
| - k8s.create: | ||
| apiVersion: v1 | ||
| kind: Namespace | ||
| metadata: | ||
| name: vm-test-troubleshoot-sc | ||
| - k8s.create: | ||
| apiVersion: kubevirt.io/v1 | ||
| kind: VirtualMachine | ||
| metadata: | ||
| name: broken-sc-vm | ||
| namespace: vm-test-troubleshoot-sc | ||
| spec: | ||
| runStrategy: Always | ||
| dataVolumeTemplates: | ||
| - metadata: | ||
| name: broken-sc-vm-rootdisk | ||
| spec: | ||
| storage: | ||
| storageClassName: non-existent-sc-xyz | ||
| accessModes: | ||
| - ReadWriteOnce | ||
| resources: | ||
| requests: | ||
| storage: 10Gi | ||
| source: | ||
| registry: | ||
| url: docker://quay.io/containerdisks/fedora:latest | ||
| template: | ||
| spec: | ||
| domain: | ||
| devices: | ||
| disks: | ||
| - name: rootdisk | ||
| disk: | ||
| bus: virtio | ||
| resources: | ||
| requests: | ||
| memory: 2Gi | ||
| terminationGracePeriodSeconds: 0 | ||
| volumes: | ||
| - name: rootdisk | ||
| dataVolume: | ||
| name: broken-sc-vm-rootdisk | ||
| - script: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| echo "Waiting for VM to enter Provisioning state..." | ||
| sleep 15 | ||
| verify: | ||
| - script: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| NS="vm-test-troubleshoot-sc" | ||
|
|
||
| echo "=== Verification: Agent identified missing StorageClass ===" | ||
|
|
||
| if ! kubectl get virtualmachine broken-sc-vm -n "$NS" > /dev/null 2>&1; then | ||
| echo "✗ VirtualMachine broken-sc-vm no longer exists" | ||
| exit 1 | ||
| fi | ||
| echo "✓ VirtualMachine broken-sc-vm still exists" | ||
| exit 0 | ||
| - llmJudge: | ||
| contains: "non-existent-sc-xyz" | ||
| cleanup: | ||
| - k8s.delete: | ||
| apiVersion: kubevirt.io/v1 | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same comment as the cloud-init task — the prompt explicitly names
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fixed |
||
| kind: VirtualMachine | ||
| metadata: | ||
| name: broken-sc-vm | ||
| namespace: vm-test-troubleshoot-sc | ||
| ignoreNotFound: true | ||
| - k8s.delete: | ||
| apiVersion: v1 | ||
| kind: Namespace | ||
| metadata: | ||
| name: vm-test-troubleshoot-sc | ||
| ignoreNotFound: true | ||
| prompt: | ||
| inline: |- | ||
| A VirtualMachine named "broken-sc-vm" in the vm-test-troubleshoot-sc namespace is not starting. | ||
| Diagnose why this VM is stuck. Report: | ||
| - The root cause of the issue | ||
| - What StorageClass is missing | ||
| - What alternative StorageClasses are available on the cluster | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit:
contains: "shutdown"is a fairly weak check — the agent could pass by saying something generic like "the VM appears to shut down" without actually diagnosing the cloud-init cause.Consider tightening to
contains: "shutdown -h now"or adding a secondcontainsfor"cloud-init"to confirm the agent identified the specific command and its source.For comparison, the missing-storageclass task's
contains: "non-existent-sc-xyz"is much stronger since it requires the exact StorageClass name.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed