feat(kubevirt): add tool for QEMU guest agent access#811
Conversation
lyarwood
left a comment
lyarwood
left a comment
There was a problem hiding this comment.
Lets start with the evals, the use cases here are far too high level and model dependant for now. Can you simplify these and break them out into their own commits before any tooling is introduced.
codingben
force-pushed
the
add-guest-info
branch
from
cf6615c to
6d2456a
Compare
codingben
force-pushed
the
add-guest-info
branch
from
6d2456a to
680a27a
Compare
codingben
force-pushed
the
add-guest-info
branch
2 times, most recently
from
10eb79b to
07aa5a4
Compare
|
/run-mcpchecker kubevirt |
| func getGuestOSInfo(ctx context.Context, dynamicClient dynamic.Interface, namespace, name string) (map[string]any, error) { | ||
| result, err := dynamicClient.Resource(vmiSubresourcesGVR). | ||
| Namespace(namespace). | ||
| Get(ctx, name+"/guestosinfo", metav1.GetOptions{}) |
There was a problem hiding this comment.
The subresources.kubevirt.io API group is blocked by the AccessControlRoundTripper because it is not discoverable via standard REST mapper (see #1061 for the same issue with pause/unpause). This path hack (name+"/guestosinfo") also relies on the dynamic client not URL-encoding the slash. Has this been tested against a real KubeVirt cluster?
There was a problem hiding this comment.
I see and will take a deeper look. I haven't tested it against a real cluster yet.
Did it work? I don't see results from this job. |
See https://github.com/containers/kubernetes-mcp-server/actions/runs/24468624290/job/71502564513 |
codingben
force-pushed
the
add-guest-info
branch
5 times, most recently
from
7287232 to
f37c6f1
Compare
| - script: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| set -e | ||
| if ! kubectl get vm test-vm -n vm-test-os-info &>/dev/null; then | ||
| echo "ERROR: VirtualMachine 'test-vm' not found" | ||
| exit 1 | ||
| fi | ||
| echo "Verification passed: VM exists" | ||
| exit 0 |
There was a problem hiding this comment.
We spoke about this briefly a few days ago but the verify step here only checks that the VM exists, not that the agent actually retrieved OS info from the guest agent. The same issue applies to the other eval tasks (get-vm-filesystems, get-vm-ip-address, list-vm-users).
You could add an llmJudge verify step to check the agent's response semantically. mcpchecker supports contains and exact fields on llmJudge, for example:
verify:
- script:
inline: |-
# existing VM existence check
- llmJudge:
contains: "Fedora"This would verify the agent's response mentions the expected OS name. Similar patterns for the other tasks:
get-vm-filesystems:contains: "/"get-vm-ip-address:contains: "10."list-vm-users:contains: "admin"
Note: llmJudge does a semantic comparison via a judge LLM, not a literal substring match, so it's fairly flexible. However it does not support custom evaluation criteria or rubrics - only contains and exact reference matching.
There was a problem hiding this comment.
Interesting. I've added it, can you please review here this revision?
codingben
force-pushed
the
add-guest-info
branch
3 times, most recently
from
5fdfeb0 to
c307c36
Compare
|
Can we retrigger the Ubuntu 24.04-arm job? I'm not sure what caused the failure there. |
|
@codingben soemtimes we have flakes on those, I imagine this is a similar case - rerunning it now anyways |
I see that the CI passed now. Can you please run MCP checker on this PR? |
|
/run-mcpchecker kubevirt |
mcpchecker MCP Evaluation ResultsCommit:
|
|
Hey, how we can proceed here? Can we merge it? |
Can you rebase this branch on main to pick up #1054, that should fix the passing rate and allow us to move forward here. |
Add five simple evaluation tasks that test basic vm_guest_info functionality: - get-vm-filesystems: Get filesystem information from inside a VM - get-vm-ip-address: Get IP address from inside the guest OS - get-vm-os-info: Get OS name and version from a VM - list-vm-users: List currently logged-in users in a VM - vm-guest-info: General test for retrieving guest agent information These evals focus on testing the basic building blocks rather than high-level use cases, making them more reliable and model-independent. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Ben Oukhanov <boukhanov@redhat.com>
|
@lyarwood Thank you! Just rebased it. Can you please run the MCP checker again? |
Add new vm_guest_info tool that enables querying information from inside VirtualMachines using the QEMU guest agent, without requiring SSH access or credentials. The tool supports querying: - os: Operating system information (name, version, kernel, hostname) - filesystem: Mounted filesystems and disk usage - network: Network interfaces and IP addresses - users: Currently logged-in users and sessions This provides a secure way to gather runtime information from VMs for monitoring, troubleshooting, and compliance purposes. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Ben Oukhanov <boukhanov@redhat.com>
|
@manusa would you mind running |
|
/run-mcpchecker kubevirt |
6 participants
Implements vm_guest_info tool to retrieve information from inside running VMs via QEMU guest agent without requiring SSH credentials.
Assisted-By: Claude noreply@anthropic.com