E2E tests: quick mode and config-file mode

[yrobla]

Description

Add Go/Ginkgo E2E tests in test/e2e/ covering two top-level thv vmcp usage modes: quick mode (thv vmcp serve --group) and config-file mode (thv vmcp init → thv vmcp validate --config → thv vmcp serve --config). Each scenario also exercises basic MCP client connectivity, confirming that tools from backend workloads are reachable through the aggregated vMCP endpoint. These tests provide the first Go-based CLI E2E coverage for vMCP features and validate the end-to-end correctness of #4885 and #4886.

Context

#4885 delivered thv vmcp init (config scaffolding subcommand) and #4886 delivered zero-config quick mode (thv vmcp serve --group without --config). Both depend on the shared library from #4879 and the Cobra command wiring from #4883. This item is the integration gate: it confirms that both modes work end-to-end with real group workloads, real port binding, and a real MCP client connection. Existing vMCP E2E tests are entirely Kubernetes/chainsaw-based (test/e2e/chainsaw/operator/); this is the first Go-based CLI E2E file covering thv vmcp.

The test file follows the Go/Ginkgo pattern already established in test/e2e/ — specifically api_workload_lifecycle_test.go (lifecycle patterns), group_test.go (group + workload setup), inspector_test.go (long-running subcommand handling), and the existing helpers.go / mcp_client_helpers.go infrastructure.

Dependencies: Depends on #4885 (thv vmcp init subcommand) and #4886 (quick mode — thv vmcp serve --group)
Blocks: #4889

Acceptance Criteria

• A new file test/e2e/vmcp_cli_test.go exists with SPDX header, package e2e_test, and Ginkgo Describe("vMCP CLI", ...) block with Label("vmcp", "e2e")
• Quick mode test: creates a group, runs a backend workload in it, calls thv vmcp serve --group <name> as a background process on a random port, polls until the vMCP endpoint is ready, connects an MCP client, asserts that ListTools returns at least one tool from the backend, then stops and cleans up
• Quick mode test: asserts that the vMCP listener is bound to 127.0.0.1 only (not 0.0.0.0), consistent with the security requirement from Implement zero-config quick mode for thv vmcp serve #4886
• Config-file mode test: creates a group, runs a backend workload in it, calls thv vmcp init --group <name> --config <tmpfile> and asserts exit code 0 and that the generated YAML file is non-empty
• Config-file mode test: calls thv vmcp validate --config <tmpfile> and asserts exit code 0
• Config-file mode test: calls thv vmcp serve --config <tmpfile> as a background process on a random port, polls until the vMCP endpoint is ready, connects an MCP client, asserts that ListTools returns at least one tool from the backend, then stops and cleans up
• thv vmcp serve with neither --config nor --group exits non-zero with a descriptive error (negative test, no background process)
• thv vmcp validate --config <nonexistent> exits non-zero (negative test for validate)
• All background thv vmcp serve processes are stopped via t.Cleanup / AfterEach even on test failure; no process leaks
• Group workloads and groups are removed in AfterEach via existing StopAndRemoveMCPServer and RemoveGroup helpers
• Config temp files are created in os.MkdirTemp subdirectories and cleaned up via DeferCleanup
• Random ports are obtained via networking.FindAvailable() (or equivalent net.Listen("tcp", ":0") approach); no hardcoded ports
• All tests pass in CI on a standard Linux amd64 runner with Docker available
• All existing tests pass (no regressions)
• Code reviewed and approved

Technical Approach

Recommended Implementation

Create test/e2e/vmcp_cli_test.go. The file contains a single top-level Describe("vMCP CLI", ...) block with two Context blocks — one for quick mode, one for config-file mode. Each has BeforeEach for setup (group + workload creation) and AfterEach for teardown (workload removal, group removal, process kill).

Background process handling is the central challenge. thv vmcp serve is a long-running process; use exec.Command (not THVCommand.Run()) to start it, capture its stdout/stderr to GinkgoWriter, and track the *exec.Cmd so it can be killed in AfterEach. The StartLongRunningTHVCommand helper in helpers.go provides this pattern but writes to GinkgoWriter directly — use it or replicate its approach.

Readiness polling uses WaitForMCPServerReady from mcp_client_helpers.go. Pass the vMCP endpoint URL (e.g., http://127.0.0.1:<port>/sse) with a 60-second timeout and 2-second poll interval.

MCP client connectivity uses NewMCPClientForSSE + MCPClientHelper.Initialize + MCPClientHelper.ListTools from mcp_client_helpers.go. Assert len(tools.Tools) > 0 to confirm backend tools are aggregated.

Port allocation: use net.Listen("tcp", "127.0.0.1:0") to acquire a free port, close the listener, then pass --port <n> (or equivalent flag) to thv vmcp serve. Alternatively use networking.FindAvailable() if it is importable from the test package.

Quick mode localhost binding check: after starting thv vmcp serve --group, verify the server is not listening on 0.0.0.0 by attempting net.Dial("tcp", "0.0.0.0:<port>") or by inspecting stdout/stderr for the bind address. The simplest check is that the MCP client connects to 127.0.0.1:<port> and the process did not fail; a stricter check inspects the serve log line for 127.0.0.1.

Config-file mode flow:

1. thv vmcp init --group <name> --config <tmpfile> — run to completion (not background), assert exit 0 and non-empty file.
2. thv vmcp validate --config <tmpfile> — run to completion, assert exit 0.
3. thv vmcp serve --config <tmpfile> --port <n> — start as background process; poll readiness; connect MCP client.

Patterns & Frameworks

• Go/Ginkgo v2: Describe / Context / It / BeforeEach / AfterEach / DeferCleanup / Eventually / By — consistent with group_test.go and api_workload_lifecycle_test.go
• t.Cleanup / DeferCleanup: use DeferCleanup (Ginkgo equivalent of t.Cleanup) for process teardown and temp file removal; never defer alone in parallel tests
• THVCommand helpers: use NewTHVCommand(config, args...).ExpectSuccess() for synchronous subcommands (init, validate, group create, run); use StartLongRunningTHVCommand or equivalent for the background serve invocation
• Eventually: Eventually(func() bool { ... }, 60*time.Second, 2*time.Second).Should(BeTrue()) for readiness polling, consistent with existing E2E tests
• SPDX header: every new Go file must open with // SPDX-FileCopyrightText: Copyright 2025 Stacklok, Inc. and // SPDX-License-Identifier: Apache-2.0
• No gomock in E2E: E2E tests use real subprocesses — gomock is only for unit/integration tests

Code Pointers

• test/e2e/helpers.go — THVCommand, TestConfig, NewTestConfig, GenerateUniqueServerName, StartLongRunningTHVCommand, WaitForMCPServer, StopAndRemoveMCPServer, RemoveGroup, CreateAndTrackGroup — all reusable from the new test file
• test/e2e/mcp_client_helpers.go — NewMCPClientForSSE, MCPClientHelper.Initialize, MCPClientHelper.ListTools, WaitForMCPServerReady — MCP connectivity helpers for the readiness poll and tool listing assertion
• test/e2e/e2e_suite_test.go — Suite infrastructure: BeforeSuite/AfterSuite, sharedConfigDir, XDG_CONFIG_HOME/XDG_DATA_HOME/HOME env overrides; the new test inherits this suite
• test/e2e/group_test.go — Pattern for group creation, workload launch (thv run fetch --group), WaitForMCPServer, StopAndRemoveMCPServer, RemoveGroup in AfterEach
• test/e2e/inspector_test.go — Pattern for long-running subcommand (thv inspector) including SIGINT cleanup in AfterEach
• test/e2e/api_workload_lifecycle_test.go — Eventually(..., 60*time.Second, 2*time.Second) polling pattern, By(...) step annotations
• test/integration/vmcp/helpers/vmcp_server.go — getFreePort via net.Listen(":0") — usable as a pattern for random port selection in E2E tests
• pkg/networking/port.go — networking.FindAvailable() — production-grade random port allocator (import if allowed from test context)

Component Interfaces

// test/e2e/vmcp_cli_test.go — top-level structure

var _ = Describe("vMCP CLI", Label("vmcp", "e2e"), func() {
    var (
        config           *e2e.TestConfig
        groupName        string
        backendName      string
        createdWorkloads []string
        vMCPCmd          *exec.Cmd
        vMCPPort         int
    )

    BeforeEach(func() {
        config = e2e.NewTestConfig()
        groupName = e2e.GenerateUniqueServerName("vmcp-e2e-group")
        backendName = e2e.GenerateUniqueServerName("vmcp-e2e-backend")
        createdWorkloads = nil
        vMCPCmd = nil
        vMCPPort = allocateFreePort() // net.Listen("tcp", "127.0.0.1:0") pattern

        err := e2e.CheckTHVBinaryAvailable(config)
        Expect(err).ToNot(HaveOccurred())
    })

    AfterEach(func() {
        // Kill vMCP serve process if running
        if vMCPCmd != nil && vMCPCmd.Process != nil {
            _ = vMCPCmd.Process.Signal(syscall.SIGINT)
            _ = vMCPCmd.Wait()
        }
        // Stop and remove backend workloads
        for _, w := range createdWorkloads {
            _ = e2e.StopAndRemoveMCPServer(config, w)
        }
        // Remove the group
        _ = e2e.RemoveGroup(config, groupName)
    })

    Context("quick mode (--group, no --config)", func() {
        It("starts vMCP and exposes backend tools", func() { ... })
        It("binds to 127.0.0.1 only", func() { ... })
    })

    Context("config-file mode (init -> validate -> serve --config)", func() {
        It("init generates a valid config file", func() { ... })
        It("validate accepts the generated config", func() { ... })
        It("serve --config starts vMCP and exposes backend tools", func() { ... })
    })

    Context("error cases", func() {
        It("exits non-zero when neither --config nor --group is given", func() { ... })
        It("validate exits non-zero for a non-existent config file", func() { ... })
    })
})

// allocateFreePort returns a free TCP port on 127.0.0.1
func allocateFreePort() int {
    l, err := net.Listen("tcp", "127.0.0.1:0")
    Expect(err).ToNot(HaveOccurred())
    defer l.Close()
    return l.Addr().(*net.TCPAddr).Port
}

Testing Strategy

Unit Tests

• Not applicable for this item — E2E tests exercise real thv subprocesses; no unit tests needed here

Integration Tests (E2E — the primary deliverable)

• Quick mode happy path: group create → thv run → thv vmcp serve --group → MCP client connects → ListTools returns >= 1 tool
• Quick mode localhost binding: vMCP serve starts and listens on 127.0.0.1:<port> (not 0.0.0.0)
• Config-file init: thv vmcp init --group <name> --config <file> exits 0 and writes a non-empty YAML file
• Config-file validate: thv vmcp validate --config <file> exits 0 for the file generated by init
• Config-file serve: thv vmcp serve --config <file> → MCP client connects → ListTools returns >= 1 tool
• Error — no flags: thv vmcp serve (no --config, no --group) exits non-zero with a descriptive error message
• Error — bad config path: thv vmcp validate --config /nonexistent/path.yaml exits non-zero

Edge Cases

• thv vmcp serve background process is cleaned up in AfterEach even if the It block panics or the assertion fails — use DeferCleanup before starting the process
• If the backend workload fails to reach "running" within the timeout, WaitForMCPServer returns an error that causes the test to fail with a clear message rather than a timeout panic
• The generated config temp file is placed in an os.MkdirTemp directory (not /tmp directly) to avoid cross-test collisions when running in parallel

Out of Scope

• Optimizer tests (--optimizer, --optimizer-embedding) — those are E2E tests: optimizer tiers and regression #4889
• TEI container lifecycle E2E tests — those are E2E tests: optimizer tiers and regression #4889
• Regression test for standalone vmcp serve post-refactor — that is E2E tests: optimizer tiers and regression #4889
• thv vmcp status command — explicitly deferred (out of scope for this plan)
• Kubernetes/chainsaw vMCP E2E tests — those remain unchanged under test/e2e/chainsaw/operator/
• MCP protocol conformance testing beyond basic ListTools connectivity
• ARM64 / Apple Silicon platform-specific test adjustments (CI runs on amd64 Linux)

References

• RFC THV-0059 — Section on test scenarios for quick mode and config-file mode
• GitHub Issue #4808 — Parent tracking issue
• test/e2e/helpers.go — THVCommand, StartLongRunningTHVCommand, group/workload helpers
• test/e2e/mcp_client_helpers.go — NewMCPClientForSSE, WaitForMCPServerReady, MCPClientHelper
• test/e2e/group_test.go — Group + workload setup/teardown pattern
• test/e2e/inspector_test.go — Long-running subcommand (background process) pattern
• test/e2e/api_workload_lifecycle_test.go — Eventually polling and By step annotations
• test/integration/vmcp/helpers/vmcp_server.go — getFreePort pattern for random port allocation
• .claude/rules/testing.md — E2E test strategy, Ginkgo/Gomega patterns, t.Cleanup / DeferCleanup usage, parallel test safety
• .claude/rules/go-style.md — SPDX headers, error handling conventions