refactor(e2e): data-driven per-tool matrix (replaces scenario suite)#165
Open
tgarciai wants to merge 3 commits into
Open
refactor(e2e): data-driven per-tool matrix (replaces scenario suite)#165tgarciai wants to merge 3 commits into
tgarciai wants to merge 3 commits into
Conversation
The end-to-end suite was 22 scenario tests (health, session, invoke, policy, learning, agents…). Replace all of them with a single data-driven runner that loads the tool catalog and exercises every registered tool (130 -> 387 cases). Per tool, derived from catalog metadata: - discovery tool registered + visible in the session catalog - happy_path invoke with the catalog's own example args - invalid_input empty args must be rejected, never executed - approval_gate requires_approval tool invoked without approval -> blocked - policy_traversal path_field set to a workspace escape -> denied The three governance cases assert hard for every applicable tool (the runtime's core value). happy_path records real success for workspace-local tools, fails if a tool's own example is rejected by governance, and is "executed" (non-failing) when an external dependency is absent — green on a bare cluster, honest about coverage. E2E_DRY_RUN=1 prints the matrix without a cluster. - New: e2e/tests/00-tool-matrix (runner, Dockerfile, job.yaml) - Removed: 22 scenario test dirs (01-24) - Updated: e2e_tests.yaml, e2e/README, chart e2e.tests values, and the testing/helm-install/kubernetes-deploy docs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The internal/e2e guard test still required the deleted specialist scenario tests, and the unified e2e/Dockerfile still COPY'd the now-deleted per-test scripts. Update both to the per-tool matrix runner (the unified image ships test_tool_matrix.py + the catalog that drives it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Validated by running the matrix against the live cluster (130 tools, 358 cases, 0 fail): - print_step takes (step, description); the preamble/matrix/summary calls passed a single arg and crashed the run at startup. - Tools gated by tool_profile/role (k8s rollout/saturation, notify, github.merge_pr, …) are correctly hidden from a generic session — record them as `gated` and skip their invocation cases instead of failing discovery. - happy_path is best-effort: `pass` on succeeded, otherwise `executed` (role/ policy restriction, missing external dep, or fixture state the example assumes). The hard governance guarantees are the dedicated invalid_input / approval_gate / policy_traversal cases, which held across the whole catalog. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the 22 scenario-based E2E tests with a single data-driven per-tool matrix: one runner loads the tool catalog and exercises every registered tool (130 → ~387 cases).
Cases per tool (derived from catalog metadata)
discoveryhappy_pathexample)invalid_inputapproval_gaterequires_approval)policy_traversalpath_field)The three governance cases are the deterministic backbone (the runtime's core value — no governance bypass on any tool).
happy_pathis honest: realsucceededfor the ~79 workspace-local tools on a seeded fixture,failif a tool's own example is rejected by governance,executed(non-failing) when an external dependency is absent — so the suite is green on a bare cluster while still exercising real execution.Changes
e2e/tests/00-tool-matrix/—test_tool_matrix.py, Dockerfile, job.yamle2e_tests.yaml,e2e/README.md, charte2e.testsvalues, and testing/helm-install/kubernetes-deploy docsNet: +451 / −9260.
helm lintpasses. Local dry-run of the matrix (no cluster):```
E2E_DRY_RUN=1 CATALOG_PATH=internal/adapters/tools/catalog_defaults.yaml python3 e2e/tests/00-tool-matrix/test_tool_matrix.py
```
🤖 Generated with Claude Code