Implementation Plan: T5-P1-A1-WP2 Hook Deny Efficacy Probe Harness#4131
Merged
Trecek merged 6 commits intoJun 27, 2026
Merged
Conversation
…d xdist-safe matrix serialization Adds the subdirectory conftest for the deny-mechanism probe harness: - TOOL_CLASSES (16 entries), SESSION_MODES, BACKENDS shared constants - EXPECTED_TOTAL_PROBE_COUNT and EXPECTED_NON_INERT_COMBINATIONS derived from HOOK_REGISTRY at import time - record_probe_row() worker-side accumulator - workeroutput IPC under key 'hook_probe_rows' (distinct from root conftest's filter-stats keys) - Controller-side pytest_testnodedown and pytest_sessionfinish serialize the matrix to .autoskillit/temp/hook_deny_strength_matrix.json Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… harness Adds TestHookDenyEfficacyProbe with parametrization over the full (tool_class x session_mode x backend) matrix for every PreToolUse+deny hook script in HOOK_REGISTRY. Each probe: - Skips inert codex rows after verifying the script is excluded from generate_codex_hooks_config() output - Invokes the guard as a subprocess via sys.executable with a clean env - Classifies the outcome (deny -> hard, else -> soft) - Records to the probe matrix via record_probe_row Includes TOOL_CLASS_PAYLOADS (matching TOOL_CLASSES.keys() exactly), SESSION_MODE_ENV / BACKEND_ENV env override dicts, helper functions (_clean_env, _invoke_guard, _build_payload, _collect_probe_params), and the standalone test_probe_collection_count meta-test that guards against silent drift between the registry and the expected matrix size. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds three meta-tests that validate the serialized hook_deny_strength_matrix.json produced by the probe harness: - test_matrix_combination_count: row count must equal EXPECTED_NON_INERT_COMBINATIONS (inert probes are not recorded) - test_works_as_is_hooks_have_soft_or_better: each works-as-is hook must have at least one soft/hard row in the matrix - test_not_applicable_hooks_appear_only_as_inert: not-applicable hooks must be absent from the matrix entirely (since their probes are inert) All three tests skip gracefully when the matrix JSON is absent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Shorten the long pip/pytest/gh command string in TOOL_CLASS_PAYLOADS (Bash + mcp_run_cmd entries) and trim the inline comment in test_probe to fit within the 99-char ruff line limit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tput A guard returning an empty dict (no stdout) was classified as 'soft', causing test_works_as_is_hooks_have_soft_or_better to pass even for a completely non-functional guard. Introduce 'none' strength when result is empty so the meta-test correctly detects inactive guards. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
works_as_is_stems used hd.scripts[0] only, missing non-first scripts of multi-script works-as-is hooks (e.g. mcp_run_skill has 5 scripts, dispatch_food_truck has 4). Probe rows for those scripts were recorded in the matrix but excluded from the coverage check, allowing gaps to pass silently. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Trecek
force-pushed
the
t5-p1-a1-wp2-produce-a-deterministic-merge-blocking-probe-ha/3998
branch
from
69ae060 to
0ae81b6
Compare
Trecek
deleted the
t5-p1-a1-wp2-produce-a-deterministic-merge-blocking-probe-ha/3998
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Create a deterministic, merge-blocking probe harness that exercises every PreToolUse deny-mechanism guard script across the full (tool_class × session_mode × backend) matrix. Three new files in
tests/execution/backends/:conftest.py— xdist-safe probe row accumulation viaworkeroutputIPC, matrix JSON serialization to.autoskillit/temp/hook_deny_strength_matrix.json, shared constants, and bothEXPECTED_TOTAL_PROBE_COUNTandEXPECTED_NON_INERT_COMBINATIONSderived fromHOOK_REGISTRY.test_hook_deny_efficacy_probe.py— parametrizedTestHookDenyEfficacyProbeclass exercising each deny-mechanism guard script as a subprocess, plus a collection-count meta-test and a scope-boundary module docstring.test_hook_strength_matrix.py— three meta-tests validating the serialized strength matrix: combination count, works-as-is strength floor, and not-applicable inert invariant.Closes #3998
Implementation Plan
Plan file:
/home/talon/projects/autoskillit-runs/impl-20260627-131941-432993/.autoskillit/temp/make-plan/T5-P1-A1-WP2_hook_deny_efficacy_probe_plan_2026-06-27_132500.md🤖 Generated with Claude Code via AutoSkillit
Token Usage Summary
* Step used a non-Anthropic provider; caching behavior may differ.
Token Efficiency
Model Usage Breakdown