Implementation Plan: T5-P1-A1-WP2 Hook Deny Efficacy Probe Harness#4134
Merged
Trecek merged 1 commit intoJun 28, 2026
Merged
Conversation
…applicable_stems The PreToolUse deny-efficacy strength matrix meta-tests had two latent bugs: - works_as_is_stems filtered only by codex_status="works-as-is", so 8 PostToolUse/SessionStart hooks (pretty_output_hook, token_summary_hook, quota_post_hook, pipeline_step_post_hook, recipe_confirmed_post_hook, review_gate_post_hook, resume_gate_post_hook, session_start_hook) were included in the assertion set. These hooks never appear in the matrix because the probe only records PreToolUse+deny combinations, so test_works_as_is_hooks_have_soft_or_better would permanently fail once the matrix JSON is present. - not_applicable_stems used hd.scripts[0], indexing only the first script per HookDef. Currently all not-applicable hooks are single-script so the bug is latent, but a future multi-script not-applicable hook would silently leak stems not covered by the assertion. Both fixes match the plan's Critical implementation notes.
Trecek
deleted the
t5-p1-a1-wp2-produce-a-deterministic-merge-blocking-probe-ha/3998
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Build a deterministic, merge-blocking probe harness that exercises every PreToolUse deny-mechanism guard script as a subprocess across the full (tool_class x session_mode x backend) matrix. Each combination is classified as deny/allow/advisory and accumulated into a strength matrix JSON artifact. Three meta-tests validate matrix completeness, strength floor for works-as-is hooks, and inert-only appearance for not-applicable hooks.
The deliverables are three files under
tests/execution/backends/:conftest.py— registry-derived count invariants, xdist-safe probe row accumulation viaworkeroutputIPCtest_hook_deny_efficacy_probe.py— parametrizedTestHookDenyEfficacyProbeclass + collection-count meta-testtest_hook_strength_matrix.py— three meta-tests consuming the serialized strength matrixCloses #3998
Implementation Plan
Plan file:
/home/talon/projects/autoskillit-runs/impl-20260627-231059-975107/.autoskillit/temp/make-plan/t5-p1-a1-wp2-hook-deny-efficacy-probe-harness_plan_2026-06-27_233100.md🤖 Generated with Claude Code via AutoSkillit
Token Usage Summary
* Step used a non-Anthropic provider; caching behavior may differ.
Token Efficiency
Model Usage Breakdown