Skip to content

Commit cbd2163

Browse files
Trecekclaude
andauthored
Rectify: check_review_posted Step Wiring (#4205)
## Summary Parts A and B of issue #4204 introduced the `check_review_posted` step and `review-effect-verification-waypoint` rule across all 4 bundled recipes, but the combined merged tree has 74 failing tests from 5 mechanical defects introduced by an incomplete round-3 remediation (`fe8931e8f`): - **RC1**: `optional: true` without `skip_when_false` left on `check_review_posted` in 3 recipes (implementation, implementation-groups, remediation) → `optional-without-skip-when` ERROR cascades to 70+ tests - **RC2**: No catch-all arm in `check_review_posted.on_result` in all 4 recipes → `unrouted-callable-verdict` ERROR (sentinel output `no_reviews_posted` is unrouted) - **RC3**: `check_review_posted.output_dir` in merge-prs uses a static relative path (`{{AUTOSKILLIT_TEMP}}/review-pr`, no `${{` template) with no `work_dir` → `run-python-requires-work-dir` ERROR - **RC4**: `_review_effect_verification_waypoint` step-selection uses `.startswith("/review-pr")` — never matches `/autoskillit:review-pr` forms → rule permanently silent → 3 "fires" tests fail - **RC4a**: `re_push_review_integration.on_success: derive_batch_ci_event` in merge-prs bypasses `check_review_posted` on the rebase-then-repush path → will cause the rule to fire on merge-prs once RC4 is fixed Part A eliminates all 5 defects. Part B (separate task, explicit authorization required) adds architectural immunity to prevent recurrence. **Note on merged tree state:** After merging both impl branches, `_analysis_bfs.py` already supports `barrier: str | frozenset[str]` (updated by Part B), so the `frozenset` barrier call in `_review_effect_verification_waypoint` is correct and requires no change. The `check_review_posted` callable, its `skill_contracts.yaml` entry, `smoke_utils` exports, and `tests/test_smoke_utils.py` are all present and passing. Only the 5 defects above need repair. Closes #4204 ## Implementation Plan Plan file: `/home/talon/projects/autoskillit-runs/remediation-20260707-160731-479581/.autoskillit/temp/rectify/rectify_check-review-posted-wiring_2026-07-07_163000_part_a.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) via AutoSkillit <!-- autoskillit:pipeline-signature steps=prepare_pr,run_arch_lenses,compose_pr,annotate_pr_diff,review_pr --> ## Token Usage Summary | Step | Model | count | uncached | output | cache_read | peak_ctx | turns | cache_write | time | |------|-------|-------|----------|--------|------------|----------|-------|-------------|------| | investigate* | fable | 1 | 17.0k | 59.5k | 2.7M | 181.9k | 71 | 236.8k | 21m 39s | | rectify* | sonnet | 1 | 225 | 59.8k | 2.5M | 148.3k | 77 | 130.0k | 30m 29s | | dry_walkthrough* | sonnet | 2 | 1.2k | 37.6k | 993.9k | 81.8k | 57 | 113.6k | 15m 1s | | implement* | sonnet | 2 | 472 | 30.7k | 3.6M | 86.7k | 166 | 131.5k | 10m 57s | | audit_impl* | sonnet | 2 | 188 | 31.7k | 947.3k | 67.4k | 62 | 112.2k | 11m 59s | | prepare_pr* | MiniMax-M3 | 1 | 44.2k | 3.9k | 199.8k | 47.3k | 14 | 0 | 52s | | compose_pr* | MiniMax-M3 | 1 | 40.0k | 3.4k | 192.1k | 41.9k | 13 | 0 | 48s | | **Total** | | | 103.3k | 226.4k | 11.2M | 181.9k | | 724.2k | 1h 31m | \* *Step used a non-Anthropic provider; caching behavior may differ.* ## Token Efficiency | Step | LoC Changed | cache_read/LoC | cache_write/LoC | output/LoC | |------|-------------|----------------|-----------------|------------| | investigate | 0 | — | — | — | | rectify | 0 | — | — | — | | dry_walkthrough | 0 | — | — | — | | implement | 1567 | 2325.4 | 83.9 | 19.6 | | audit_impl | 0 | — | — | — | | prepare_pr | 0 | — | — | — | | compose_pr | 0 | — | — | — | | **Total** | **1567** | 7147.9 | 462.1 | 144.5 | ## Model Usage Breakdown | Model | steps | uncached | output | cache_read | cache_write | time | |-------|-------|----------|--------|------------|-------------|------| | fable | 1 | 17.0k | 59.5k | 2.7M | 236.8k | 21m 39s | | sonnet | 4 | 2.1k | 159.7k | 8.1M | 487.3k | 1h 8m | | MiniMax-M3 | 2 | 84.2k | 7.2k | 391.9k | 0 | 1m 40s | --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 270a0f8 commit cbd2163

77 files changed

Lines changed: 1374 additions & 173 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

src/autoskillit/core/types/_type_backend.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,8 @@ class BackendCapabilities:
142142
has_unguarded_filesystem_access: bool = field(default=False)
143143
# True when backend's git metadata directories (.git/worktrees/) are writable
144144
git_metadata_writable: bool = field(default=True)
145+
# True when the backend can make outbound GitHub API write calls without sandbox restriction
146+
github_api_callable: bool = field(default=False)
145147
# Native skill invocation prefix character used by this backend's model/CLI.
146148
# Claude Code uses "/" (slash-commands via the Skill tool).
147149
# Codex uses "$" (dollar-mention via extract_tool_mentions).
@@ -262,6 +264,7 @@ def model_class(model: str) -> str:
262264
supports_context_window_suffix=True,
263265
has_unguarded_filesystem_access=False,
264266
git_metadata_writable=True,
267+
github_api_callable=True,
265268
skill_sigil="/",
266269
session_dir_persistent=False,
267270
supports_model_invocation_gating=True,
@@ -313,6 +316,7 @@ class SkillSessionConfig:
313316
resume_message: str | None = None
314317
sandbox_mode: str = "workspace-write"
315318
backend_override: str | None = None
319+
network_access: bool = False
316320

317321

318322
@dataclass(frozen=True, slots=True)

src/autoskillit/core/types/_type_constants_registries.py

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -302,10 +302,14 @@ class SkillCapabilityDef:
302302

303303
description: str
304304
codex_status: Literal["works-as-is", "degraded", "fix-required", "not-applicable"]
305+
required_sandbox_overrides: frozenset[str] = frozenset()
306+
worker_routable: bool = False
305307

306308
@property
307309
def required_backends(self) -> frozenset[str]:
308-
if self.codex_status == "not-applicable":
310+
# worker_routable=True → reroute (REROUTE), not reject: required_backends must be empty
311+
# so the compat gate doesn't fire for routable capabilities.
312+
if self.codex_status == "not-applicable" and not self.worker_routable:
309313
return frozenset({AGENT_BACKEND_CLAUDE_CODE})
310314
return frozenset()
311315

@@ -353,6 +357,16 @@ def required_backends(self) -> frozenset[str]:
353357
"git worktree add, git checkout -b)"
354358
),
355359
codex_status="not-applicable",
360+
worker_routable=True,
361+
),
362+
"github_api_write": SkillCapabilityDef(
363+
description=(
364+
"Skill makes outbound GitHub API write calls (gh pr review, gh api --method POST, "
365+
"gh pr create, gh pr merge, gh issue create/edit/close, etc.) that require network "
366+
"access. On Codex workers, enables network_access=true in the workspace-write sandbox."
367+
),
368+
codex_status="fix-required",
369+
required_sandbox_overrides=frozenset({"sandbox_workspace_write.network_access=true"}),
356370
),
357371
}
358372

src/autoskillit/core/types/_type_protocols_execution.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ async def run(
7373
caller_session_id: str | None = None,
7474
inspector_eligible: bool = False,
7575
inspector_model: str = "",
76+
network_access: bool = False,
7677
) -> SkillResult: ...
7778

7879
async def dispatch_food_truck(

src/autoskillit/execution/backends/_backend_cmd_builder_base.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,7 @@ def _apply_config(self, config: SkillSessionConfig) -> dict[str, Any]:
163163
"resume_checkpoint": config.resume_checkpoint,
164164
"resume_message": config.resume_message,
165165
"sandbox_mode": config.sandbox_mode,
166+
"network_access": config.network_access,
166167
}
167168

168169

src/autoskillit/execution/backends/codex.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -592,6 +592,7 @@ def capabilities(self) -> BackendCapabilities:
592592
# field; rules_backend_compat.py emits ERROR for
593593
# git_metadata_write skills on backends where this is False.
594594
git_metadata_writable=False, # sandbox excludes .git metadata path
595+
github_api_callable=False,
595596
skill_sigil="$",
596597
session_dir_persistent=True,
597598
supports_model_invocation_gating=False,
@@ -692,6 +693,7 @@ def build_skill_session_cmd(
692693
resume_checkpoint: SessionCheckpoint | None = None,
693694
resume_message: str | None = None,
694695
sandbox_mode: str = "workspace-write",
696+
network_access: bool = False,
695697
) -> CmdSpec:
696698
if config is not None:
697699
cfg = self._apply_config(config)
@@ -712,6 +714,7 @@ def build_skill_session_cmd(
712714
resume_checkpoint = cfg["resume_checkpoint"]
713715
resume_message = cfg["resume_message"]
714716
sandbox_mode = cfg["sandbox_mode"]
717+
network_access = cfg.get("network_access", False)
715718
if plugin_source is not None:
716719
logger.warning("codex_plugin_source_discarded", plugin_source=str(plugin_source))
717720
if output_format != OutputFormat.JSON:
@@ -790,9 +793,13 @@ def build_skill_session_cmd(
790793
required=SKILL_SESSION_REQUIRED_ENV | {MCP_CLIENT_BACKEND_ENV_VAR},
791794
)
792795

796+
_net_overrides: list[str] = []
797+
if network_access:
798+
_net_overrides.append("sandbox_workspace_write.network_access=true")
793799
cmd = _codex_exec_base(
794800
sandbox=sandbox_mode,
795801
bypass_hook_trust=self.capabilities.mcp_config_capable,
802+
extra_overrides=_net_overrides,
796803
)
797804
if model:
798805
cmd += [CodexFlags.MODEL, self.translate_model(model)]

src/autoskillit/execution/headless/__init__.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,7 @@ async def run_headless_core(
145145
caller_session_id: str | None = None,
146146
inspector_eligible: bool = False,
147147
inspector_model: str = "",
148+
network_access: bool = False,
148149
) -> SkillResult:
149150
"""Shared headless runner used by run_skill.
150151
@@ -193,6 +194,7 @@ async def run_headless_core(
193194
sandbox_mode="read-only"
194195
if readonly_skill
195196
else ctx.backend.capabilities.default_skill_sandbox_mode,
197+
network_access=network_access,
196198
)
197199
step_backend: CodingAgentBackend | None = None
198200
if backend_override is not None:
@@ -303,6 +305,7 @@ async def run(
303305
caller_session_id: str | None = None,
304306
inspector_eligible: bool = False,
305307
inspector_model: str = "",
308+
network_access: bool = False,
306309
) -> SkillResult:
307310
cfg = self._ctx.config.run_skill
308311
effective_timeout = timeout if timeout is not None else cfg.timeout
@@ -344,6 +347,7 @@ async def run(
344347
caller_session_id=caller_session_id,
345348
inspector_eligible=inspector_eligible,
346349
inspector_model=inspector_model,
350+
network_access=network_access,
347351
)
348352

349353
async def dispatch_food_truck(

src/autoskillit/recipe/_analysis_bfs.py

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -99,12 +99,16 @@ def _build_success_step_graph(recipe: Recipe) -> dict[str, set[str]]:
9999
def bfs_reachable_without_barrier(
100100
recipe: Recipe,
101101
start: str,
102-
barrier: str,
102+
barrier: str | frozenset[str],
103103
) -> set[str]:
104104
"""BFS from ``start`` through success-path routing edges, stopping at ``barrier``.
105105
106106
Returns all step names reachable from ``start`` without crossing ``barrier``.
107-
The barrier step itself is not included in the returned set.
107+
The barrier step itself is included in the returned set — it is visited but
108+
not expanded beyond.
109+
110+
``barrier`` may be a single step name (str) or a frozenset of step names.
111+
When a frozenset is provided, any of the named steps acts as a barrier.
108112
109113
Only follows on_result conditions and on_success edges — error paths
110114
(on_failure, on_context_limit) are excluded because they are not
@@ -113,8 +117,9 @@ def bfs_reachable_without_barrier(
113117
This is the canonical implementation of the BFS-barrier pattern previously
114118
duplicated inline in ``push-before-audit`` and ``merge-base-unpublished``.
115119
"""
120+
barriers: set[str] = {barrier} if isinstance(barrier, str) else set(barrier)
116121
graph = _build_success_step_graph(recipe)
117-
return _bfs_capped(graph, {start}, {barrier})
122+
return _bfs_capped(graph, {start}, barriers)
118123

119124

120125
def _bfs_capped(

src/autoskillit/recipe/rules/graph/rules_graph_review.py

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,61 @@ def _check_run_skill_missing_context_limit(ctx: ValidationContext) -> list[RuleF
163163
return findings
164164

165165

166+
_REVIEW_EFFECT_ADVANCE_GATES = frozenset({"check_review_loop", "derive_batch_ci_event"})
167+
168+
169+
def _get_review_pr_steps(ctx: ValidationContext) -> list[str]:
170+
"""Return step IDs that dispatch the review-pr skill (either prefix form).
171+
172+
Exposed as a module-level function (not inlined in the rule body) so that
173+
silence tests can call it directly and assert the rule is triggered on the
174+
recipe under test before asserting zero findings.
175+
"""
176+
return [
177+
step_id
178+
for step_id, step in ctx.recipe.steps.items()
179+
if step.tool == "run_skill" and step.skill_name == "review-pr"
180+
]
181+
182+
183+
@semantic_rule(
184+
name="review-effect-verification-waypoint",
185+
severity=Severity.ERROR,
186+
description=(
187+
"No advance gate (check_review_loop or derive_batch_ci_event) must be reachable "
188+
"from any review-pr skill step on the success path without first crossing "
189+
"check_review_posted. Applies to all steps dispatching the review-pr skill "
190+
"regardless of step id (covers review_pr_integration in merge-prs.yaml)."
191+
),
192+
)
193+
def _review_effect_verification_waypoint(ctx: ValidationContext) -> list[RuleFinding]:
194+
effect_steps = _get_review_pr_steps(ctx)
195+
if not effect_steps:
196+
return []
197+
findings = []
198+
for start in effect_steps:
199+
reachable = bfs_reachable_without_barrier(
200+
ctx.recipe,
201+
start=start,
202+
barrier=frozenset({"check_review_posted"}),
203+
)
204+
for advance_gate in _REVIEW_EFFECT_ADVANCE_GATES:
205+
if advance_gate in reachable:
206+
findings.append(
207+
make_finding(
208+
rule_name="review-effect-verification-waypoint",
209+
step_name=start,
210+
message=(
211+
f"Step '{start}' (review-pr skill) can reach '{advance_gate}' "
212+
f"without crossing check_review_posted. Insert a check_review_posted "
213+
f"run_python step on every success path from '{start}' to "
214+
f"'{advance_gate}'."
215+
),
216+
)
217+
)
218+
return findings
219+
220+
166221
@semantic_rule(
167222
name="review-mode-reentry-waypoint-guard",
168223
description=(

src/autoskillit/recipe/schema.py

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
CaptureEntrySpec,
1818
DispatchGateType,
1919
RecipeSource,
20+
resolve_skill_name,
2021
)
2122

2223
AUTOSKILLIT_VERSION_KEY: Final = "autoskillit_version"
@@ -155,6 +156,16 @@ def __post_init__(self) -> None:
155156
f"the list, producing duplicates. Set retries: 0."
156157
)
157158

159+
@property
160+
def skill_name(self) -> str | None:
161+
"""Canonical skill name, resolving /name and /autoskillit:name forms.
162+
163+
Returns None if with_args has no skill_command or if resolve_skill_name
164+
cannot extract a name (template-only, unrecognized format, etc.).
165+
Use this in rule predicates instead of raw string operations on with_args.
166+
"""
167+
return resolve_skill_name(str((self.with_args or {}).get("skill_command", "")))
168+
158169

159170
@dataclass(frozen=True, slots=True)
160171
class RecipeBlock:

src/autoskillit/recipe/skill_contracts.yaml

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2574,6 +2574,35 @@ callable_contracts:
25742574
type: str
25752575
- name: had_blocking
25762576
type: str
2577+
autoskillit.smoke_utils.check_review_posted:
2578+
description: >
2579+
Verifies that batch_review_response_{pr_number}.json exists in output_dir
2580+
(github mode) or always passes (local mode). No subprocess calls.
2581+
output_dir must be an absolute path.
2582+
inputs:
2583+
- name: pr_number
2584+
type: int
2585+
required: true
2586+
- name: output_dir
2587+
type: str
2588+
required: true
2589+
- name: mode
2590+
type: str
2591+
required: true
2592+
allowed_values:
2593+
- local
2594+
- github
2595+
outputs:
2596+
- name: reviews_posted
2597+
type: str
2598+
allowed_values:
2599+
- "true"
2600+
- "false"
2601+
- name: sentinel
2602+
type: str
2603+
allowed_values:
2604+
- no_reviews_posted
2605+
- ""
25772606
autoskillit.smoke_utils.check_bug_report_non_empty:
25782607
inputs:
25792608
- name: workspace

0 commit comments

Comments
 (0)