Skip to content

Commit 85b2db3

Browse files
clkaoiamcxa
authored andcommitted
Fix haiku-bare CI false-discovery via isolated plugin staging (spacedock-dev#200) (spacedock-dev#160)
* test: widen haiku xfail classifier on guardrail suite Replace alias-by-alias OR chain (test_feedback_keepalive) and brittle equality `model == "claude-haiku-4-5"` (test_gate_guardrail) with a single classifier expression `"haiku" in model.lower()` so the pytest alias `haiku`, the canonical id `claude-haiku-4-5`, and concrete runtime variants like `claude-haiku-4-5-20251001` all classify as the same xfail class. Reason strings cite spacedock-dev#200. Per spacedock-dev#200 AC-1. * test: strip extended-thinking blocks before self-approval scrub Opus extended-thinking output can leak literal `<thinking>...</thinking>` blocks into the FO's externalized text. Those blocks may contain internal reasoning prose like "approved ... advancing" which would falsely trip the self-approval scrub regex even when the FO's actual externalized output correctly held at the gate. Strip thinking blocks at the single `fo_text_output` chokepoint (claude branch only — codex doesn't emit thinking tags) so downstream assertions see only externalized FO text. Localized to this test rather than `LogParser.fo_texts()` because the self-approval scrub is unique to the gate-guardrail suite; modifying the shared parser would expand blast radius without benefit. Per spacedock-dev#200 AC-6. * plan: AC-4 defer decision for haiku-bare FO prose surgery Document the explicit defer decision for AC-4 with rationale: Pattern A is a model-capability gap (xfail in commit a is the right answer); Pattern B duplicates spacedock-dev#160's scope (do not fork); any prose change spanning both Patterns would touch core dispatch contract and exceed the dispatch guidance's diff-size guardrail. No new follow-up issue filed — spacedock-dev#160 owns the Pattern B class and the long-term retirement option is already in this entity body. Per spacedock-dev#200 AC-4. * plan: append implementation stage report for spacedock-dev#200 Three-commit summary (f13d2d4b xfail widening, 411250c0 AC-6 thinking strip, 0ed252a1 AC-4 defer) with local verification evidence: make test-static green (513 passed); bare-haiku invocations XFAIL across haiku alias and concrete claude-haiku-4-5-20251001 variant on both test_gate_guardrail and test_feedback_keepalive. * plan: append validation stage report for spacedock-dev#200 * test: stage plugin under isolated HOME for FO subprocesses Pattern A wrong-discovery on haiku-bare (spacedock-dev#200) was caused by plugin-path leakage, not by a model-capability gap as previously hypothesized. When the FO subprocess loaded the plugin via `--plugin-dir <repo_root>`, every Read/Skill tool result included absolute paths under the spacedock checkout. Haiku-bare under `--effort low` then treated the most-frequent absolute path in its context as "the project," ignored its own `git rev-parse` result, and operated on the spacedock repo's real `docs/plans/` workflow instead of the test-project fixture. Fix: copy the plugin under `clean_home/spacedock-plugin/` per FO subprocess and pass `--plugin-dir` at the staged path. The clean_home tmpdir is per-test-isolated and shares no path prefix with the spacedock checkout, so haiku no longer has a real-repo location to gravitate toward. Confirmed locally with `pytest tests/test_gate_guardrail.py --runtime claude --model haiku --effort low --team-mode bare --runxfail`: 7/7 checks PASS in 50s (was 1/7 in CI on PR spacedock-dev#132 and PR spacedock-dev#159). Static suite still green (513 passed). Excludes the repo's `plugins/spacedock` self-symlink (`-> ..`) from staging — that symlink is for marketplace discovery, not `--plugin-dir`, and `shutil.copytree(symlinks=False)` would otherwise follow it into infinite recursion. * test: remove haiku xfail from test_gate_guardrail The Pattern A bootstrap failure that motivated this xfail is fixed at the test-harness layer by staging the plugin under the isolated HOME (prior commit). Local verification: haiku-bare on this test now PASSES 7/7 in 50s, where it previously FAILED 1/7 by operating on the wrong workflow. Keep the test_feedback_keepalive xfail in place — that one tracks a different root cause (anthropics/claude-code#26426 keep-alive discipline), unrelated to plugin paths. Drop the unused `request` fixture and the team-mode resolution block that only fed the removed xfail predicate. Per spacedock-dev#200. * plan: document Pattern A test-harness root cause and stage 2 work Add an `## AC-4 addendum` section overturning the original "Pattern A is a model-capability gap" rationale. Plugin-path leakage via `--plugin-dir <repo_root>` is the actual trigger; the harness fix (commit 89f04009) addresses it directly. AC-4's "defer FO prose surgery" decision still stands but for the revised reason that no FO prose change would have addressed a test-harness-layer bug. Append a `## Stage Report: implementation (cycle 2 — captain-authorized scope expansion)` documenting the three additional commits and the local verification (haiku-bare gate_guardrail PASSED 7/7 in 50s, static suite 513 passed). Per spacedock-dev#200. * plan: append cycle-2 validation re-verification stage report Independent re-verification of the captain-authorized cycle-2 scope expansion. All three commits verified (89f04009 plugin staging excludes self-symlink; 9768af2f drops gate_guardrail xfail; 6474ff78 entity body addendum). Live re-run from this worktree: haiku-bare gate_guardrail PASSED in 38.88s; fo-log shows FO discovered the test-project workflow via the staged plugin path with zero spacedock-checkout path leakage. Static suite green (513 passed). Flagged AC-1 spirit-vs-text divergence for captain (test now PASSES instead of XFAIL — strictly better than the literal AC-1 text required). Per spacedock-dev#200. * plan: rewrite AC-1 to match cycle-2 end state for spacedock-dev#200 Cycle-2 fixed Pattern A at the harness layer (commit 89f04009), removed the test_gate_guardrail xfail (commit 9768af2f), and validation cycle-2 flagged that AC-1's literal text no longer matches the new state. Rewrite AC-1 to: - Drop the requirement that `"haiku" in model.lower()` appear in test_gate_guardrail.py (it doesn't anymore). - State that test_gate_guardrail PASSES on haiku-bare across all three model-name variants. - State that test_feedback_keepalive continues to XFAIL on haiku pending #26426 (separate root cause). - Update `Verified by:` with the corresponding grep + pytest commands. - Keep AC-1 as an end-state property. AC-2 through AC-6, the Test plan, and existing stage reports are unchanged. AC-4 addendum already covers the rationale shift.
1 parent dcefeaf commit 85b2db3

4 files changed

Lines changed: 171 additions & 36 deletions

File tree

0 commit comments

Comments
 (0)