Commit 85b2db3
Fix haiku-bare CI false-discovery via isolated plugin staging (spacedock-dev#200) (spacedock-dev#160)
* test: widen haiku xfail classifier on guardrail suite
Replace alias-by-alias OR chain (test_feedback_keepalive) and brittle
equality `model == "claude-haiku-4-5"` (test_gate_guardrail) with a
single classifier expression `"haiku" in model.lower()` so the pytest
alias `haiku`, the canonical id `claude-haiku-4-5`, and concrete runtime
variants like `claude-haiku-4-5-20251001` all classify as the same
xfail class. Reason strings cite spacedock-dev#200.
Per spacedock-dev#200 AC-1.
* test: strip extended-thinking blocks before self-approval scrub
Opus extended-thinking output can leak literal `<thinking>...</thinking>`
blocks into the FO's externalized text. Those blocks may contain
internal reasoning prose like "approved ... advancing" which would
falsely trip the self-approval scrub regex even when the FO's actual
externalized output correctly held at the gate.
Strip thinking blocks at the single `fo_text_output` chokepoint
(claude branch only — codex doesn't emit thinking tags) so downstream
assertions see only externalized FO text. Localized to this test
rather than `LogParser.fo_texts()` because the self-approval scrub is
unique to the gate-guardrail suite; modifying the shared parser would
expand blast radius without benefit.
Per spacedock-dev#200 AC-6.
* plan: AC-4 defer decision for haiku-bare FO prose surgery
Document the explicit defer decision for AC-4 with rationale: Pattern A
is a model-capability gap (xfail in commit a is the right answer);
Pattern B duplicates spacedock-dev#160's scope (do not fork); any prose change
spanning both Patterns would touch core dispatch contract and exceed
the dispatch guidance's diff-size guardrail.
No new follow-up issue filed — spacedock-dev#160 owns the Pattern B class and the
long-term retirement option is already in this entity body.
Per spacedock-dev#200 AC-4.
* plan: append implementation stage report for spacedock-dev#200
Three-commit summary (f13d2d4b xfail widening, 411250c0 AC-6 thinking
strip, 0ed252a1 AC-4 defer) with local verification evidence:
make test-static green (513 passed); bare-haiku invocations XFAIL
across haiku alias and concrete claude-haiku-4-5-20251001 variant on
both test_gate_guardrail and test_feedback_keepalive.
* plan: append validation stage report for spacedock-dev#200
* test: stage plugin under isolated HOME for FO subprocesses
Pattern A wrong-discovery on haiku-bare (spacedock-dev#200) was caused by
plugin-path leakage, not by a model-capability gap as previously
hypothesized. When the FO subprocess loaded the plugin via
`--plugin-dir <repo_root>`, every Read/Skill tool result included
absolute paths under the spacedock checkout. Haiku-bare under
`--effort low` then treated the most-frequent absolute path in its
context as "the project," ignored its own `git rev-parse` result, and
operated on the spacedock repo's real `docs/plans/` workflow instead
of the test-project fixture.
Fix: copy the plugin under `clean_home/spacedock-plugin/` per FO
subprocess and pass `--plugin-dir` at the staged path. The clean_home
tmpdir is per-test-isolated and shares no path prefix with the
spacedock checkout, so haiku no longer has a real-repo location to
gravitate toward.
Confirmed locally with `pytest tests/test_gate_guardrail.py
--runtime claude --model haiku --effort low --team-mode bare
--runxfail`: 7/7 checks PASS in 50s (was 1/7 in CI on PR spacedock-dev#132 and
PR spacedock-dev#159). Static suite still green (513 passed).
Excludes the repo's `plugins/spacedock` self-symlink (`-> ..`) from
staging — that symlink is for marketplace discovery, not
`--plugin-dir`, and `shutil.copytree(symlinks=False)` would otherwise
follow it into infinite recursion.
* test: remove haiku xfail from test_gate_guardrail
The Pattern A bootstrap failure that motivated this xfail is fixed at
the test-harness layer by staging the plugin under the isolated HOME
(prior commit). Local verification: haiku-bare on this test now
PASSES 7/7 in 50s, where it previously FAILED 1/7 by operating on the
wrong workflow.
Keep the test_feedback_keepalive xfail in place — that one tracks a
different root cause (anthropics/claude-code#26426 keep-alive
discipline), unrelated to plugin paths.
Drop the unused `request` fixture and the team-mode resolution block
that only fed the removed xfail predicate.
Per spacedock-dev#200.
* plan: document Pattern A test-harness root cause and stage 2 work
Add an `## AC-4 addendum` section overturning the original "Pattern A
is a model-capability gap" rationale. Plugin-path leakage via
`--plugin-dir <repo_root>` is the actual trigger; the harness fix
(commit 89f04009) addresses it directly. AC-4's "defer FO prose
surgery" decision still stands but for the revised reason that no FO
prose change would have addressed a test-harness-layer bug.
Append a `## Stage Report: implementation (cycle 2 — captain-authorized
scope expansion)` documenting the three additional commits and the
local verification (haiku-bare gate_guardrail PASSED 7/7 in 50s,
static suite 513 passed).
Per spacedock-dev#200.
* plan: append cycle-2 validation re-verification stage report
Independent re-verification of the captain-authorized cycle-2 scope
expansion. All three commits verified (89f04009 plugin staging excludes
self-symlink; 9768af2f drops gate_guardrail xfail; 6474ff78 entity body
addendum). Live re-run from this worktree: haiku-bare gate_guardrail
PASSED in 38.88s; fo-log shows FO discovered the test-project workflow
via the staged plugin path with zero spacedock-checkout path leakage.
Static suite green (513 passed). Flagged AC-1 spirit-vs-text divergence
for captain (test now PASSES instead of XFAIL — strictly better than
the literal AC-1 text required).
Per spacedock-dev#200.
* plan: rewrite AC-1 to match cycle-2 end state for spacedock-dev#200
Cycle-2 fixed Pattern A at the harness layer (commit 89f04009),
removed the test_gate_guardrail xfail (commit 9768af2f), and validation
cycle-2 flagged that AC-1's literal text no longer matches the new
state. Rewrite AC-1 to:
- Drop the requirement that `"haiku" in model.lower()` appear in
test_gate_guardrail.py (it doesn't anymore).
- State that test_gate_guardrail PASSES on haiku-bare across all three
model-name variants.
- State that test_feedback_keepalive continues to XFAIL on haiku
pending #26426 (separate root cause).
- Update `Verified by:` with the corresponding grep + pytest commands.
- Keep AC-1 as an end-state property.
AC-2 through AC-6, the Test plan, and existing stage reports are
unchanged. AC-4 addendum already covers the rationale shift.1 parent dcefeaf commit 85b2db3
4 files changed
Lines changed: 171 additions & 36 deletions
File tree
- docs/plans
- scripts
- tests
0 commit comments