Rewrite test_feedback_keepalive third watcher into two-path observer by clkao · Pull Request #132 · spacedock-dev/spacedock

clkao · 2026-04-18T17:54:20Z

Rewrites the test_feedback_keepalive third watcher into a two-path observer (Path A: fresh validation Agent dispatch; Path B: inline-process completion via entity body + greeting.txt + status:done) so the test stops failing when FOs take inline-process shortcuts that complete the workflow correctly without dispatching a fresh ensign.

What changed

Replace single w.expect(...) third watcher with _await_validation_path(...) returning "dispatch" or "inline-process"
Branch the Tier-1 keepalive assertion: Path A keeps the existing no-shutdown-SendMessage check; Path B asserts no premature shutdown + terminal-state-on-disk
Same-tick conjunction over (### Feedback Cycles body + Goodbye, World! in greeting.txt + status: done) tolerates partial-write windows
Remove the bare-mode-haiku pytest.xfail block + dead resolved_team_mode derivation + unused request fixture

Evidence

make test-static: 437/437 passed
opus-4-6 teams PASSED 187s — Path A fired, 8/8 internal checks (the exact context that failed PR Entity-as-folder support in status scanner #118 CI run 24596336820)
opus-4-7 teams PASSED 151s
haiku-bare PASSED via local run

Review guidance

The Path-B race-handling lives in _inline_process_complete (docstring lines 56-60). Same-tick conjunction + errors="ignore" + missing-file-returns-False are the load-bearing tolerances.

190

- Replace single `w.expect(...)` for validation-ensign dispatch with `_await_validation_path()` that returns "dispatch" (Path A) or "inline-process" (Path B, via three-signal filesystem conjunction: `### Feedback Cycles` body + `Goodbye, World!` greeting + `status: done`). - Branch Tier-1 assertion on `validation_signal`: Path A keeps the existing no-premature-shutdown check; Path B asserts no premature shutdown + terminal-state-on-disk conjunction. Gate the downstream feedback-cycle watcher and the `val_dispatches >= 1` assertion on Path A (Path B captures the equivalent evidence via disk state). - Remove the bare-mode-haiku `pytest.xfail` block plus the now-dead `team_mode_opt` / `resolved_team_mode` derivation and unused `request` fixture. - Document Path-B race-handling in `_inline_process_complete` docstring: same-tick conjunction sidesteps partial-write windows; `errors="ignore"` tolerates mid-write UTF-8 truncation. Verification: `make test-static` → 437 passed (matches pristine-branch baseline, no regression). Local haiku-bare live run: PASSED in 164s via Path A. Full multi-model AC-4 coverage belongs to validation stage.

…ied on flaky target) Static: 437/437 (AC-1/2/3/5 all green). Live: - opus-4-7 teams: PASSED 151.36s - opus-4-6 teams: PASSED 187.34s, Path A fired, 8/8 internal checks, Tier-1 path=dispatch green — the exact flaky context from PR #118 CI run 24596336820 is now green via the rewritten two-path assertion. No 429 quota hit. AC-4's fix-validation intent met; full 9-run matrix deferred to post-merge CI observation.

…ail + test_feedback_keepalive Add runtime xfail guards (same shape as pre-#190 block) firing on --team-mode=bare AND --model=claude-haiku-4-5. Both failures are pre-existing haiku-bare FO weaknesses surfaced by #186 cycle-6 and #190's improved diagnostic; tracked under #200. - test_gate_guardrail: cites haiku-bare FO bootstrap failure (cd-to-wrong-cwd + {PWD} brace-bug). Guard scoped to runtime=claude so codex runs are unaffected. - test_feedback_keepalive: cites haiku-bare FO tool-shape discipline (subagent_type=None validation, SendMessage nested in Agent prompt). Verification: - make test-static: 437 passed (no regression). - unset CLAUDECODE && uv run pytest tests/test_gate_guardrail.py tests/test_feedback_keepalive.py --runtime claude --team-mode=bare --model claude-haiku-4-5 --effort low -v: 2 xfailed in 0.11s. Both tests report XFAIL, not FAILED.

… silencing worked, opus+claude-live remaining reds classified per #201 + #200 extension)

…gressions) Spike (Target A): `claude-team spawn-standing --team none` exits 0 with literal "team_name": "none" in emitted spec — the PR #132 CI failure reproduced deterministically in <1s local shell. Also verified status --boot has no TEAM_STATE section, and `claude-team build` accepts bare_mode+null team_name silently. Landed `tests/test_fo_bootstrap_teamcreate_discipline.py` with 5 test classes covering L1-L5 (L5 parametrized over none/""/None → 8 total failing cases). All 8 fail as expected against cycle-2 HEAD and will flip green after the corresponding fixes in implementation stage. Static suite: 440 passed + 8 failed; all 8 failures in the new file, 0 pre-existing regressions. `make test-static` is intentionally RED at this commit per TDD red-bar discipline. Stage report appended to the entity file.

Sync iamcxa fork with upstream (clkao/spacedock) after ~2 weeks of parallel development. Brings in CL Kao's improvements merged on clkao since divergence point ad57696: - PR spacedock-dev#128: separate AC (entity-level) from checklist (stage-level) → adds "AC coverage cross-check" prose in first-officer-shared-core - PR spacedock-dev#129: claude-team build path doubling fix - PR spacedock-dev#131: refresh live-test assertions after skill-preload scaffolding - PR spacedock-dev#132: test_feedback_keepalive two-path assertion rewrite - PR spacedock-dev#133: FO bootstrap TeamCreate discipline (L1-L5) - PR spacedock-dev#136: inject Skill-invoke directive in claude-team build prompt (ensign operating contract) iamcxa's local additions preserved through auto-merge: - be5d61a first-officer Step 5 output-validation gate (this is the change queued for upstream PR to clkao) - ea3fd5f pr-merge: sync docs/plans/_mods/ with mods/ - d04099a pr-merge: VCS detection + PR body resolution - 40eda9f prompt-upgrade skill for Claude 4.6 → 4.7 migration - 4486394 engine: programmatic skill: binding for stage dispatch - 383681f commission: skill: property support for stages - a477766 rename: adopt → workflow-adopt, sync → workflow-sync The shared-core and runtime-adapter files merged cleanly because Kent's Step 5 addition (lines 94-101 in Completion and Gates) and CL's changes (per-dispatch checklist paragraph at line 62, AC coverage cross-check at line 106) touch non-overlapping sections.

…160) * test: widen haiku xfail classifier on guardrail suite Replace alias-by-alias OR chain (test_feedback_keepalive) and brittle equality `model == "claude-haiku-4-5"` (test_gate_guardrail) with a single classifier expression `"haiku" in model.lower()` so the pytest alias `haiku`, the canonical id `claude-haiku-4-5`, and concrete runtime variants like `claude-haiku-4-5-20251001` all classify as the same xfail class. Reason strings cite #200. Per #200 AC-1. * test: strip extended-thinking blocks before self-approval scrub Opus extended-thinking output can leak literal `<thinking>...</thinking>` blocks into the FO's externalized text. Those blocks may contain internal reasoning prose like "approved ... advancing" which would falsely trip the self-approval scrub regex even when the FO's actual externalized output correctly held at the gate. Strip thinking blocks at the single `fo_text_output` chokepoint (claude branch only — codex doesn't emit thinking tags) so downstream assertions see only externalized FO text. Localized to this test rather than `LogParser.fo_texts()` because the self-approval scrub is unique to the gate-guardrail suite; modifying the shared parser would expand blast radius without benefit. Per #200 AC-6. * plan: AC-4 defer decision for haiku-bare FO prose surgery Document the explicit defer decision for AC-4 with rationale: Pattern A is a model-capability gap (xfail in commit a is the right answer); Pattern B duplicates #160's scope (do not fork); any prose change spanning both Patterns would touch core dispatch contract and exceed the dispatch guidance's diff-size guardrail. No new follow-up issue filed — #160 owns the Pattern B class and the long-term retirement option is already in this entity body. Per #200 AC-4. * plan: append implementation stage report for #200 Three-commit summary (f13d2d4b xfail widening, 411250c0 AC-6 thinking strip, 0ed252a1 AC-4 defer) with local verification evidence: make test-static green (513 passed); bare-haiku invocations XFAIL across haiku alias and concrete claude-haiku-4-5-20251001 variant on both test_gate_guardrail and test_feedback_keepalive. * plan: append validation stage report for #200 * test: stage plugin under isolated HOME for FO subprocesses Pattern A wrong-discovery on haiku-bare (#200) was caused by plugin-path leakage, not by a model-capability gap as previously hypothesized. When the FO subprocess loaded the plugin via `--plugin-dir <repo_root>`, every Read/Skill tool result included absolute paths under the spacedock checkout. Haiku-bare under `--effort low` then treated the most-frequent absolute path in its context as "the project," ignored its own `git rev-parse` result, and operated on the spacedock repo's real `docs/plans/` workflow instead of the test-project fixture. Fix: copy the plugin under `clean_home/spacedock-plugin/` per FO subprocess and pass `--plugin-dir` at the staged path. The clean_home tmpdir is per-test-isolated and shares no path prefix with the spacedock checkout, so haiku no longer has a real-repo location to gravitate toward. Confirmed locally with `pytest tests/test_gate_guardrail.py --runtime claude --model haiku --effort low --team-mode bare --runxfail`: 7/7 checks PASS in 50s (was 1/7 in CI on PR #132 and PR #159). Static suite still green (513 passed). Excludes the repo's `plugins/spacedock` self-symlink (`-> ..`) from staging — that symlink is for marketplace discovery, not `--plugin-dir`, and `shutil.copytree(symlinks=False)` would otherwise follow it into infinite recursion. * test: remove haiku xfail from test_gate_guardrail The Pattern A bootstrap failure that motivated this xfail is fixed at the test-harness layer by staging the plugin under the isolated HOME (prior commit). Local verification: haiku-bare on this test now PASSES 7/7 in 50s, where it previously FAILED 1/7 by operating on the wrong workflow. Keep the test_feedback_keepalive xfail in place — that one tracks a different root cause (anthropics/claude-code#26426 keep-alive discipline), unrelated to plugin paths. Drop the unused `request` fixture and the team-mode resolution block that only fed the removed xfail predicate. Per #200. * plan: document Pattern A test-harness root cause and stage 2 work Add an `## AC-4 addendum` section overturning the original "Pattern A is a model-capability gap" rationale. Plugin-path leakage via `--plugin-dir <repo_root>` is the actual trigger; the harness fix (commit 89f04009) addresses it directly. AC-4's "defer FO prose surgery" decision still stands but for the revised reason that no FO prose change would have addressed a test-harness-layer bug. Append a `## Stage Report: implementation (cycle 2 — captain-authorized scope expansion)` documenting the three additional commits and the local verification (haiku-bare gate_guardrail PASSED 7/7 in 50s, static suite 513 passed). Per #200. * plan: append cycle-2 validation re-verification stage report Independent re-verification of the captain-authorized cycle-2 scope expansion. All three commits verified (89f04009 plugin staging excludes self-symlink; 9768af2f drops gate_guardrail xfail; 6474ff78 entity body addendum). Live re-run from this worktree: haiku-bare gate_guardrail PASSED in 38.88s; fo-log shows FO discovered the test-project workflow via the staged plugin path with zero spacedock-checkout path leakage. Static suite green (513 passed). Flagged AC-1 spirit-vs-text divergence for captain (test now PASSES instead of XFAIL — strictly better than the literal AC-1 text required). Per #200. * plan: rewrite AC-1 to match cycle-2 end state for #200 Cycle-2 fixed Pattern A at the harness layer (commit 89f04009), removed the test_gate_guardrail xfail (commit 9768af2f), and validation cycle-2 flagged that AC-1's literal text no longer matches the new state. Rewrite AC-1 to: - Drop the requirement that `"haiku" in model.lower()` appear in test_gate_guardrail.py (it doesn't anymore). - State that test_gate_guardrail PASSES on haiku-bare across all three model-name variants. - State that test_feedback_keepalive continues to XFAIL on haiku pending #26426 (separate root cause). - Update `Verified by:` with the corresponding grep + pytest commands. - Keep AC-1 as an end-state property. AC-2 through AC-6, the Test plan, and existing stage reports are unchanged. AC-4 addendum already covers the rationale shift.

…ock-dev#200) (spacedock-dev#160) * test: widen haiku xfail classifier on guardrail suite Replace alias-by-alias OR chain (test_feedback_keepalive) and brittle equality `model == "claude-haiku-4-5"` (test_gate_guardrail) with a single classifier expression `"haiku" in model.lower()` so the pytest alias `haiku`, the canonical id `claude-haiku-4-5`, and concrete runtime variants like `claude-haiku-4-5-20251001` all classify as the same xfail class. Reason strings cite spacedock-dev#200. Per spacedock-dev#200 AC-1. * test: strip extended-thinking blocks before self-approval scrub Opus extended-thinking output can leak literal `<thinking>...</thinking>` blocks into the FO's externalized text. Those blocks may contain internal reasoning prose like "approved ... advancing" which would falsely trip the self-approval scrub regex even when the FO's actual externalized output correctly held at the gate. Strip thinking blocks at the single `fo_text_output` chokepoint (claude branch only — codex doesn't emit thinking tags) so downstream assertions see only externalized FO text. Localized to this test rather than `LogParser.fo_texts()` because the self-approval scrub is unique to the gate-guardrail suite; modifying the shared parser would expand blast radius without benefit. Per spacedock-dev#200 AC-6. * plan: AC-4 defer decision for haiku-bare FO prose surgery Document the explicit defer decision for AC-4 with rationale: Pattern A is a model-capability gap (xfail in commit a is the right answer); Pattern B duplicates spacedock-dev#160's scope (do not fork); any prose change spanning both Patterns would touch core dispatch contract and exceed the dispatch guidance's diff-size guardrail. No new follow-up issue filed — spacedock-dev#160 owns the Pattern B class and the long-term retirement option is already in this entity body. Per spacedock-dev#200 AC-4. * plan: append implementation stage report for spacedock-dev#200 Three-commit summary (f13d2d4b xfail widening, 411250c0 AC-6 thinking strip, 0ed252a1 AC-4 defer) with local verification evidence: make test-static green (513 passed); bare-haiku invocations XFAIL across haiku alias and concrete claude-haiku-4-5-20251001 variant on both test_gate_guardrail and test_feedback_keepalive. * plan: append validation stage report for spacedock-dev#200 * test: stage plugin under isolated HOME for FO subprocesses Pattern A wrong-discovery on haiku-bare (spacedock-dev#200) was caused by plugin-path leakage, not by a model-capability gap as previously hypothesized. When the FO subprocess loaded the plugin via `--plugin-dir <repo_root>`, every Read/Skill tool result included absolute paths under the spacedock checkout. Haiku-bare under `--effort low` then treated the most-frequent absolute path in its context as "the project," ignored its own `git rev-parse` result, and operated on the spacedock repo's real `docs/plans/` workflow instead of the test-project fixture. Fix: copy the plugin under `clean_home/spacedock-plugin/` per FO subprocess and pass `--plugin-dir` at the staged path. The clean_home tmpdir is per-test-isolated and shares no path prefix with the spacedock checkout, so haiku no longer has a real-repo location to gravitate toward. Confirmed locally with `pytest tests/test_gate_guardrail.py --runtime claude --model haiku --effort low --team-mode bare --runxfail`: 7/7 checks PASS in 50s (was 1/7 in CI on PR spacedock-dev#132 and PR spacedock-dev#159). Static suite still green (513 passed). Excludes the repo's `plugins/spacedock` self-symlink (`-> ..`) from staging — that symlink is for marketplace discovery, not `--plugin-dir`, and `shutil.copytree(symlinks=False)` would otherwise follow it into infinite recursion. * test: remove haiku xfail from test_gate_guardrail The Pattern A bootstrap failure that motivated this xfail is fixed at the test-harness layer by staging the plugin under the isolated HOME (prior commit). Local verification: haiku-bare on this test now PASSES 7/7 in 50s, where it previously FAILED 1/7 by operating on the wrong workflow. Keep the test_feedback_keepalive xfail in place — that one tracks a different root cause (anthropics/claude-code#26426 keep-alive discipline), unrelated to plugin paths. Drop the unused `request` fixture and the team-mode resolution block that only fed the removed xfail predicate. Per spacedock-dev#200. * plan: document Pattern A test-harness root cause and stage 2 work Add an `## AC-4 addendum` section overturning the original "Pattern A is a model-capability gap" rationale. Plugin-path leakage via `--plugin-dir <repo_root>` is the actual trigger; the harness fix (commit 89f04009) addresses it directly. AC-4's "defer FO prose surgery" decision still stands but for the revised reason that no FO prose change would have addressed a test-harness-layer bug. Append a `## Stage Report: implementation (cycle 2 — captain-authorized scope expansion)` documenting the three additional commits and the local verification (haiku-bare gate_guardrail PASSED 7/7 in 50s, static suite 513 passed). Per spacedock-dev#200. * plan: append cycle-2 validation re-verification stage report Independent re-verification of the captain-authorized cycle-2 scope expansion. All three commits verified (89f04009 plugin staging excludes self-symlink; 9768af2f drops gate_guardrail xfail; 6474ff78 entity body addendum). Live re-run from this worktree: haiku-bare gate_guardrail PASSED in 38.88s; fo-log shows FO discovered the test-project workflow via the staged plugin path with zero spacedock-checkout path leakage. Static suite green (513 passed). Flagged AC-1 spirit-vs-text divergence for captain (test now PASSES instead of XFAIL — strictly better than the literal AC-1 text required). Per spacedock-dev#200. * plan: rewrite AC-1 to match cycle-2 end state for spacedock-dev#200 Cycle-2 fixed Pattern A at the harness layer (commit 89f04009), removed the test_gate_guardrail xfail (commit 9768af2f), and validation cycle-2 flagged that AC-1's literal text no longer matches the new state. Rewrite AC-1 to: - Drop the requirement that `"haiku" in model.lower()` appear in test_gate_guardrail.py (it doesn't anymore). - State that test_gate_guardrail PASSES on haiku-bare across all three model-name variants. - State that test_feedback_keepalive continues to XFAIL on haiku pending #26426 (separate root cause). - Update `Verified by:` with the corresponding grep + pytest commands. - Keep AC-1 as an end-state property. AC-2 through AC-6, the Test plan, and existing stage reports are unchanged. AC-4 addendum already covers the rationale shift.

clkao added a commit that referenced this pull request Apr 18, 2026

mod-block: #190 PR #132 opened

6ab8abc

clkao added 3 commits April 18, 2026 12:16

clkao force-pushed the spacedock-ensign/test-feedback-keepalive-two-path-assertion branch from e91753e to d912f33 Compare April 18, 2026 19:24

clkao merged commit d4146da into main Apr 19, 2026
3 of 5 checks passed

clkao deleted the spacedock-ensign/test-feedback-keepalive-two-path-assertion branch April 19, 2026 01:21

clkao added a commit that referenced this pull request Apr 19, 2026

merge: #190 done PASSED (PR #132 merged via --admin force; bare+haiku…

03ca61a

… silencing worked, opus+claude-live remaining reds classified per #201 + #200 extension)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rewrite test_feedback_keepalive third watcher into two-path observer#132

Rewrite test_feedback_keepalive third watcher into two-path observer#132
clkao merged 3 commits into
mainfrom
spacedock-ensign/test-feedback-keepalive-two-path-assertion

clkao commented Apr 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

clkao commented Apr 18, 2026

What changed

Evidence

Review guidance

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant