Unpin CI default back to opus (opus-4-7) — #186#125
Merged
Conversation
…tatus: done) AC-1 re-verified, AC-2 dependency map written at implementation kickoff. AC-3 live diagnosis SKIPPED per dispatch step 3 (#183 is status: validation, PR #122 not yet merged — running AC-3 now would produce fo-logs whose timing is dominated by Category-B ensign blocking-sleep, defeating the symptom-(i)/(ii)/(iii)/(iv) distinction per the entity body's evidence- integrity rationale). Recorded AC-3 prep scaffolding (test invocation, artifact-capture snippet, per-event summary template, task_id emitter-attribution plan) so the next implementation pass can start AC-3 immediately after #183 merges. AC-4/AC-5/AC-6 SKIPPED as downstream of AC-3. AC-7 N/A in this blocked cycle (zero workflow diffs). No production-code edits.
…acts Five sequential runs of test_feedback_keepalive on opus-4-7 --effort low with KEEP_TEST_DIR=1 captured under docs/plans/_evidence/green-opus-4-7-full-suite/. Every run PASSED. Wallclocks 131s, 118s, 175s, 119s, 121s — all well under the 240s validation-dispatch deadline. Zero `task_id` errors in any fo-log. Run 5 shows a single benign tool_use_error (Exit code 128 from git adding an embedded worktree) unrelated to the C1 symptom class. The merges of #183 (BashOutput polling), #185 (three-signal OR-gates in test_feedback_keepalive + test_standing_teammate_spawn), and #172 (lazy standing-teammate-spawn) — all landed on main between #186's cycle-1 dispatch and this cycle-3 re-dispatch — together constitute the de-facto mechanism fix: no additional production change is needed to meet AC-4 at 5/5 on opus-4-7.
AC-7 per cycle-3 dispatch: once mechanism-fix verification hits 5/5 on opus-4-7 locally, flip the CI default back. All five AC-4 verify runs of tests/test_feedback_keepalive.py on opus-4-7 --effort low PASSED cleanly (see prior commit's evidence under docs/plans/_evidence/). Zero `task_id` errors, validation dispatch observed within 240s in every run. This reverts #181's pin in: - Makefile: `OPUS_MODEL ?= claude-opus-4-6` -> `OPUS_MODEL ?= opus` - .github/workflows/runtime-live-e2e.yml: both `EFFECTIVE_MODEL` and `MODEL_FLAG` defaults revert to `opus` and removes the inline `# Pinned to ...` comments. The $(OPUS_MODEL) plumbing and `--model "$MODEL_FLAG"` override path from #181 are preserved — they remain useful as an override knob.
…C-7 unpin DONE - AC-3: 3/3 PASS on opus-4-7 local; per-run fo-log line-number + tool_use_id citations recorded. None of symptoms (i)/(ii)/(iii)/(iv) reproduce — effective mechanism fix = #183 + #185 + #172 combined landing on main. - AC-4: 5/5 PASS on opus-4-7 --effort low (wallclocks 131/118/175/119/121s). - AC-7: unpin committed in 4f736d1d (Makefile + runtime-live-e2e.yml only; HARD scope fence against skills/first-officer/references/* respected). - Post-unpin opus-4-6 regression PASS (99.7s); static suite 435 passed. - AC-5 / AC-6 deferred to validation stage per dispatch step 11. - Budget ~$8.50 vs $80 ceiling.
…on opus-4-7 local Single `make test-live-claude-opus OPUS_MODEL=opus` run on branch tip a9c78e8 (with unpin commit a730858 in place). Exit 0. Serial tier: 1 passed, 3 skipped, 1 xpassed, exit 0 (161s) Parallel tier: 3 passed, 3 skipped, 8 xfailed, 2 xpassed, exit 0 (914s) Combined wallclock: ~18 min. All four non-XFAIL/non-SKIP tests PASSED on opus-4-7: - test_gate_guardrail (serial) - test_feedback_keepalive (parallel/gw3) - test_merge_hook_guardrail (parallel/gw3) - test_standing_teammate_spawn (parallel/gw1) Cycle-3 CI failures (test_standing_teammate_spawn StepTimeout; test_gate_guardrail Phase-3 checks) did not reproduce locally. Unpin gate satisfied per captain's cycle-4 scope reframe (1x full-suite = unpin gate; CI handles reliability after merge). Unpin commit a730858 retained. Evidence: docs/plans/_evidence/green-opus-4-7-full-suite/cycle4-fullsuite-run1.log
…ocally on opus-4-7; gate-guardrail evidence vacuous (test ignores --model) Four isolation runs at opus-4-7 low-effort with KEEP_TEST_DIR=1: - test_standing_teammate_spawn: 0/2 FAIL. Reproduces reliably in isolation. Root cause is the watcher predicate for "SendMessage to echo-agent" matching the FO's own shutdown_request during teardown (Category A, pre-existing #185 surface). Ensign's ping->ECHO roundtrip happens inside the subagent transcript, invisible to the parent fo-log. Archive body lacks "ECHO: ping" in both runs; run 2 shows the FO tearing down cleanly at $1.52/95s before the ensign completes (Category C3 signal; explicitly out-of-scope for #186). - test_gate_guardrail: 2/2 PASS but evidence is vacuous. Test source at tests/test_gate_guardrail.py:48-52 passes no --model to extra_args and the test signature doesn't accept the model fixture, so the --model claude-opus-4-7 CLI flag is silently ignored. Both runs ran on claude-sonnet-4-6. Fixing this requires plumbing the model fixture through extra_args similar to test_feedback_keepalive. Recommendation: BOUNCE for investigation. Do NOT merge the unpin. standing_teammate needs a Category-A predicate fix and a C3 scope decision; gate_guardrail needs model-fixture plumbing before any future cycle can claim opus-4-7 coverage. Budget spend: ~$4.4 of ~$10 allotted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e cycle 1-5 reports Three changes bundled per captain dispatch: 1. tests/test_gate_guardrail.py — plumb `model` + `effort` fixtures through to `extra_args`. Cycle 5 proved the test was silently running on haiku/sonnet even under `--model opus-4-7`, making all prior "gate_guardrail green on opus-4-7" claims vacuous. 2. .github/workflows/runtime-live-e2e.yml — check out the merged head (`refs/pull/<N>/merge`) instead of the bare PR head SHA across all five jobs. Testing the post-merge tree catches base-drift: #128/#129 hit flakes this session that would not have shown up against post-#188 main. Security model unchanged (env-approval gate stays the trust boundary); comment + `tests/README.md` + `test_runtime_live_e2e_workflow.py` updated to match. Per-job provenance now prints the PR head SHA ("diff under review") AND the resolved merge SHA recorded post-checkout, so audit links remain meaningful. 3. docs/plans/green-opus-4-7-full-suite.md — condense stage reports for ideation cycles 1-2 + implementation cycles 1/3/4/5 into summary lines retaining all factual outcomes, commit SHAs, and evidence-file paths. 513 → 201 lines before appending cycle 6's report.
… parallel tier) Cycle-6 one-shot `make test-live-claude-opus OPUS_MODEL=opus` hit exit 1 after 5m03s: serial-tier `test_gate_guardrail` PASSED on real opus-4-7 (first non-vacuous pass — cycle-6 fixture-plumbing fix worked), but parallel tier's three non-XFAIL tests (keepalive, merge_hook, standing_teammate) all FAILED with `api_error_status: 429` and `<synthetic>` "You're out of extra usage" responses. Account quota hit ~5 min into the parallel tier; reset at 3am PT. Triage: failures are NOT base-drift (merge-ref migration can't cause an account-side quota), NOT the #194 FO ECHO: ping flake (no ECHO: ping watchers fired — quota hit before first ensign dispatch), NOT an opus-4-7 mechanism regression (cycle 3's 5/5 + cycle 4's 1/1 proved the runtime-under-budget works; cycle 6's serial-tier gate_guardrail PASS on real opus-4-7 confirms). Per dispatch spec verdict tree: "any failure NOT attributable to #194 → flag for captain triage". Recommendation: HOLD branch, re-run `make test-live-claude-opus` after 3am PT quota reset; do NOT revert unpin preemptively; do NOT merge until a clean full-suite run lands. Evidence: docs/plans/_evidence/green-opus-4-7-full-suite/cycle6/ cycle6-fullsuite-run1.log (501 lines — preserves all three StepFailure traces with the embedded 429 / synthetic records).
clkao
force-pushed
the
spacedock-ensign/green-opus-4-7-full-suite
branch
from
a9c78e8 to
3c0c521
Compare
clkao
added a commit
that referenced
this pull request
Apr 18, 2026
clkao
added a commit
that referenced
this pull request
Apr 18, 2026
…npin validated, only red was bare-mode test_feedback_keepalive brittle watcher tracked by #190)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Unpin the CI default from
claude-opus-4-6back toopus(opus-4-7). The underlying opus-4-7 ensign regression that prompted #181's pin no longer reproduces on current main — the cumulative landings of #183 (BashOutput polling discipline), #185 (three-signal data-flow predicates + bare-mode haiku xfail), and #172 (lazy standing-teammate spawn) together closed the gap. 5/5 localtest_feedback_keepaliveruns on opus-4-7--effort lowpass cleanly, with zerotask_iderrors and validation dispatch within 240s in every run.What changed
.github/workflows/runtime-live-e2e.yml— bothEFFECTIVE_MODELandMODEL_FLAGdefault blocks revert fromclaude-opus-4-6toopus; inline pin comments removed.$MODEL_OVERRIDEplumbing preservedMakefile—OPUS_MODEL ?= claude-opus-4-6reverts toOPUS_MODEL ?= opus; inline pin comment removed.$(OPUS_MODEL)override preserveddocs/plans/_evidence/green-opus-4-7-full-suite/— five fo-log.jsonl + five pytest run-logs preserved as audit trail for AC-3/AC-4docs/plans/green-opus-4-7-full-suite.md— cycle-3 scope + Stage ReportNo production code changes. Scope fence against
skills/first-officer/references/*held — no prose mitigations.Evidence
Agenttool_use observed within 240s in every run; test predicate_agent_targets_stageaccepts the shape in all three.task_iderrors. Zero timeouts.Review guidance
The ensign's honest finding: the opus-4-7 symptom did not reproduce on a branch base that already included #183 + #185 + #172. No new mechanism fix was needed. The effective fix was already in main; this PR flips the default to let CI exercise it.
If post-merge CI surfaces a regression, the unpin commit is a 6-line revert. The CI run 24590771304's
task_idtool_use_erroris attributable to Claude Code SDK'sTaskUpdate/TodoWriteprimitive — out of spacedock scope.Deferred
test_standing_teammate_spawn3× on opus-4-7. Captain-optional extra validation; the test was at ~80% pre-Resolve folder-mode entity dispatch via worktree fallback (fixes #184) #185 and cycle-2's OR-gates make it more robust.make test-live-claude-opus2-consecutive-clean. Natural validation-stage gate.#186