Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 79 additions & 21 deletions .github/workflows/runtime-live-e2e.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
name: Runtime Live E2E

on:
pull_request:
workflow_dispatch:
inputs:
pr_number:
Expand All @@ -15,9 +16,15 @@ permissions:
jobs:
claude-live:
runs-on: ubuntu-latest
environment:
name: CI-E2E
deployment: false
env:
DISABLE_AUTOUPDATER: "1"
PR_NUMBER: ${{ inputs.pr_number }}
KEEP_TEST_DIR: "1"
TRIGGER_SOURCE: ${{ github.event_name }}
PR_NUMBER: ${{ github.event.pull_request.number }}
DISPATCH_PR_NUMBER: ${{ inputs.pr_number }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
steps:
- name: Check required secret
Expand All @@ -31,9 +38,21 @@ jobs:
uses: actions/github-script@v7
with:
script: |
const prNumber = Number(process.env.PR_NUMBER);
const triggerSource = process.env.TRIGGER_SOURCE;
let rawPrNumber;

if (triggerSource === "pull_request") {
rawPrNumber = process.env.PR_NUMBER;
} else if (triggerSource === "workflow_dispatch") {
rawPrNumber = process.env.DISPATCH_PR_NUMBER;
} else {
core.setFailed(`Unsupported trigger source: ${JSON.stringify(triggerSource)}`);
return;
}

const prNumber = Number(rawPrNumber);
if (!Number.isInteger(prNumber) || prNumber <= 0) {
core.setFailed(`workflow_dispatch input pr_number must be a positive integer, got ${JSON.stringify(process.env.PR_NUMBER)}`);
core.setFailed(`${triggerSource} PR number must be a positive integer, got ${JSON.stringify(rawPrNumber)}`);
return;
}

Expand All @@ -59,6 +78,7 @@ jobs:

await core.summary
.addHeading("Runtime Live E2E Provenance")
.addRaw(`- Trigger source: ${triggerSource}\n`)
.addRaw(`- PR number: #${prNumber}\n`)
.addRaw(`- Tested workflow SHA: \`${context.sha}\`\n`)
.addRaw(`- Current PR head SHA: \`${pull.head.sha}\`\n`)
Expand All @@ -78,6 +98,11 @@ jobs:

- uses: astral-sh/setup-uv@v6

- name: Prepare live artifact root
run: |
mkdir -p "$RUNNER_TEMP/spacedock-live/$GITHUB_JOB"
echo "SPACEDOCK_TEST_TMP_ROOT=$RUNNER_TEMP/spacedock-live/$GITHUB_JOB" >> "$GITHUB_ENV"

- name: Install Claude Code
run: |
curl -fsSL https://claude.ai/install.sh | bash
Expand All @@ -87,28 +112,34 @@ jobs:
run: |
git config --global user.name "github-actions[bot]"
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --global init.defaultBranch main

- name: Show tool versions
run: |
claude --version
uv --version

- name: Run Claude live suite
run: |
set -euo pipefail
unset CLAUDECODE && uv run tests/test_gate_guardrail.py --runtime claude
unset CLAUDECODE && uv run tests/test_rejection_flow.py --runtime claude
unset CLAUDECODE && uv run tests/test_scaffolding_guardrail.py
unset CLAUDECODE && uv run tests/test_feedback_keepalive.py
unset CLAUDECODE && uv run tests/test_dispatch_completion_signal.py
unset CLAUDECODE && uv run tests/test_merge_hook_guardrail.py --runtime claude
unset CLAUDECODE && uv run tests/test_push_main_before_pr.py
unset CLAUDECODE && uv run tests/test_rebase_branch_before_push.py
run: make test-live-claude

- name: Upload Claude live artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: runtime-live-e2e-claude-live
path: ${{ runner.temp }}/spacedock-live/${{ github.job }}
if-no-files-found: warn

codex-live:
runs-on: ubuntu-latest
environment:
name: CI-E2E-CODEX
deployment: false
env:
PR_NUMBER: ${{ inputs.pr_number }}
KEEP_TEST_DIR: "1"
TRIGGER_SOURCE: ${{ github.event_name }}
PR_NUMBER: ${{ github.event.pull_request.number }}
DISPATCH_PR_NUMBER: ${{ inputs.pr_number }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
steps:
- name: Check required secret
Expand All @@ -122,9 +153,21 @@ jobs:
uses: actions/github-script@v7
with:
script: |
const prNumber = Number(process.env.PR_NUMBER);
const triggerSource = process.env.TRIGGER_SOURCE;
let rawPrNumber;

if (triggerSource === "pull_request") {
rawPrNumber = process.env.PR_NUMBER;
} else if (triggerSource === "workflow_dispatch") {
rawPrNumber = process.env.DISPATCH_PR_NUMBER;
} else {
core.setFailed(`Unsupported trigger source: ${JSON.stringify(triggerSource)}`);
return;
}

const prNumber = Number(rawPrNumber);
if (!Number.isInteger(prNumber) || prNumber <= 0) {
core.setFailed(`workflow_dispatch input pr_number must be a positive integer, got ${JSON.stringify(process.env.PR_NUMBER)}`);
core.setFailed(`${triggerSource} PR number must be a positive integer, got ${JSON.stringify(rawPrNumber)}`);
return;
}

Expand All @@ -150,6 +193,7 @@ jobs:

await core.summary
.addHeading("Runtime Live E2E Provenance")
.addRaw(`- Trigger source: ${triggerSource}\n`)
.addRaw(`- PR number: #${prNumber}\n`)
.addRaw(`- Tested workflow SHA: \`${context.sha}\`\n`)
.addRaw(`- Current PR head SHA: \`${pull.head.sha}\`\n`)
Expand All @@ -169,22 +213,36 @@ jobs:

- uses: astral-sh/setup-uv@v6

- name: Prepare live artifact root
run: |
mkdir -p "$RUNNER_TEMP/spacedock-live/$GITHUB_JOB"
echo "SPACEDOCK_TEST_TMP_ROOT=$RUNNER_TEMP/spacedock-live/$GITHUB_JOB" >> "$GITHUB_ENV"

- name: Install Codex CLI
run: npm install --global @openai/codex

- name: Login Codex with API key
run: |
printenv OPENAI_API_KEY | codex login --with-api-key

- name: Configure git identity
run: |
git config --global user.name "github-actions[bot]"
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --global init.defaultBranch main

- name: Show tool versions
run: |
codex --version
uv --version

- name: Run Codex live suite
run: |
set -euo pipefail
uv run tests/test_gate_guardrail.py --runtime codex
uv run tests/test_rejection_flow.py --runtime codex
uv run tests/test_merge_hook_guardrail.py --runtime codex
run: make test-live-codex

- name: Upload Codex live artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: runtime-live-e2e-codex-live
path: ${{ runner.temp }}/spacedock-live/${{ github.job }}
if-no-files-found: warn
18 changes: 17 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: test-static test-e2e
.PHONY: test-static test-e2e test-live-claude test-live-codex

TEST ?= tests/test_gate_guardrail.py
RUNTIME ?= claude
Expand All @@ -8,3 +8,19 @@ test-static:

test-e2e:
unset CLAUDECODE && uv run $(TEST) --runtime $(RUNTIME)

test-live-claude:
unset CLAUDECODE && set -euo pipefail && \
uv run tests/test_gate_guardrail.py --runtime claude && \
uv run tests/test_rejection_flow.py --runtime claude && \
uv run tests/test_feedback_keepalive.py && \
uv run tests/test_merge_hook_guardrail.py --runtime claude && \
uv run tests/test_push_main_before_pr.py && \
uv run tests/test_rebase_branch_before_push.py
# SKIPPED: test_scaffolding_guardrail.py — FO violates issue-filing guardrail. Track: file new task
# SKIPPED: test_dispatch_completion_signal.py — FO drops SendMessage block. Track: #120

test-live-codex:
uv run tests/test_gate_guardrail.py --runtime codex && \
uv run tests/test_rejection_flow.py --runtime codex && \
uv run tests/test_merge_hook_guardrail.py --runtime codex
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
---
id: 100
title: "status tool: add workflow directory discovery"
status: validation
status: done
source: CL observation — Codex startup uses raw rg for discovery
started: 2026-04-13T15:57:56Z
completed:
verdict:
completed: 2026-04-14T01:08:21Z
verdict: PASSED
score: 0.60
worktree: .worktrees/spacedock-ensign-status-workflow-discovery
worktree:
issue:
pr: #85
archived: 2026-04-14T01:08:37Z
---

The startup procedure (step 2) requires searching for README.md files with `commissioned-by: spacedock@` frontmatter to discover workflow directories. Every runtime (Claude Code, Codex) reimplements this as a raw grep/rg call before it can invoke `status --boot`.
Expand Down
35 changes: 34 additions & 1 deletion docs/plans/build-dispatch-structured-helper.md
Original file line number Diff line number Diff line change
Expand Up @@ -456,7 +456,40 @@ The new instructions in the `## Dispatch Adapter` section, after the team health
| 4 | **Static content test invalidation.** Replacing the adapter's `Agent()` template changes the text that `test_agent_content.py` assertions match. | High (certain) | Low | The implementation must update existing assertions in `test_agent_content.py` that reference the old template wording. Specifically: `test_assembled_claude_first_officer_has_team_health_check` (AC1 asserts `test -f` which stays), `test_assembled_claude_first_officer_dispatch_template_has_team_mode_completion_signal` (must be updated to check the break-glass template or the helper instruction). Each affected test is enumerated in the implementation checklist. |
| 5 | **`parse_stages_block` extension regresses existing callers.** Adding `feedback-to`, `agent`, `fresh` fields to the output dict could break callers that iterate over dict keys. | Low | Low | The extension adds optional keys to dicts in a list. Existing callers use `.get()` or explicit key access — they never iterate all keys. The implementation must verify this by grepping all `parse_stages_block` call sites. |

## Out of scope
## Implementation Notes (gate-approved 2026-04-13)

The ideation gate was approved with seven follow-up items from the staff review. These are mandatory for the implementer to address; they are not optional polish.

1. **Correct the line references to the old dispatch template.** The ideation quotes "lines 48-57" (and in one place "lines 51-57") of `skills/first-officer/references/claude-first-officer-runtime.md` as the replacement target. The reviewer confirmed the actual `Agent(...)` block spans **lines 50-57** (line 50 opens the code block, lines 51-56 hold the Agent call, line 57 closes). Re-verify against the current file at implementation start and update any checklist or assertion that references specific line numbers.

2. **Spell out worktree path and branch derivation in the prompt-assembly rules.** Prompt component #3 (worktree instructions, conditional on `worktree: true`) must name where each value comes from:
- `worktree_path` is read from the entity frontmatter's `worktree` field (relative or absolute — specify one and normalize).
- `branch` is derived as `{worker_key}/{slug}`, where `worker_key` is the dispatch_agent_id with `:` replaced by `-` (per the existing claude-first-officer runtime adapter convention at around line 30-35).
- Cross-reference to Validation Rule 4 (worktree path must exist on disk) so the reader sees which field is being validated.

3. **Resolve the feedback-to detection ambiguity.** Validation Rule 5 fires when the stage being dispatched is a feedback-to target, but the stdin schema as written doesn't carry an explicit "this is a feedback re-dispatch" flag. Pick one of the following and encode it in both the input schema and the rule:
- **Option A (recommended):** FO sets `is_feedback_reflow: true` in stdin when routing a rejection. Helper trusts the flag. Simple, explicit.
- **Option B:** Helper scans the workflow README for any stage whose `feedback-to` field matches the target stage name, then requires `feedback_context` to be present when that pattern holds. More magical; helper must read the README.
- Whichever is chosen, the worked example in the Output JSON Schema section should include a feedback-reflow example alongside the ideation example.

4. **Add a break-glass fallback test.** AC-12 asserts the break-glass prose exists in the runtime adapter. That is not enough. Add one of:
- **Unit test** in `tests/test_claude_team.py` that forces the helper to exit 1 (e.g., via malformed stdin) and asserts the FO's break-glass prose is triggered — this requires a small harness because the FO's recovery path is driven by the runtime prose, not code. If a pure unit test is not feasible, use the static-content harness to assert the break-glass template is reachable and syntactically usable as an `Agent()` call.
- **Static assertion** that the break-glass template in the adapter, when rendered with stub values, produces a valid Python function call — parse it with `ast.parse` as a sanity check.

5. **Split the E2E test line in the test-plan table.** The "high ($2-3)" cost cell is ambiguous. Replace with two lines:
- `test_structured_dispatch_happy_path_e2e` — FO assembles JSON, calls helper, forwards to Agent(), minimal worker completes. Estimated cost **$0.50–$1** on haiku/low.
- `test_structured_dispatch_multistage_e2e` — deferred to #134 (runtime-specific-tests-on-pr) rather than a new standalone E2E, because the multi-stage path is already exercised by `test_dispatch_completion_signal.py` and the full FO pipeline. The implementer should confirm the existing completion-signal test goes green after the helper lands, and note this in the stage report.

6. **Document the FO's input-assembly guardrail.** The helper validates its input, but the FO's prose-based JSON assembly is still a failure vector: if the FO sets `bare_mode: false` when teams are not active (or vice versa), the helper rejects, the FO falls back to break-glass, and the completion signal can be lost. Either:
- Add a one-sentence guardrail note in the new runtime-adapter prose at the place where the FO is told to assemble stdin: "the `bare_mode` field must match the current dispatch context — never infer it from the stage, always from the live team state." and
- Add a static assertion in `tests/test_agent_content.py` that the new prose contains that guardrail sentence verbatim.

7. **Resolve the reuse path (SendMessage) scope question.** The runtime adapter has two dispatch surfaces: initial `Agent()` dispatch and `SendMessage(to="team-lead")` reuse-advance of an already-alive ensign. The ideation only covers the initial `Agent()` path. Decide, in this task, which of the following the implementation ships:
- **Option X (narrow):** `claude-team build` serves only initial `Agent()` dispatch. Reuse continues to use the existing prose-based `SendMessage` template in the runtime adapter. The runtime adapter then has two dispatch surfaces — document this explicitly. Future task to unify.
- **Option Y (wide):** `claude-team build` serves both paths. Output schema gains a `dispatch_kind: "agent"|"send_message"` field (or a sibling `claude-team send` subcommand). Single dispatch surface in the adapter.
- Option X is the lower-risk landing for this task; Option Y is the cleaner end-state. Pick one before implementation starts. If Option X, add a task to the Related section filing the eventual Option Y work.

These notes are all implementable without another ideation cycle. Any item that cannot be resolved during implementation (e.g., discovering a structural blocker for item 7) must be raised to the captain before writing code, not silently deferred.

- Codex and Gemini runtime adapter updates (separate tasks after this lands).
- Generalizing the pattern to other fuzzy-template sites (feedback rejection flow, gate presentation, event loop) — Phase 4 of issue #63.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
id: 146
title: "Release-branch runtime live E2E matrix parameterization"
status: backlog
source: "CL direction during 2026-04-13 session — keep PR live CI on fixed defaults, but allow manual/release full-matrix runtime runs"
score: 0.61
started:
completed:
verdict:
worktree:
issue: #89
pr:
---

## Problem Statement

Task 145 moved the runtime live suite onto PRs with environment-backed approvals and split Claude/Codex environments. That gives operators a good default CI path, but it is intentionally fixed: the PR-triggered workflow runs the default jobs only, and the environment approval UI cannot collect `workflow_dispatch` inputs such as model overrides or matrix selection.

We still need an operator-friendly way to run broader live validation for release branches and targeted manual checks without weakening the default PR path. That follow-up should add dispatch-time parameterization for manual runs while keeping ordinary PR CI simple and stable.

## Recommended Approach

Keep the `pull_request` path opinionated and fixed. Extend `workflow_dispatch` so manual/API-triggered runs can request a broader runtime matrix for release validation.

The likely control surface is:

1. target PR/ref selection
2. default vs full-matrix mode
3. optional Claude model override
4. optional Codex model override
5. any additional release-only axes the live suite needs

Those values should be provided when the workflow is dispatched. Environment approval remains a separate release gate for the already-configured jobs and should not be treated as an input form.

## Scope Notes

- This task is about dispatch-time parameterization for manual and release-branch live runs.
- It should not change the default PR-triggered job set for ordinary pull requests.
- It should document the operator flow clearly: dispatch with inputs, approve the relevant environments, then inspect per-job artifacts and results.
- It may require plumbing model inputs through the current Codex and Claude live test entrypoints so overrides actually reach the invoked runtime.

## Acceptance Criteria

1. Ordinary PR-triggered runs keep the current default runtime jobs and do not require new inputs.
- Test: inspect `.github/workflows/runtime-live-e2e.yml` and confirm the `pull_request` path still runs the default jobs with no extra operator input required.
2. `workflow_dispatch` supports explicit inputs for manual/release live runs, including matrix selection and optional runtime/model overrides.
- Test: inspect the workflow input block and confirm the dispatched jobs derive their runtime configuration from those inputs.
3. The workflow documentation explains that `workflow_dispatch` inputs are supplied at run creation time, not at environment approval time.
- Test: inspect `tests/README.md` and confirm the operator caveat is documented.
4. The live test entrypoints actually honor any new manual/runtime override inputs that the workflow exposes.
- Test: inspect the relevant test scripts/helpers and run targeted offline checks to confirm the workflow wiring matches the harness CLI surface.
5. The repo keeps offline coverage for the workflow structure and dispatch-time parameter logic.
- Test: run the targeted workflow/helper tests covering the added input surface.

## Test Plan

- Workflow-structure inspection: verify the fixed `pull_request` path remains unchanged while `workflow_dispatch` gains the intended parameter surface. Cost/complexity: low. No E2E required.
- Focused offline tests: update workflow/helper tests for the new input block and any runtime-argument plumbing. Cost/complexity: medium. No E2E required.
- Manual release-path smoke: dispatch the workflow with a non-default matrix/model configuration and confirm the resulting jobs reflect the requested inputs before approval. Cost/complexity: medium. E2E required: yes, but only for final validation.
Loading
Loading