Add external public API integration runner (#69)

* Capture install command output in tests

Capture install CLI child output in the dual-backend integration tests
instead of inheriting the test runner streams. The rejection tests now
assert the exact stderr messages, and the success tests assert their
stdout status messages without printing them during passing runs.

Validation:
cargo check
cargo build
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo +nightly fmt

* Add external public API runner smoke slice

Add scripts/public_api_suite.py as a Python MCP stdio runner that accepts a prebuilt mcp-repl binary and verifies the R console smoke path through the public repl tool.

Document the runner, record the active migration plan, and remove duplicate raw Rust R console smoke coverage now covered externally.

Validation:
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl
- python3 -m py_compile scripts/public_api_suite.py
- cargo test --test server_smoke
- cargo test --test docs_contracts docs_index_lists_main_docs
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test
- cargo +nightly fmt
- cargo test --test docs_contracts plans_layout_exists
- git diff --staged --check

* Migrate timeout recovery smoke to public API runner

Add an external public API case that verifies R timeout, busy-input discard, and later recovery through the real MCP stdio surface. Remove the duplicate Rust snapshot smoke test now covered by the external runner, and update testing docs plus the active runner plan.

Validation:
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl --case r-timeout-busy-recovers
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl
- python3 -m py_compile scripts/public_api_suite.py
- cargo test --test write_stdin_batch
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test
- cargo +nightly fmt
- cargo insta pending-snapshots
- git diff --check

* Migrate reset state check to public API runner

Move the R repl_reset state-clearing check into scripts/public_api_suite.py so it runs against a built server over MCP stdio. Drop the duplicate Rust public surface test and update testing docs/plan to point at the runner coverage.

Validation:
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl --case r-reset-clears-state
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl
- cargo test --test repl_surface
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test
- cargo +nightly fmt

* Run public API suite in CI

Add the external Python public API suite as its own post-build CI step on every matrix target. Use the platform-specific debug binary path so the suite runs against the binary Cargo just built.

Update testing docs and the active migration plan to record that CI now runs this external public API check.

Validation:
- cargo build
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test
- cargo +nightly fmt

* Reduce slow timeout waits in tests

Shorten artificial timeout sleeps in write_stdin coverage and batch compatible echo-prefix checks into one public session so the slow behavior suite avoids extra fixed waits and real server startups.

Also tighten the external public API timeout recovery case while preserving the busy-discard and recovery assertions.

Validation:
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test
- cargo +nightly fmt

* Silence routine server startup stderr

Remove the routine server startup stderr notice so passing real-server tests do not inherit status noise. Add a CLI regression that starts the real server with closed stdin, asserts the routine notice is absent, and verifies stderr diagnostics remain.

Validation:
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test
- cargo +nightly fmt

* Add quiet nextest CI profile

Add a checked-in nextest ci profile with quiet passing-test output and a default filter that keeps real client integration binaries out of the ordinary suite. Switch CI's ordinary Rust test step to `cargo nextest run --profile ci --show-progress none`, preserving the Windows serial constraint and the separate real Codex integration `cargo test` step.

Validation:
- cargo nextest run --profile ci --show-progress none
- cargo test --test docs_contracts ci_uses_quiet_nextest_profile_for_routine_suite
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test
- cargo +nightly fmt

* Serialize REPL integration tests under nextest

Add a nextest `repl-integration` group with `max-threads = 1` and assign the
process-backed REPL integration binaries to it. Document that the group is for
timing-sensitive server/worker transitions, and extend the docs contract test
so the CI profile keeps this scheduling rule.

Review finding addressed:

- [P1] Serialize timing-sensitive tests under nextest — /Users/tomasz/github/posit-dev/mcp-repl/.github/workflows/ci.yml:98-98
  On non-Windows this new nextest step runs individual tests concurrently, but these integration tests still use in-process `lock_test_mutex()` to serialize timing-sensitive REPL sessions; nextest launches each test in a separate process, so the lock no longer protects them. I can reproduce `cargo nextest run --profile ci --show-progress none --test write_stdin_behavior` failing on macOS in `follow_up_after_timeout_spills_when_prefix_and_reply_exceed_threshold`, while the same run with `--test-threads 1` passes. Please serialize these tests globally or with nextest test groups before switching CI.

Response:
The CI profile now serializes the affected REPL integration binaries with a
nextest test group, preserving per-test server isolation while limiting host
concurrency for the timing-sensitive suites.

Validation:
- cargo test --test docs_contracts ci_uses_quiet_nextest_profile_for_routine_suite
- cargo nextest show-config test-groups --profile ci --groups repl-integration
- cargo nextest run --profile ci --show-progress none --test write_stdin_behavior
- cargo +nightly fmt
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test

* Move pager smoke into public API runner

Add a pager command scenario to the external Python runner, including per-case server args and environment so it starts mcp-repl in pager mode with a small page size. Remove the duplicated Rust pager smoke test and update the testing docs.

Validation:
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl
- cargo +nightly fmt
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test

* Move interrupt prefix coverage to public runner

Add an external public API case that exercises R control-D restart state clearing and control-C interruption with remaining input against the built MCP server. The interrupt path waits for an explicit marker before sending the control prefix, so the check does not rely on a tiny sleep race.

Remove the matching Rust prefix tests and update the public runner docs.

Validation:
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl --case r-interrupt-restart-prefixes --timeout 45
- cargo test --test interrupt
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test
- cargo +nightly fmt
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl --timeout 45

* Move output bundle coverage to public runner

Add Python public API scenarios for files-mode output bundles, including text-only bundles, count pruning, timeout backfill through a gate, and size-cap omission. Remove duplicated broad Rust integration tests while keeping narrower Rust edge coverage in place.

Validation:
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl --case r-output-bundle-files --case r-output-bundle-size-limit
- cargo test --test write_stdin_behavior timeout_output_bundle_is_disclosed_only_after_poll_crosses_hard_spill_threshold -- --exact
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test
- cargo +nightly fmt
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl
- python3 -m py_compile scripts/public_api_suite.py

* Narrow nextest REPL residue group

Limit the CI nextest `repl-integration` group to the remaining timing-sensitive Rust residue now that pager smoke, interrupt/restart prefixes, and broad files-mode output bundle coverage have moved into the Python public runner.

Update the testing docs, active migration plan, and docs contract so pager flag, R manual, and R vignette binaries stay in the ordinary nextest pool.

Validation:
- cargo test --test docs_contracts ci_uses_quiet_nextest_profile_for_routine_suite
- cargo nextest show-config test-groups --profile ci --groups repl-integration
- cargo nextest run --profile ci --show-progress none --test pager_flags --test r_manuals --test r_vignettes
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test
- cargo +nightly fmt
- cargo nextest run --profile ci --show-progress none

* Pin R public API suite cases

Pin every r-* public API suite case with --interpreter r so an inherited
MCP_REPL_INTERPRETER value cannot redirect those cases to another backend. Add
a regression test that checks all R suite case metadata carries the explicit R
interpreter flag.

Review finding:
[P2] Pin the R interpreter for R public API cases — /Users/tomasz/github/posit-dev/mcp-repl/scripts/public_api_suite.py:782-782
When `MCP_REPL_INTERPRETER` is present in the environment, these `r-*` cases inherit it because the command only adds `--sandbox` and per-case args. Setting `MCP_REPL_INTERPRETER=python` makes the suite launch the Python backend (`r-console-basic` can still pass on `1+1`, while later R-specific cases fail), so the new public API suite no longer reliably exercises R; pass `--interpreter r` for these cases or scrub that env var.

Response:
Added an R suite-case constructor that prepends --interpreter r to every r-* case,
including cases with additional per-case server args.

Validation:
- python3 -m unittest scripts/test_public_api_suite.py (failed before the fix, passes after)
- MCP_REPL_INTERPRETER=python python3 scripts/public_api_suite.py --binary target/debug/mcp-repl
- cargo check
- cargo build
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test
- cargo +nightly fmt

* Separate local and CI nextest profiles

Keep the default nextest profile as the local full test loop so authenticated Codex and Claude integration binaries are included. Add a CI profile filter for unauthenticated workflow runs and remove the CI Codex install/integration step.

Refresh agent/testing docs around the public API suite, nextest usage, validation surfaces, and snapshot workflow. Pin the Codex integration fixture to the Spark model while keeping Claude on haiku, and lock the contract in docs_contracts.

Validation:
- cargo check
- cargo build
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo nextest run --show-progress none
- cargo test
- cargo +nightly fmt
- cargo test --test docs_contracts
- git diff --check

* Detect client auth before live integrations

Codex and Claude live integration tests now preflight CLI launch/auth state and print explicit skip banners when a client cannot run.
The local nextest profile shows successful output for those binaries so skip reasons are visible during normal local runs.
Claude preserves the minimal host CLI environment needed for first-party auth probes while keeping test config isolated.

Validation:
- cargo check
- cargo build
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo nextest run --show-progress none
- cargo test
- cargo +nightly fmt
- git diff --check

* Remove obsolete nextest serial scheduling

Drop the repl-integration nextest test group so local runs use normal nextest scheduling.
Update the docs and docs contract to describe the CI profile as only filtering unauthenticated real-client integrations.
Normalize optional Codex workspace metadata from the wire snapshot so the snapshot is stable under normal parallel scheduling.

Validation:
- cargo test --test docs_contracts
- cargo test --test codex_approvals_tui unix_impl::normalize_wire_snapshot_drops_volatile_turn_metadata_fields
- cargo check
- cargo build
- python3 scripts/public_api_suite.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo nextest run --show-progress none
- cargo test
- cargo +nightly fmt
- git diff --check

* Rename integration test runner

Move the external public API runner under tests/run_integration_tests.py
so the real-binary suite lives with the test harnesses. Update CI,
AGENTS.md, and testing docs to reference the new path, and extend docs
contracts to reject the stale script path.

Validation:
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo nextest run --show-progress none
- cargo test
- cargo +nightly fmt
- git diff --check

* Split cargo test from explicit Rust test suite

Make plain cargo test a small Cargo compatibility check by opting the binary
unit-test target and integration targets out of default Cargo discovery. Add an
explicit Python wrapper around nextest to run those Rust targets, including a CI
profile that excludes live client integrations and a clippy mode for the opted
out targets.

Also stabilize the interrupt prompt-shaped-output test with an explicit marker
instead of a sleep-only fixture, and update CI, AGENTS, docs, and docs-contract
coverage for the new split test entry points.

Validation:
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- python3 tests/run_rust_tests.py --clippy
- python3 tests/run_rust_tests.py --profile default
- cargo test
- cargo +nightly fmt --all -- --check

* Restore real Codex CI integration coverage

Run CI's ordinary Rust suite with nextest directly while keeping all Rust test
targets discoverable by Cargo. Remove the transitional explicit Rust test
wrapper and the Cargo.toml opt-outs that made `cargo test` incomplete.

Install the real Codex CLI in CI and run the Codex integration binary
separately. The Codex smoke test now defaults to backend auto-selection: use
live Spark when `codex login status`, model availability, and local auth are
present; otherwise use the mocked provider. The forced mock and live paths are
documented for targeted validation.

Validation:
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo nextest run --show-progress none
- cargo test
- cargo +nightly fmt --all -- --check
- MCP_REPL_CODEX_BACKEND=mock cargo test -j 1 --test codex_approvals_tui codex_exec_auto_backend_smoke -- --test-threads=1

* Use cargo test for CI Rust suite

Rename the Codex client integration target from codex_approvals_tui to codex_integration so the name matches its broader exec, install, sandbox metadata, and mock-provider coverage.

Remove the nextest profile and CI dependency. CI now installs Codex before the Rust suite, forces MCP_REPL_CODEX_BACKEND=mock, and runs the standard cargo test path with quiet output. This keeps Rust test discovery on Cargo's default path instead of maintaining two Rust runners.

Validation:
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- MCP_REPL_CODEX_BACKEND=mock cargo test -j 1 --test codex_integration codex_exec_auto_backend_smoke -- --test-threads=1
- cargo test --test docs_contracts
- cargo test --quiet
- cargo +nightly fmt --all -- --check

* Shorten interrupt drain setup timeout

Use a 5ms timeout for the prompt-shaped child-output interrupt drain
setup request. This keeps the setup call focused on entering the busy
state before the child output is likely to be drained into the initial
timeout response.

Validation:
- MCP_REPL_CODEX_BACKEND=mock cargo test --quiet --test interrupt files_interrupt_drain_preserves_prompt_shaped_child_stdout -- --nocapture
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- MCP_REPL_CODEX_BACKEND=mock cargo test --quiet
- cargo +nightly fmt

* Stabilize timeout bundle CI tests

Shorten the initial timeout in the hidden timeout bundle spill tests so
slow first-call startup cannot let the initial reply reach the oversized
output disclosure path before the busy follow-up.

Update the Linux Codex wire snapshot for the current Codex CLI metadata,
which no longer includes x-codex-turn-metadata.workspaces.

Validation:
- MCP_REPL_CODEX_BACKEND=mock cargo test --test write_stdin_behavior busy_follow_up_reuses_hidden_timeout_bundle_when_it_first_spills -- --nocapture
- MCP_REPL_CODEX_BACKEND=mock cargo test --test codex_integration codex_exec_wire_sandbox_state_meta -- --nocapture
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- MCP_REPL_CODEX_BACKEND=mock cargo test --quiet
- cargo +nightly fmt

* Clarify public API runner sandbox plan

Clarify that the external Python runner is not itself sandboxed, but each spawned mcp-repl binary still owns and must exercise the sandbox contract.

The next slice now calls for workspace-write sandbox coverage in the Python runner before continuing representative real-binary migration out of Rust.

Validation: cargo test --test docs_contracts

Author: t-kalinowski

.github/workflows/ci.yml

@@ -78,16 +78,17 @@ jobs:
       - name: cargo build
         run: cargo build
 
-      - name: cargo clippy
-        run: cargo clippy --all-targets --all-features -- -D warnings
-
-      - name: cargo test
+      - name: Python public API suite
         if: matrix.os != 'windows-2022'
-        run: cargo test
+        run: python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
 
-      - name: cargo test (windows serial)
+      - name: Python public API suite (windows)
         if: matrix.os == 'windows-2022'
-        run: cargo test -j 1 -- --test-threads=1
+        shell: pwsh
+        run: python tests/run_integration_tests.py --binary target/debug/mcp-repl.exe
+
+      - name: cargo clippy
+        run: cargo clippy --all-targets --all-features -- -D warnings
 
       - name: Install Codex CLI
         if: matrix.os != 'windows-2022'
@@ -110,13 +111,17 @@ jobs:
           $env:PATH = "$npmPrefix;$env:PATH"
           & (Join-Path $npmPrefix "codex.cmd") --version
 
-      - name: cargo test (real codex integrations)
+      - name: cargo test
         if: matrix.os != 'windows-2022'
-        run: cargo test -j 1 --test codex_approvals_tui -- --test-threads=1
+        env:
+          MCP_REPL_CODEX_BACKEND: mock
+        run: cargo test --quiet
 
-      - name: cargo test (real codex integrations, windows serial)
+      - name: cargo test (windows serial)
         if: matrix.os == 'windows-2022'
-        run: cargo test -j 1 --test codex_approvals_tui -- --test-threads=1
+        env:
+          MCP_REPL_CODEX_BACKEND: mock
+        run: cargo test -j 1 --quiet -- --test-threads=1
 
       - name: cargo +nightly fmt
         run: cargo +nightly fmt --all -- --check

AGENTS.md

@@ -7,19 +7,26 @@ Keep this file short. It is a table of contents, not the full manual.
 - If you modified code, run all required checks before replying:
   - `cargo check`
   - `cargo build`
+  - `python3 tests/run_integration_tests.py --binary target/debug/mcp-repl`
   - `cargo clippy --all-targets --all-features -- -D warnings`
-  - `cargo test`
+  - `cargo test --quiet`
   - `cargo +nightly fmt`
+- For docs-only changes, run the narrow docs validation that covers the edited
+  files, usually `cargo test --test docs_contracts`.
+- When changing Codex backend selection or CI real-client wiring, also run:
+  - `MCP_REPL_CODEX_BACKEND=mock cargo test -j 1 --test codex_integration codex_exec_auto_backend_smoke -- --test-threads=1`
 - Treat all clippy warnings as failures. Do not leave warning cleanup for later.
 - Never pass `--vanilla` to `R` or `Rscript` unless the user explicitly asks for it.
 
 ## Start Here
 
 - `docs/index.md`: source-of-truth map for repository docs.
-- `docs/architecture.md`: subsystem map for the binary, worker, sandbox, and eval surfaces.
+- `docs/architecture.md`: subsystem map for the CLI, server, worker, sandbox, output, and validation surfaces.
 - `docs/testing.md`: public verification surface and snapshot workflow.
 - `docs/debugging.md`: debug logs, `--debug-repl`, and stdio tracing.
 - `docs/sandbox.md`: sandbox modes and writable-root policy.
+- `docs/output_timeline.md`: visible output ordering across sideband and raw streams.
+- `docs/worker_sideband_protocol.md`: current server/worker IPC contract.
 - `docs/plans/AGENTS.md`: when to create checked-in execution plans.
 
 ## Glossary
@@ -41,8 +48,8 @@ Keep this file short. It is a table of contents, not the full manual.
 - Sandbox metadata: Codex per-tool-call `_meta["codex/sandbox-state-meta"]` used by `--sandbox inherit` to choose the effective worker sandbox for that call.
 - Writable root: An absolute path that a `workspace-write` worker may write, subject to forced read-only subpaths like `.git`, `.codex`, and `.agents`.
 - Session temp directory: The server-allocated per-session temp path exposed to the worker as `TMPDIR` and `MCP_REPL_R_SESSION_TMPDIR`.
-- Sideband IPC: The JSON-lines server/worker pipe for structural facts such as `readline_start`, `readline_result`, `plot_image`, `request_end`, and `session_end`.
-- stdout/stderr pipes: The normal process output streams captured by the server. They are the authoritative visible text source; sideband only helps interpret them.
+- Sideband IPC: The JSON-lines server/worker pipe for structural facts such as `readline_start`, `readline_input`, `readline_discard`, `output_text`, `plot_image`, and `session_end`.
+- Raw output capture: The stdout/stderr pipes or PTY stream captured by the server for unowned visible text. Sideband carries worker-owned text and structural facts.
 - Output timeline: The server-side reconstruction of visible output order from captured stdout/stderr plus sideband facts.
 - Server-owned: State, files, or notices created and retained by the main server process, not by the runtime or the worker. Use this for output bundles, response finalization, debug logs, and server temp roots.
 - Worker-originated text: Text that came from the worker REPL or worker child processes and can be written to `transcript.txt`.
@@ -60,7 +67,10 @@ Keep this file short. It is a table of contents, not the full manual.
   - `cargo insta test`
   - `cargo insta pending-snapshots`
   - `cargo insta review` or `cargo insta accept` / `cargo insta reject`
-- CI-style validation: `cargo insta test --check --unreferenced=reject`
+- CI-style validation: `cargo insta test --check`
+- Do not add `--unreferenced=reject` to the general snapshot check; this
+  repository keeps valid platform-specific snapshots that are unreferenced on
+  other platforms.
 - For broad intentional snapshot migrations: `cargo insta test --force-update-snapshots --accept`
 - Do not delete `tests/snapshots/*.snap.new` manually. Use `cargo insta reject`.
 

docs/architecture.md

@@ -63,7 +63,15 @@ The repository is organized around a few concrete subsystems rather than deep pa
 
 ### Validation harnesses
 
-- `tests/` is the primary public validation surface. The tests exercise tool behavior, snapshots, sandboxing, and client integrations through the exposed MCP interface.
+- `tests/run_integration_tests.py` starts an already-built `mcp-repl` binary and
+  exercises public MCP tools over stdio. It covers representative real-binary
+  behavior that should not depend on Rust internals.
+- `tests/` contains the Rust public API, snapshot, sandbox, backend, install,
+  protocol-worker, and client-integration suites. Most tests exercise behavior
+  through the exposed MCP interface using the shared harness in `tests/common/`.
+- CI uses Cargo's standard Rust test runner after installing the real Codex CLI,
+  with the Codex backend forced to the mocked provider. The tests should not
+  depend on special local scheduling.
 
 ## Design Constraints
 

docs/debugging.md

@@ -84,7 +84,7 @@ Useful environment variables:
 
 ## External wire trace proxy
 
-The built-in event log only sees what reaches `mcp-repl` after startup. If you need the exact stdio traffic between an MCP client and the server, use the external proxy in [scripts/mcp-stdio-trace.py](/Users/tomasz/github/t-kalinowski/mcp-repl/scripts/mcp-stdio-trace.py).
+The built-in event log only sees what reaches `mcp-repl` after startup. If you need the exact stdio traffic between an MCP client and the server, use the external proxy in [scripts/mcp-stdio-trace.py](../scripts/mcp-stdio-trace.py).
 
 What it does:
 

docs/plans/active/public-api-runner.md

@@ -0,0 +1,67 @@
+# External Public API Runner
+
+## Summary
+
+- Move public MCP behavior checks, including sandbox-visible real-binary behavior, toward an external Python runner that starts a built `mcp-repl` binary over stdio.
+- Keep Rust tests for unit contracts, snapshot normalization, protocol-worker conformance, platform-specific mechanics, and behavior that is not yet covered externally.
+
+## Status
+
+- State: active
+- Last updated: 2026-05-18
+- Current phase: implementation
+
+## Current Direction
+
+- Grow the minimal Python runner with small, real-client scenarios that speak MCP directly with newline-delimited JSON-RPC.
+- Treat sandboxing as product behavior for the external suite. The test runner process is outside the sandbox, but each case starts a built `mcp-repl` binary with an explicit sandbox state and verifies the worker is launched inside that policy through public MCP calls.
+- Reintroduce sandbox coverage in the Python runner now, starting with the default `workspace-write` behavior and then adding read-only or full-access contrasts where they prove public behavior.
+- Keep each migrated case focused enough that matching Rust integration coverage can be removed or reduced in the same change.
+- Use `danger-full-access` only for individual external cases whose purpose is unrelated to sandboxing and where disabling sandbox enforcement does not hide the product behavior under test.
+- Keep existing Rust tests discoverable by `cargo test` until their scenario is migrated or removed in the same change that adds equivalent Python coverage.
+
+## Long-Term Direction
+
+- Migrate representative public API integration scenarios out of Rust when the Python runner covers the same real-binary behavior, including sandbox behavior that is observable through public MCP tool calls.
+- Keep protocol-worker conformance tests, Rust-only contract tests, and deeply platform-specific sandbox launch mechanics in Rust unless there is a clearer public external scenario for the same contract.
+
+## Phase Status
+
+- Phase 0: completed - add the runner shell and first R console smoke case.
+- Phase 1: completed - migrate another small real-client scenario with timeout or busy-worker behavior.
+- Phase 2: completed - run the external suite in CI after the debug binary is built.
+- Phase 3: pending - reintroduce sandbox scenarios in the Python runner and continue migrating duplicate real-binary Rust integration coverage case by case.
+
+## Locked Decisions
+
+- The external suite must accept a prebuilt binary path instead of building the binary itself.
+- The runner should call MCP tools over stdio and avoid internal Rust helpers.
+- CI runs the external suite as its own step after `cargo build` on each matrix target.
+- Do not opt Rust test targets out of Cargo discovery in anticipation of future migration work.
+
+## Open Questions
+
+- Which sandbox scenarios have public external equivalents and which should remain Rust-only launch or platform-mechanics coverage.
+- Which additional public scenarios should migrate into the external suite before the parent migration is complete.
+
+## Next Safe Slice
+
+- Add a Python-runner sandbox case that starts the binary under `workspace-write`, proves an in-workspace write succeeds, and proves an out-of-policy write is blocked through the public `repl` tool.
+- In the same or next small slice, migrate another representative real-binary Rust integration scenario to the Python runner and remove or reduce only the matching Rust coverage.
+
+## Stop Conditions
+
+- Stop if a migrated scenario requires internal server state inspection instead of public MCP requests.
+- Stop if runner behavior needs platform-specific process supervision beyond the simple stdio client.
+
+## Decision Log
+
+- 2026-05-17: Chose a narrow first slice with one R `repl` smoke case to prove the runner can initialize the real binary and call public tools before moving more complex scenarios.
+- 2026-05-17: Added an R timeout/busy/recovery case to the external runner and removed the matching Rust snapshot smoke test.
+- 2026-05-17: Added an R `repl_reset` state-clearing case to the external runner and removed the duplicate Rust public surface test.
+- 2026-05-17: Added the external public API suite to the cross-platform CI workflow as a separate post-build step.
+- 2026-05-17: Added an R interrupt/restart-prefix scenario with explicit interrupt readiness polling and removed duplicate Rust prefix tests.
+- 2026-05-17: Added files-mode output-bundle scenarios for text bundles, pruning, timeout backfill, and size-cap omission, then removed duplicate broad Rust integration coverage.
+- 2026-05-17: Removed obsolete serial scheduling after verifying the remaining Rust REPL binaries pass under normal Cargo test scheduling.
+- 2026-05-18: Reaffirmed that unmigrated Rust scenarios must remain discoverable by `cargo test`; migrations should replace Rust coverage with equivalent Python coverage in the same change, not disable tests ahead of time.
+- 2026-05-18: Clarified that the external runner itself is not sandboxed, but the spawned `mcp-repl` binary still owns the sandbox contract; the next slice should restore sandbox coverage in the Python runner starting with `workspace-write`.

docs/testing.md

@@ -5,14 +5,14 @@ This file is the entrypoint for deciding how to verify a change.
 
 ## Core Test Surface
 
-- `tests/repl_surface.rs`: basic `repl` and `repl_reset` behavior.
-- `tests/repl_surface.rs` and `tests/python_backend.rs`: IPC ownership coverage. Only the main worker may own sideband fds; user-spawned children must not. `tests/python_backend.rs` also covers detached-idle oversized-output behavior, Unix Python PTY-backed C stdio, CPython `input()` through the readline path, and the absence of direct-fd stdin shims through the public `repl` API.
-- `tests/server_smoke.rs`: end-to-end MCP session smoke coverage.
-- `tests/write_stdin_behavior.rs`: timeout polling, oversized text replies, and transcript-file behavior through the public `repl` API.
+- `tests/run_integration_tests.py`: external real-binary checks over MCP stdio, including basic R `repl`, pager command handling, files-mode output bundles, timeout/busy recovery, interrupt/restart prefixes, and `repl_reset` state clearing.
+- `tests/common/`: shared Rust MCP harness for public tool calls, transcript snapshots, sandbox assertions, and client-install fixtures.
+- `tests/repl_surface.rs`, `tests/server_smoke.rs`, `tests/mcp_transcripts.rs`, and `tests/write_stdin_*.rs`: core `repl`/`repl_reset` behavior, timeout polling, oversized text replies, transcript-file behavior, and snapshot coverage through the public tool API.
+- `tests/pager*.rs` and `tests/oversized_output_cli.rs`: pager mode, files mode, and oversized-output CLI behavior.
+- `tests/python_*.rs`, `tests/r_*.rs`, `tests/plot_images.rs`, and `tests/python_plot_images.rs`: backend-specific public behavior, help/manual surfaces, PTY-backed Python readline behavior, and image output.
 - `tests/zod_protocol.rs`: protocol-worker conformance, including PTY launch with sideband IPC kept separate from visible PTY output.
 - `tests/sandbox.rs` and `tests/sandbox_state_updates.rs`: sandbox policy behavior and Codex per-tool-call sandbox metadata.
-- `tests/plot_images.rs` and `tests/python_plot_images.rs`: plot/image behavior through the public tool surface.
-- `tests/codex_approvals_tui.rs` and `tests/claude_integration.rs`: client integration coverage.
+- `tests/install_*.rs`, `tests/codex_integration.rs`, and `tests/claude_integration.rs`: install-path and real client integration coverage.
 - `tests/docs_contracts.rs`: docs map and snapshot-facing documentation contracts.
 
 ## Snapshot Workflow
@@ -22,18 +22,115 @@ This file is the entrypoint for deciding how to verify a change.
   - `cargo insta test`
   - `cargo insta pending-snapshots`
   - `cargo insta review` or `cargo insta accept` / `cargo insta reject`
+- CI-style validation: `cargo insta test --check`
+- Do not add `--unreferenced=reject` to the general snapshot check; this
+  repository keeps valid platform-specific snapshots that are unreferenced on
+  other platforms.
 - Do not delete `tests/snapshots/*.snap.new` manually. Use `cargo insta reject`.
 
+## External Public API Suite
+
+Build the binary first, then run the Python suite:
+
+```sh
+cargo build
+python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
+```
+
+The runner starts the real server over MCP stdio and calls public tools only. It
+uses `--sandbox danger-full-access` by default so the suite stays focused on
+client protocol behavior rather than sandbox policy.
+
+Use `--case <name>` to run one public API case while iterating.
+
+CI runs this suite after `cargo build` in the main cross-platform workflow,
+using the debug binary built for each matrix target.
+
+## Rust Suite
+
+Use Cargo's standard Rust test runner:
+
+```sh
+cargo test
+```
+
+The Rust suite uses plain `cargo test` as its single runner. Plain `cargo test`
+remains the full Cargo compatibility path. It must continue to discover the
+binary unit tests and Rust integration targets. CI passes Cargo's `--quiet`
+flag to keep successful logs compact.
+
+```sh
+cargo test --quiet
+```
+
+CI installs Codex before `cargo test` and sets `MCP_REPL_CODEX_BACKEND=mock`,
+so the Codex integration target runs through the mocked provider as part of the
+ordinary Rust suite. Windows keeps the Rust suite fully serial with `-j 1` and
+`--test-threads=1`.
+
+Do not opt Rust test targets out of Cargo discovery in anticipation of a future
+Python migration; migrate a scenario only when the Rust coverage is deleted or
+reduced in the same change that adds equivalent external coverage.
+
+## Real Client Integrations
+
+CI installs Codex before the Rust suite. The Codex CI integration does not
+require OpenAI authentication because the test config points Codex at a local
+mock provider.
+
+By default, the Codex integration uses `MCP_REPL_CODEX_BACKEND=auto`: it checks
+whether Codex is logged in, checks whether `gpt-5.3-codex-spark` is available,
+and uses that live backend when both checks pass. Otherwise it uses the mocked
+provider. Set `MCP_REPL_CODEX_BACKEND=live` or `MCP_REPL_CODEX_BACKEND=mock`
+to force one path.
+
+When changing Codex backend selection or CI real-client wiring, run the forced
+mock path explicitly:
+
+```sh
+MCP_REPL_CODEX_BACKEND=mock cargo test -j 1 --test codex_integration codex_exec_auto_backend_smoke -- --test-threads=1
+```
+
+To validate the authenticated live path directly on a machine with Spark access:
+
+```sh
+MCP_REPL_CODEX_BACKEND=live cargo test -j 1 --test codex_integration codex_exec_auto_backend_smoke -- --test-threads=1
+```
+
+Local full verification includes the Codex and Claude integration binaries when
+those clients are installed. Codex uses the Spark model
+(`gpt-5.3-codex-spark`) in its isolated test config. Claude uses `haiku`.
+If a required client binary is unavailable, the matching integration test prints
+a skip banner with the reason. Codex backend selection prints a `CODEX` banner
+showing whether the test selected live Spark or the mocked provider.
+
+To run only those integrations:
+
+```sh
+cargo test --quiet --test codex_integration --test claude_integration
+```
+
+CI runs the Codex integration target as part of `cargo test`; Claude integration
+remains local because provider authentication is unavailable in CI.
+
 ## Full Verification Before Replying
 
 If you modify code, run:
 
 - `cargo check`
 - `cargo build`
+- `python3 tests/run_integration_tests.py --binary target/debug/mcp-repl`
 - `cargo clippy --all-targets --all-features -- -D warnings`
-- `cargo test`
+- `cargo test --quiet`
 - `cargo +nightly fmt`
 
+For docs-only changes, run the narrow validation that covers the edited docs.
+For agent-facing docs, that is usually:
+
+```sh
+cargo test --test docs_contracts
+```
+
 ## Debug-Then-Validate Loop
 
 When behavior is unclear:

src/server.rs

@@ -785,7 +785,6 @@ pub async fn run(
     sandbox_plan: SandboxCliPlan,
     oversized_output: OversizedOutputMode,
 ) -> Result<(), Box<dyn std::error::Error>> {
-    eprintln!("starting mcp-repl server");
     let backend = worker_launch.builtin_backend().unwrap_or(Backend::R);
     crate::event_log::log(
         "server_run_begin",