[P2.2] Generic adapter test harness (mock CLIs)

[frankbria]

Phase 2 — Capabilities & safety. Reusable test scaffolding so every provider PR is
cheap to validate.

Scope

• A bats helper that, given an adapter, asserts: (a) command-build produces expected
  argv for representative options, (b) a recorded provider-output fixture normalizes to
  the expected internal struct, (c) the capabilities record is well-formed.
• A mock-CLI pattern per provider (executable stub emitting canned structured output;
  must take >1s per call per the e2e early-failure rule).
• Fixtures dir for raw provider outputs.

Acceptance

• Harness used by the Claude reference adapter; ready for [P3.1]+ to plug in.

Depends on: [P1.2]. Blocks: [P3.1], [P4.1]–[P4.5].

[github-actions]

Triage: Feature / Enhancement

Category: Enhancement (feature work) — applied the enhancement label.

Reasoning:

• This is a well-scoped engineering work item that proposes building new reusable test scaffolding (a bats adapter-test helper, per-provider mock-CLI pattern, and a fixtures directory). That is feature/infrastructure work, not a bug report or support question.
• It is part of the structured multi-provider roadmap (already carries the multi-provider domain label) — [P2.2], with explicit Depends on: [P1.2] and Blocks: [P3.1], [P4.1]–[P4.5].
• Not a duplicate: [P3.1] Codex adapter (pilot) — build + parse + capabilities + tests + docs #317 ([P3.1]) depends on this harness rather than duplicating it; no other open issue covers this scaffolding.

Recommended next steps:

• Ensure [P1.2] (the dependency) is resolved before starting, since the harness consumes the adapter interface it defines.
• When implementing, follow the repo's E2E mock-CLI convention noted in the scope: the mock CLI stub must take >1s per call to avoid Ralph's early-failure detection (sub-second exits are treated as startup failures — see tests/e2e/helpers/e2e_helper.bash).
• Land the harness wired up against the Claude reference adapter so [P3.1]+ can plug in, per the acceptance criteria.

No assignment change made — assignment is reserved for confirmed bugs per the triage policy.

Generated by Issue Triage Assistant for issue #316 · opus48 311.9K · ◷

[coderabbitai]

Coding Plan

Summary

• Create a dedicated adapter_harness.bash helper defining three assertion functions that formalize the adapter testing contract: command-build argv verification, output-to-analysis normalization, and capabilities record validation
• Generalize the existing E2E mock CLI pattern into a reusable installer function with call tracking, per-call responses, and enforced >1s sleep
• Establish tests/fixtures/providers/ as the canonical location for provider output samples, expected analysis results, and capabilities records
• Validate the entire harness by implementing Claude as the reference adapter with comprehensive tests covering all three JSON formats plus edge cases

Design Choices

Design Choice 1: Capabilities record format

Options Considered:

1. Define capabilities as a JSON file per provider in tests/fixtures/providers/<name>/capabilities.json
2. Define capabilities as bash associative arrays declared in the adapter test file
3. Define capabilities inline within the harness assertion function

Chosen Option: 1

Rationale: JSON files provide a language-agnostic contract that future non-bash tooling can consume and that P3.1+ adapters will be required to supply.

Design Choice 2: Harness location and scope

Options Considered:

1. Place in tests/helpers/adapter_harness.bash for use across unit, integration, and e2e tests
2. Place in tests/e2e/helpers/ since mock CLIs are subprocess-based like e2e tests

Chosen Option: 1

Rationale: The harness assertions for command-build and capabilities validation don't require subprocesses and should be available to unit/integration tests. The mock CLI installer can still be used by e2e tests that load it.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Adapter Test Harness Infrastructure

Create the foundational test infrastructure that defines the adapter testing contract. This phase establishes the patterns that all future provider adapters must satisfy.

Task 1: Create adapter harness helper

Define the core assertion functions that test adapter compliance with the Ralph provider contract.

• Create tests/helpers/adapter_harness.bash with three main assertion functions
• assert_command_argv: Takes a provider name, a command-build function reference, input options map, and expected argv array; calls the build function and compares resulting argv using existing assert_equal patterns
• assert_normalizes_to: Takes a provider output fixture file and expected analysis JSON; runs the fixture through parse_json_response/analyze_response and compares key fields (exit_signal, has_completion_signal, files_modified, has_permission_denials, etc.)
• assert_capabilities_valid: Takes a provider name and validates its capabilities JSON against required schema (provider name, cli_command, supported output_formats array, supports object with boolean flags)
• Include helper load_provider_capabilities() to read and cache capabilities JSON from fixtures directory

Task 2: Create generic mock CLI installer

Generalize the E2E mock CLI pattern into a reusable function that any provider can use.

• Add install_provider_mock() function to tests/helpers/adapter_harness.bash
• Accept parameters: provider name, CLI command name, mock directory path
• Generate executable script at $mock_dir/bin/$cli_cmd following the E2E pattern: call counter in calls/.count, argv logging to calls/argv_N.log, per-call responses from responses/N.out with exit codes from responses/N.exit
• Enforce mandatory sleep >1s (configurable via responses/N.sleep, default 1.5s) to satisfy the early-failure detection rule
• Include queue_provider_response() helper to write fixture content to the correct response file
• Include get_mock_call_count() and get_mock_argv() helpers for test assertions

Task 3: Create provider fixtures directory structure

Establish the canonical location and organization for provider output fixtures.

• Create tests/fixtures/providers/ directory structure
• Create tests/fixtures/providers/README.md documenting the expected structure: <provider>/outputs/ for raw CLI outputs, <provider>/expected/ for expected normalized analysis, <provider>/capabilities.json for the capabilities record
• Define the capabilities JSON schema in the README: required fields (provider, cli_command, output_formats), optional fields (supports, required_version, session_continuity)

🤖 Prompt for AI agents

Create the foundational adapter test harness infrastructure. The goal is to
establish reusable BATS helpers and fixture directory structures that define the
adapter testing contract.

**File: `tests/helpers/adapter_harness.bash`**
- Create this new file with three core assertion functions and supporting
helpers:
- `assert_command_argv`: Takes a provider name, a command-build function
reference, input options map, and expected argv array; calls the build function
and compares resulting argv using existing `assert_equal` patterns
- `assert_normalizes_to`: Takes a provider output fixture file and expected
analysis JSON; runs the fixture through `parse_json_response`/`analyze_response`
and compares key fields (`exit_signal`, `has_completion_signal`,
`files_modified`, `has_permission_denials`, etc.)
- `assert_capabilities_valid`: Takes a provider name and validates its
capabilities JSON against required schema (provider name, cli_command, supported
output_formats array, supports object with boolean flags)
- `load_provider_capabilities()`: Reads and caches capabilities JSON from
fixtures directory
- `install_provider_mock()`: Accepts parameters (provider name, CLI command
name, mock directory path); generates an executable script at
`$mock_dir/bin/$cli_cmd` following the E2E pattern with call counter in
`calls/.count`, argv logging to `calls/argv_N.log`, per-call responses from
`responses/N.out` with exit codes from `responses/N.exit`; enforces mandatory
sleep >1s (configurable via `responses/N.sleep`, default 1.5s)
- `queue_provider_response()`: Writes fixture content to the correct response
file
  - `get_mock_call_count()` and `get_mock_argv()`: Helpers for test assertions

**Directory & Documentation: `tests/fixtures/providers/`**
- Create the `tests/fixtures/providers/` directory structure
- Create `tests/fixtures/providers/README.md` documenting:
- Expected structure: `<provider>/outputs/` for raw CLI outputs,
`<provider>/expected/` for expected normalized analysis,
`<provider>/capabilities.json` for the capabilities record
- Capabilities JSON schema with required fields (`provider`, `cli_command`,
`output_formats`) and optional fields (`supports`, `required_version`,
`session_continuity`)

Phase 2: Claude Reference Adapter Implementation

Implement the Claude adapter test suite using the harness, serving as both validation of the harness and the reference implementation for future adapters.

Task 1: Create Claude output fixtures

Populate fixture files covering Claude's three JSON output formats and key edge cases.

• Create tests/fixtures/providers/claude/outputs/ with representative samples:
  • flat_json_complete.json: Legacy flat format with exit_signal: true
  • flat_json_in_progress.json: Flat format with status: IN_PROGRESS
  • cli_object_success.json: Claude CLI {result, sessionId, metadata} format
  • cli_array_streaming.json: JSONL array format [{type:"system"}, {type:"result"}]
  • permission_denial.json: Output containing has_permission_denials indicators
  • text_with_ralph_status.txt: Plain text with embedded ---RALPH_STATUS--- block
• Create tests/fixtures/providers/claude/expected/ with corresponding expected analysis JSON for each output fixture
• Extract fixture content from existing inline heredocs in tests/unit/test_json_parsing.bats and tests/helpers/fixtures.bash where applicable

Task 2: Define Claude capabilities record

Create the formal capabilities declaration for the Claude adapter.

• Create tests/fixtures/providers/claude/capabilities.json declaring:
  • provider: "claude", cli_command: "claude"
  • output_formats: ["flat_json", "cli_object", "cli_array", "text"]
  • supports: json_output, session_continuity, model_selection, effort_levels, append_system_prompt, allowed_tools, resume_session
  • required_version: minimum Claude CLI version (reference --version check in existing code)
• Document each capability's meaning in comments or accompanying README section

Task 3: Create Claude adapter test file

Write the BATS test file that uses the harness to validate Claude as the reference adapter.

• Create tests/integration/test_adapter_claude.bats (integration level since it sources lib files)
• Load test_helper, adapter_harness, and source lib/response_analyzer.sh
• Test group 1 - Command-build assertions: Test that build_claude_command() produces correct argv for representative option combinations (model override, effort level, resume session, allowed tools, append system prompt)
• Test group 2 - Normalization assertions: For each fixture in claude/outputs/, call assert_normalizes_to against corresponding claude/expected/ file
• Test group 3 - Capabilities validation: Call assert_capabilities_valid "claude" to verify the capabilities record schema
• Include negative tests: malformed output gracefully fails normalization, missing capabilities fields fail validation

Task 4: Create Claude mock CLI for harness validation

Implement a Claude-specific mock using the generic installer to validate the mock pattern works end-to-end.

• Add setup/teardown in test_adapter_claude.bats that calls install_provider_mock "claude" "claude" "$TEST_DIR/mock"
• Write at least one test that queues a fixture response, runs a simulated loop iteration, and asserts on call count and argv capture
• Verify the >1s sleep requirement is enforced (test should take >1s wall time)
• This validates the harness mock infrastructure is ready for P3.1+ providers

🤖 Prompt for AI agents

Implement the Claude reference adapter using the harness built in Phase 1. This
phase validates the harness by implementing Claude as the reference adapter with
comprehensive fixtures, a capabilities record, and integration tests.

**Fixtures: `tests/fixtures/providers/claude/outputs/`**
- Create the following fixture files (extract content from existing inline
heredocs in `tests/unit/test_json_parsing.bats` and
`tests/helpers/fixtures.bash` where applicable):
  - `flat_json_complete.json`: Legacy flat format with `exit_signal: true`
  - `flat_json_in_progress.json`: Flat format with `status: IN_PROGRESS`
  - `cli_object_success.json`: Claude CLI `{result, sessionId, metadata}` format
- `cli_array_streaming.json`: JSONL array format `[{type:"system"},
{type:"result"}]`
- `permission_denial.json`: Output containing `has_permission_denials`
indicators
- `text_with_ralph_status.txt`: Plain text with embedded `---RALPH_STATUS---`
block

**Fixtures: `tests/fixtures/providers/claude/expected/`**
- Create corresponding expected analysis JSON files for each output fixture
above

**Capabilities: `tests/fixtures/providers/claude/capabilities.json`**
- Declare:
  - `provider`: "claude", `cli_command`: "claude"
  - `output_formats`: ["flat_json", "cli_object", "cli_array", "text"]
- `supports`: `json_output`, `session_continuity`, `model_selection`,
`effort_levels`, `append_system_prompt`, `allowed_tools`, `resume_session`
- `required_version`: minimum Claude CLI version (reference the `--version`
check in existing code)
- Document each capability's meaning in comments or an accompanying README
section

**Test File: `tests/integration/test_adapter_claude.bats`**
- Load `test_helper`, `adapter_harness`, and source `lib/response_analyzer.sh`
- Test group 1 - Command-build assertions: Use `assert_command_argv` to test
that `build_claude_command()` produces correct argv for representative option
combinations (model override, effort level, resume session, allowed tools,
append system prompt)
- Test group 2 - Normalization assertions: For each fixture in
`claude/outputs/`, call `assert_normalizes_to` against the corresponding
`claude/expected/` file
- Test group 3 - Capabilities validation: Call `assert_capabilities_valid
"claude"` to verify the capabilities record schema
- Include negative tests: malformed output gracefully fails normalization,
missing capabilities fields fail validation
- Add setup/teardown that calls `install_provider_mock "claude" "claude"
"$TEST_DIR/mock"`
- Write at least one test that queues a fixture response, runs a simulated loop
iteration, and asserts on call count and argv capture
- Verify the >1s sleep requirement is enforced (test should take >1s wall time)

Research

The project uses BATS v1.12.0 for testing with shared helpers in tests/helpers/ (test_helper.bash, mocks.bash, fixtures.bash). The E2E tests already implement a disk-based mock CLI pattern in tests/e2e/helpers/e2e_helper.bash that creates executable stubs with call tracking, per-call responses, and mandatory >1s sleep. The normalization target is the analysis object in .ralph/.response_analysis produced by lib/response_analyzer.sh, which handles three Claude JSON formats (flat, CLI object, CLI array) plus text fallback. No formal adapter abstraction exists yet—this harness will establish the testing contract that future provider adapters (P3.1+) must satisfy.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Create the foundational adapter test harness infrastructure. The goal is to
establish reusable BATS helpers and fixture directory structures that define the
adapter testing contract.

**File: `tests/helpers/adapter_harness.bash`**
- Create this new file with three core assertion functions and supporting
helpers:
- `assert_command_argv`: Takes a provider name, a command-build function
reference, input options map, and expected argv array; calls the build function
and compares resulting argv using existing `assert_equal` patterns
- `assert_normalizes_to`: Takes a provider output fixture file and expected
analysis JSON; runs the fixture through `parse_json_response`/`analyze_response`
and compares key fields (`exit_signal`, `has_completion_signal`,
`files_modified`, `has_permission_denials`, etc.)
- `assert_capabilities_valid`: Takes a provider name and validates its
capabilities JSON against required schema (provider name, cli_command, supported
output_formats array, supports object with boolean flags)
- `load_provider_capabilities()`: Reads and caches capabilities JSON from
fixtures directory
- `install_provider_mock()`: Accepts parameters (provider name, CLI command
name, mock directory path); generates an executable script at
`$mock_dir/bin/$cli_cmd` following the E2E pattern with call counter in
`calls/.count`, argv logging to `calls/argv_N.log`, per-call responses from
`responses/N.out` with exit codes from `responses/N.exit`; enforces mandatory
sleep >1s (configurable via `responses/N.sleep`, default 1.5s)
- `queue_provider_response()`: Writes fixture content to the correct response
file
  - `get_mock_call_count()` and `get_mock_argv()`: Helpers for test assertions

**Directory & Documentation: `tests/fixtures/providers/`**
- Create the `tests/fixtures/providers/` directory structure
- Create `tests/fixtures/providers/README.md` documenting:
- Expected structure: `<provider>/outputs/` for raw CLI outputs,
`<provider>/expected/` for expected normalized analysis,
`<provider>/capabilities.json` for the capabilities record
- Capabilities JSON schema with required fields (`provider`, `cli_command`,
`output_formats`) and optional fields (`supports`, `required_version`,
`session_continuity`)
===============================================================================

Task: 2

Implement the Claude reference adapter using the harness built in Phase 1. This
phase validates the harness by implementing Claude as the reference adapter with
comprehensive fixtures, a capabilities record, and integration tests.

**Fixtures: `tests/fixtures/providers/claude/outputs/`**
- Create the following fixture files (extract content from existing inline
heredocs in `tests/unit/test_json_parsing.bats` and
`tests/helpers/fixtures.bash` where applicable):
  - `flat_json_complete.json`: Legacy flat format with `exit_signal: true`
  - `flat_json_in_progress.json`: Flat format with `status: IN_PROGRESS`
  - `cli_object_success.json`: Claude CLI `{result, sessionId, metadata}` format
- `cli_array_streaming.json`: JSONL array format `[{type:"system"},
{type:"result"}]`
- `permission_denial.json`: Output containing `has_permission_denials`
indicators
- `text_with_ralph_status.txt`: Plain text with embedded `---RALPH_STATUS---`
block

**Fixtures: `tests/fixtures/providers/claude/expected/`**
- Create corresponding expected analysis JSON files for each output fixture
above

**Capabilities: `tests/fixtures/providers/claude/capabilities.json`**
- Declare:
  - `provider`: "claude", `cli_command`: "claude"
  - `output_formats`: ["flat_json", "cli_object", "cli_array", "text"]
- `supports`: `json_output`, `session_continuity`, `model_selection`,
`effort_levels`, `append_system_prompt`, `allowed_tools`, `resume_session`
- `required_version`: minimum Claude CLI version (reference the `--version`
check in existing code)
- Document each capability's meaning in comments or an accompanying README
section

**Test File: `tests/integration/test_adapter_claude.bats`**
- Load `test_helper`, `adapter_harness`, and source `lib/response_analyzer.sh`
- Test group 1 - Command-build assertions: Use `assert_command_argv` to test
that `build_claude_command()` produces correct argv for representative option
combinations (model override, effort level, resume session, allowed tools,
append system prompt)
- Test group 2 - Normalization assertions: For each fixture in
`claude/outputs/`, call `assert_normalizes_to` against the corresponding
`claude/expected/` file
- Test group 3 - Capabilities validation: Call `assert_capabilities_valid
"claude"` to verify the capabilities record schema
- Include negative tests: malformed output gracefully fails normalization,
missing capabilities fields fail validation
- Add setup/teardown that calls `install_provider_mock "claude" "claude"
"$TEST_DIR/mock"`
- Write at least one test that queues a fixture response, runs a simulated loop
iteration, and asserts on call count and argv capture
- Verify the >1s sleep requirement is enforced (test should take >1s wall time)

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!