Skip to content

provider-test-obligations.md

Latest commit

 

History

History
408 lines (295 loc) · 17.5 KB

provider-test-obligations.md

File metadata and controls

408 lines (295 loc) · 17.5 KB
Raw

Provider Test Obligations

Mandatory test categories, coverage floors, and artifact requirements for every provider in pi_agent_rust.

Overview

Every provider in the codebase carries a ProviderTestObligations struct (defined in src/provider_metadata.rs) that declares which test tiers are required:

pub struct ProviderTestObligations {
    pub unit: bool,      // Unit-level checks (identity, request mapping, auth, tools)
    pub contract: bool,  // Wire-format contract tests with mock HTTP
    pub conformance: bool, // VCR cassette-based streaming conformance
    pub e2e: bool,       // End-to-end agent loop tests
}

The constant TEST_REQUIRED sets all four to true and is the default for every provider in PROVIDER_METADATA. CI enforces these obligations via tests/provider_unit_checklist.rs.

Tier 1: Unit Tests (Checklist)

Enforced by: tests/provider_unit_checklist.rs Scope: Pure-logic validation with no HTTP, no VCR, no async runtime.

Required Test Classes

Every native provider must pass all six checklist classes. If you add a new native provider and skip any class, the meta-test checklist_all_native_providers_enumerated will fail CI.

# Class What It Validates Macro/Pattern
1 Identity name(), api(), model_id() return non-empty, correct values checklist_all_native_providers_have_identity (single test enumerates all providers)
2 Request mapping build_request() produces valid serializable JSON with at least one field checklist_request_mapping! macro per provider
3 Auth/header composition API key from StreamOptions.api_key flows into request (not silently dropped) checklist_*_auth_key_flows_through tests
4 URL/endpoint resolution api() returns the expected API type string for the provider checklist_providers_have_default_endpoint (single test enumerates all)
5 Tool-call serialization build_request() with ToolDef entries produces provider-specific tool wire format checklist_tool_serialization! macro per provider
6 VCR fixture presence At least N VCR cassettes exist matching verify_{provider}_*.json checklist_vcr_fixture_coverage_floor

VCR Coverage Floor (per provider)

Provider Minimum Cassettes Rationale
anthropic 3 simple_text + tool_call + error_auth
openai 3 simple_text + tool_call + error_auth
gemini 3 simple_text + tool_call + error_auth
cohere 3 simple_text + tool_call + error_auth
azure 1 at least simple_text
bedrock 1 at least simple_text
vertex 1 at least simple_text
copilot 1 at least simple_text
gitlab 1 at least simple_text

Adding a New Provider to the Checklist

  1. Add the provider to the providers vec in checklist_all_native_providers_have_identity
  2. Add a checklist_request_mapping! invocation
  3. Add a checklist_tool_serialization! invocation with the correct JSON path for tool definitions
  4. Add an auth flow-through test
  5. Add the provider to known_providers in checklist_providers_have_default_endpoint
  6. Add the provider to required_providers in checklist_vcr_fixture_coverage_floor
  7. Update the count in checklist_all_native_providers_enumerated

Provider-Specific Tool JSON Paths

Different providers serialize tool definitions at different JSON paths:

Provider Tool JSON Path
Anthropic tools[] (with input_schema)
OpenAI tools[] (with function.parameters)
Gemini tools[0].functionDeclarations[]
Cohere tools[] (with function)
Bedrock toolConfig.tools[]

Tier 2: Contract Tests (Mock HTTP)

Enforced by: tests/provider_native_contract.rs Scope: Full request-response cycle using MockHttpRequest/MockHttpResponse from the test harness. Tests run async but use canned HTTP responses, not real networks.

Required Scenarios

Each native provider must have contract tests covering:

# Scenario What It Validates
1 Simple text response Provider correctly decodes a streaming text response into TextDelta + Done events
2 Tool call response Provider decodes a tool-call response into ToolCallStart/ToolCallDelta/ToolCallEnd events
3 Auth header construction The correct auth header (Authorization: Bearer, X-API-Key, api-key, etc.) is sent
4 Request payload shape URL path, Content-Type, and body JSON structure match provider spec

Test Infrastructure

// Helper to create a ModelEntry for contract tests
fn make_model_entry(provider: &str, model_id: &str, base_url: &str) -> ModelEntry;

// Helper to create SSE-formatted mock responses
fn text_event_stream_response(body: String) -> MockHttpResponse;

// Helper to inspect captured request
fn request_header(headers: &[(String, String)], key: &str) -> Option<String>;
fn request_body_json(request: &MockHttpRequest) -> serde_json::Value;

// Drive a provider stream to completion and collect all events
fn collect_stream_events(provider, context, options) -> Vec<StreamEvent>;

SSE Body Generators

Each provider has a dedicated SSE body generator function that produces valid SSE data matching the provider's wire format:

  • anthropic_simple_sse() / anthropic_tool_call_sse()
  • openai_simple_sse() / openai_tool_call_sse()
  • gemini_simple_sse() / gemini_tool_call_sse()
  • cohere_simple_sse() / cohere_tool_call_sse()
  • azure_simple_sse() / azure_tool_call_sse()
  • etc.

These generators produce the minimum valid SSE stream for the scenario. They are the canonical reference for each provider's wire format.

Tier 3: Error Path Tests (VCR)

Enforced by: tests/provider_error_paths.rs Scope: Deterministic offline testing of HTTP error handling and malformed SSE using VCR cassette playback.

Required Error Scenarios

# Scenario Status Expected Behavior
1 HTTP 500 500 stream() returns Err containing "HTTP 500" and response body text
2 Wrong Content-Type 200 stream() returns Err with "protocol error" and "content-type"
3 Missing Content-Type 200 stream() returns Err with "missing content-type"
4 Invalid JSON in SSE 200 Stream yields Err with "JSON" or "parse"
5 Invalid UTF-8 200 Stream yields Err with "SSE error" (uses base64 VCR body chunks)

VCR Cassette Helper

fn vcr_client(
    test_name: &str,           // Cassette file name
    url: &str,                 // Expected request URL
    request_body: Value,       // Expected request body (must match exactly)
    status: u16,               // HTTP response status
    response_headers: Vec<(String, String)>,
    response_chunks: Vec<String>,
) -> (Client, TempDir);       // Client with VCR + temp dir (keep alive!)

For invalid UTF-8 tests, use vcr_client_bytes() which accepts Vec<Vec<u8>> and encodes as base64 in the cassette.

Provider-Specific Request Body Builders

Each provider has a body builder that produces the exact JSON the provider serializes:

fn anthropic_body(model: &str, prompt: &str) -> Value;  // messages + model + stream + max_tokens
fn openai_body(model: &str, prompt: &str) -> Value;     // messages + model + stream + stream_options
fn gemini_body(prompt: &str) -> Value;                   // contents + generationConfig
fn azure_body(prompt: &str) -> Value;                    // messages + stream + stream_options (no model)

These must exactly match what the provider's stream() method serializes, because VCR matching compares request bodies field-by-field.

Tier 4: Streaming Conformance Tests (VCR Cassettes)

Enforced by: tests/provider_streaming.rs + per-provider sub-modules in tests/provider_streaming/ Scope: Full streaming round-trips with VCR cassette record/playback against real API formats.

Sub-Module Structure

tests/provider_streaming.rs        # Root: shared helpers, VCR config, StreamOutcome/StreamSummary
tests/provider_streaming/
    anthropic.rs                   # Anthropic-specific streaming tests
    openai.rs                      # OpenAI Chat Completions tests
    openai_responses.rs            # OpenAI Responses API tests
    gemini.rs                      # Gemini streaming tests
    azure.rs                       # Azure OpenAI streaming tests
    cohere.rs                      # Cohere streaming tests

Required Scenarios Per Provider

# Scenario VCR Cassette Pattern Validates
1 Simple text verify_{provider}_simple_text.json Basic text streaming: Start → TextDelta(s) → Done
2 Tool call verify_{provider}_tool_call_single.json Tool use: ToolCallStart → ToolCallDelta(s) → ToolCallEnd → Done
3 Unicode text verify_{provider}_unicode_text.json Non-ASCII content preservation through streaming
4 Auth error (401) verify_{provider}_error_auth_401.json Auth failure produces Error event or stream error
5 Bad request (400) verify_{provider}_error_bad_request_400.json Malformed request produces Error event
6 Rate limit (429) verify_{provider}_error_rate_limit_429.json Rate limit produces Error event

StreamEvent Sequence Validation

The StreamSummary struct tracks the complete event timeline:

pub struct StreamSummary {
    pub timeline: Vec<String>,    // Ordered event type names
    pub event_count: usize,
    pub has_start: bool,          // Must be true for successful streams
    pub has_done: bool,           // Must be true for completed streams
    pub has_error_event: bool,    // True for error scenarios
    pub text: String,             // Accumulated text content
    pub thinking: String,         // Accumulated thinking content
    pub tool_calls: Vec<ToolCall>,
    pub text_deltas: usize,       // Count of TextDelta events
    pub stop_reason: Option<StopReason>,
    pub stream_error: Option<String>,
}

VCR Modes

Mode Env Var Behavior
Playback (default) VCR_MODE=playback or unset Replay from cassette files
Record VCR_MODE=record Record real API interactions (requires API keys)
Auto VCR_MODE=auto Use cassette if available, record otherwise
Strict VCR_STRICT=1 Fail on request mismatch instead of falling through

VCR Cassette Location

All VCR cassettes live in tests/fixtures/vcr/ and follow the naming convention:

verify_{provider}_{scenario}.json

Current cassette inventory includes 90+ files covering the 10 native provider implementation modules plus OpenAI-compatible presets (alibaba-cn, kimi-for-coding, minimax, modelscope, moonshotai-cn, nebius, ovhcloud, scaleway, sap_ai_core, siliconflow, upstage, venice, zai, zhipuai-coding-plan, etc.).

Tier 5: E2E Tests

Enforced by: tests/e2e_*.rs Scope: Full agent loop with provider (VCR playback), verifying multi-turn conversations and tool use scenarios end-to-end.

E2E Test Approach

  • Use VCR playback (VCR_MODE=playback) to avoid real API calls
  • Set PI_TEST_MODE=1 for deterministic system prompts
  • Use --thinking off for deterministic test behavior
  • Use isolation flags: --no-tools --no-extensions --no-skills --no-prompt-templates --no-themes

E2E Requirements for Native Providers

Each native provider should have at least one E2E test demonstrating:

  1. A complete agent turn (user message → provider response → rendered output)
  2. Correct StreamEvent translation through the agent loop
  3. Session persistence of the interaction

OpenAI-Compatible Preset Obligations

Providers with onboarding: OpenAICompatiblePreset route through the shared OpenAIProvider or OpenAIResponsesProvider. Their test obligations are lighter:

Obligation Required? What to Provide
Unit (identity) No (covered by shared OpenAI tests) N/A
Unit (request mapping) No (covered by shared OpenAI tests) N/A
VCR cassettes Yes At minimum: verify_{provider}_simple_text.json, verify_{provider}_error_auth_401.json, verify_{provider}_tool_call_single.json
Error paths No (covered by shared OpenAI error tests) N/A
E2E No (covered by shared OpenAI E2E tests) N/A

The VCR cassettes validate that the provider's actual API response format is compatible with the OpenAI parser. This catches providers that claim OpenAI compatibility but have subtle wire format differences.

Test Helper Reference

tests/common/ Shared Infrastructure

Module Purpose
common/harness.rs TestHarness with JSONL logging, artifact tracking, MockHttpRequest/MockHttpResponse
common/mod.rs run_async() helper for blocking on async code in tests

Key Helper Functions

// Minimal context with one user message
fn minimal_context() -> Context;
fn simple_context() -> Context;

// Context with one or two ToolDef entries
fn context_with_tools() -> Context;

// StreamOptions with test API key
fn default_options() -> StreamOptions;
fn options_with_key(key: &str) -> StreamOptions;

// Count VCR cassettes matching a provider prefix
fn count_cassettes(provider_prefix: &str) -> usize;

// Collect all stream events until Done
fn collect_stream_events(provider, context, options) -> Vec<StreamEvent>;
async fn collect_events<S: Stream>(stream: S) -> StreamOutcome;

// Summarize event timeline for assertions
fn summarize_events(outcome: &StreamOutcome) -> StreamSummary;

Running Provider Tests

# All provider tests (unit + contract + conformance + error paths)
cargo test provider

# Specific provider
cargo test anthropic
cargo test openai
cargo test gemini

# Unit checklist only
cargo test provider_unit_checklist

# Contract tests only
cargo test provider_native_contract

# Error path tests only
cargo test provider_error_paths

# Streaming conformance (VCR playback)
cargo test provider_streaming

# Streaming conformance for one provider
cargo test provider_streaming::anthropic_

# Record new VCR cassettes (requires API key)
ANTHROPIC_API_KEY=sk-ant-... VCR_MODE=record cargo test provider_streaming::anthropic_

Adding Tests for a New Provider: Step-by-Step

For a Native Provider

  1. Unit checklist (tests/provider_unit_checklist.rs):

    • Add to checklist_all_native_providers_have_identity providers vec
    • Add checklist_request_mapping! invocation
    • Add checklist_tool_serialization! invocation
    • Add auth flow-through test
    • Add to checklist_providers_have_default_endpoint known_providers vec
    • Add to checklist_vcr_fixture_coverage_floor required_providers vec
    • Increment count in checklist_all_native_providers_enumerated

  2. Contract tests (tests/provider_native_contract.rs):

    • Add SSE body generator function (e.g., your_provider_simple_sse())
    • Add simple text streaming test
    • Add tool call streaming test
    • Add auth header verification test

  3. Error path tests (tests/provider_error_paths.rs):

    • Add request body builder (e.g., your_provider_body())
    • Add HTTP 500 test
    • Add malformed SSE test

  4. Streaming conformance (tests/provider_streaming/):

    • Create tests/provider_streaming/your_provider.rs
    • Add #[path = "provider_streaming/your_provider.rs"] mod your_provider; to tests/provider_streaming.rs
    • Record VCR cassettes for all 6 scenarios
    • Write tests that assert on StreamSummary fields

  5. VCR cassettes (tests/fixtures/vcr/):

    • verify_your_provider_simple_text.json
    • verify_your_provider_tool_call_single.json
    • verify_your_provider_unicode_text.json
    • verify_your_provider_error_auth_401.json
    • verify_your_provider_error_bad_request_400.json
    • verify_your_provider_error_rate_limit_429.json

For an OpenAI-Compatible Preset

  1. VCR cassettes (tests/fixtures/vcr/):

    • verify_your_provider_simple_text.json
    • verify_your_provider_tool_call_single.json
    • verify_your_provider_error_auth_401.json

  2. Add to VCR coverage floor in checklist_vcr_fixture_coverage_floor (if the provider is prominent enough to warrant CI enforcement)

Common Pitfalls

  1. VCR body mismatch: VCR matching compares request bodies exactly (after JSON normalization). If your provider adds extra fields (e.g., stream_options), your test body builder must include them too.

  2. Forgetting _dir in VCR tests: The TempDir returned by vcr_client() must be kept alive for the test duration. If dropped early, the cassette file is deleted and playback fails.

  3. Missing oauth_config: None: Every ModelEntry construction in tests must include oauth_config: None.

  4. Provider-specific URL patterns: Some providers append paths differently. Gemini appends ?alt=sse&key=... as query params. Azure uses a completely different URL structure. Ensure your test URLs match what the provider actually sends.

  5. Base64 body chunks for invalid UTF-8: Standard VCR cassettes store response body as UTF-8 strings. For tests that need raw bytes (invalid UTF-8), use vcr_client_bytes() with body_chunks_base64.

  6. common::run_async vs asupersync::test_utils::run_test: Contract and error path tests use common::run_async() for simplicity. Streaming conformance tests may use the full runtime. Both patterns are acceptable.