feat: universal LLM provider gateway with pluggable search

[tsensei]

Summary

Universal provider gateway — use any OpenAI-compatible LLM endpoint with OpenReels. Independently configurable search backends.

Providers:

• --provider openrouter — 300+ models via OpenRouter (default model: anthropic/claude-sonnet-4)
• --provider openai-compatible --llm-base-url <url> --llm-model <model> — Ollama, Together, Groq, vLLM, any OpenAI-compatible endpoint
• --llm-model <model> — override model for any provider (including existing ones)

Search:

• --search-provider native — use provider's built-in search (Anthropic, OpenAI, Gemini)
• --search-provider tavily — Tavily web search for any provider
• --search-provider none — parametric knowledge only (single-pass, half token cost)
• Auto-detect: native providers use built-in, others use Tavily if TAVILY_API_KEY is set

Resilience:

• Tool-calling error recovery: models without tool support gracefully fall back to parametric knowledge
• Pass 2 retry with cached search: structuring failures retry without re-running Tavily (saves search credits)
• Cost estimation fallback: unknown providers show $0.00 instead of crashing

Web UI: OpenRouter and Custom (OpenAI-compatible) in LLM dropdown, conditional model/URL inputs, search provider selector.

Test Coverage

Tests: 349 → 386 (+37 new)

All new code paths covered:

• 3 new test files (openrouter, openai-compatible, tavily)
• 4 extended test files (base, factory, validate-env, cost-estimator)
• Covers: search injection, no-tools parametric path, tool-calling error fallback, Pass 2 retry, factory search resolution, env validation, cost fallback

Pre-Landing Review

No issues found. Enum completeness verified: all 5 LLMProviderKey values handled in every consumer (factory, cost estimator, worker key map, validation).

TODOS

• Added: Model capability detection for openai-compatible (P2) — probe providers for structured output + tool calling support

Test plan

• All vitest tests pass (386 tests, 44 files)
• TypeScript type check clean (tsc --noEmit)
• Biome lint clean on new files
• Manual test: --provider openrouter --llm-model google/gemini-2.5-flash --search-provider tavily runs full pipeline
• Backward compat: --provider anthropic unchanged

Documentation

9 files updated:

• CLAUDE.md: added openrouter.ts, openai-compatible.ts, search/tavily.ts to project structure
• README.md: expanded LLM provider table with OpenRouter + OpenAI-compatible, added Search row, added 4 new CLI flags
• docs/providers/llm.mdx: added OpenRouter and OpenAI-compatible rows, new Search Providers section with Tavily docs, updated VLM verification models
• docs/providers/index.mdx: added OpenRouter/OpenAI-compatible to LLM row, added Search capability row
• docs/getting-started/api-keys.mdx: added OpenRouter, OpenAI-compatible, and Tavily API key entries
• docs/getting-started/cli.mdx: added --llm-model, --llm-base-url, --search-provider flags, expanded --provider choices
• docs/contributing/architecture.mdx: added openrouter.ts, openai-compatible.ts, search/ directory to provider tree
• docs/contributing/adding-providers.mdx: updated type union example
• CHANGELOG.md: voice polish, moved internal details to "For contributors" subsection

🤖 Generated with Claude Code

[tsensei]

@claude review

[claude]

Claude finished @tsensei's task in 2m 13s —— View job

---

Code Review

• Gathering context and reading changed files
• Review core provider implementations (factory, base, new providers)
• Review search provider and resilience logic
• Review CLI args and validation
• Review server/worker changes
• Review web UI changes
• Review tests
• Post full review feedback

---

This is a well-structured PR with clear separation of concerns. The pluggable search design is clean, the resilience additions are thoughtful, and the test coverage is solid. A few issues worth addressing before merge:

---

Medium Issues

1. Silent fallback in createVerificationModel for openai-compatible may error at runtime

src/providers/factory.ts — the openai-compatible case falls back to Anthropic when OPENREELS_LLM_BASE_URL isn't set:

case "openai-compatible": {
  const baseUrl = process.env["OPENREELS_LLM_BASE_URL"];
  if (!baseUrl) {
    // Fall back to Anthropic for verification if no base URL configured
    const anthropic = createAnthropic();  // ← needs ANTHROPIC_API_KEY
    return anthropic(model ?? "claude-sonnet-4-6");
  }

A user running Ollama (no Anthropic key) will hit this path and get a runtime auth failure on stock verification, not a clear error at startup. The fallback should either throw or at least log a warning:

if (!baseUrl) {
  throw new Error(
    "OPENREELS_LLM_BASE_URL must be set for openai-compatible verification model"
  );
}

Fix this →

---

2. Overly broad string matching for tool-calling error fallback

src/providers/llm/base.ts:98-107 — the substring check for tool errors is too loose:

if (
  msg.includes("tool") ||     // ← matches "timeout occurred during tooling"
  msg.includes("Tool") ||
  msg.includes("function_call") ||
  msg.includes("tools_not_supported")
) {

"tool" matches many unrelated messages (network errors, log messages, etc.) and would silently swallow genuine failures, falling back to parametric knowledge without the user knowing. Consider matching on error codes or more specific patterns like "tool_use_not_supported" / "tools not supported" / tighter regex. At minimum, consider logging the full error message when the fallback triggers.

Fix this →

---

3. Duplicate "no Tavily key" warning

Both src/cli/validate-env.ts:88-97 and src/providers/factory.ts:107-113 emit nearly identical warnings for the same condition (non-native provider, no Tavily key). A user running --provider openrouter without a Tavily key will see the warning twice — once during env validation and again when the pipeline starts. The one in validate-env.ts should be the canonical location; the one in factory.ts should be removed.

---

4. searchProvider typed as string in server.ts

src/server.ts:187 — the CreateJobBody interface types this as string:

interface CreateJobBody {
  providers?: {
    // ...
    searchProvider?: string;  // ← should be SearchProviderKey
  };
}

This bypasses type safety all the way to the worker cast. Import SearchProviderKey and use it here. Same applies to JobData.providers.searchProvider in src/worker.ts:54.

Fix this →

---

Minor Issues

5. Hardcoded "openreels-custom" in two places

src/providers/llm/openai-compatible.ts:19 and src/providers/factory.ts:239 both hardcode the name "openreels-custom" independently. Extract to a constant to keep them in sync.

6. Web UI missing client-side validation for openai-compatible

web/src/pages/HomePage.tsx — when llmProvider === "openai-compatible", the form lets users submit without a base URL or model. The server/CLI catches it, but adding a disabled check on the submit button (or inline field hint) would improve UX by failing faster.

7. Non-null assertion throw lastError!

src/providers/llm/base.ts:150 — lastError! is safe here (loop runs ≥1 times), but this could use a clearer fallback:

throw lastError ?? new Error(`${this.id} Pass 2 failed after ${MAX_STRUCTURE_RETRIES + 1} attempts`);

---

Positive Notes

• The resolveSearchTools function cleanly centralizes search routing logic in one place — good for future search backends.
• The Pass 2 retry with cached search results is a smart optimization that avoids burning search credits on structuring failures.
• NATIVE_SEARCH_PROVIDERS as a Set makes membership checks idiomatic and easy to extend.
• The test coverage for the new paths is thorough — the factory search resolution tests (inject Tavily, inject empty, throw on invalid native, etc.) are exactly the right cases to cover.
• Enum completeness in the switch statements (with explicit default cases) prevents silent breakage when new providers are added.

---

Overall this is merge-ready pending the medium issues (#1 silent Anthropic fallback, #2 broad error matching, #3 duplicate warning, #4 string vs SearchProviderKey).

[tsensei]

Code review response

Pushed 3921579 addressing findings #1, #2, and #4. Here's the full breakdown:

Fixed

#1 — Silent Anthropic fallback in createVerificationModel ✅
Now throws a clear error when both OPENREELS_LLM_BASE_URL and ANTHROPIC_API_KEY are missing, and logs a warning when falling back to Anthropic for VLM verification. User sees actionable guidance: set the base URL, set an Anthropic key, or use --no-stock-verify.

#2 — Broad tool-error string matching ✅
Replaced msg.includes("tool") with specific patterns: tools_not_supported, tool_use, tool_calls, function_call, does not support tools, etc. Also logs the original error message when the fallback triggers, so the user can see exactly what went wrong.

#4 — string vs SearchProviderKey typing ✅
searchProvider in both CreateJobBody (server.ts) and JobData (worker.ts) now uses SearchProviderKey instead of string.

---

Not fixing (by design)

#3 — Duplicate "no Tavily key" warning
These are not duplicates. validateEnv only runs in the CLI path (src/index.ts). The factory's resolveSearchTools warning runs in both CLI and worker (web UI) paths. Removing the factory warning would leave web UI users with no warning at all. Removing the validate-env warning would lose the pre-flight check that runs before any pipeline work starts. The CLI user sees both, but they serve different stages: env validation vs runtime resolution.

#5 — Hardcoded "openreels-custom" in two places
This is a display/debug label for the AI SDK provider name. It appears in OpenAICompatibleLLM (the LLM class) and createVerificationModel (VLM factory). These are independent contexts that happen to share a string literal. Extracting a constant for a two-occurrence string that will never diverge is over-engineering.

#6 — Web UI missing client-side validation for openai-compatible
Server-side validation catches missing base URL and model ID and returns a 400. Adding client-side disabled state is a nice UX improvement but not a merge blocker. Can follow up.

#7 — Non-null assertion throw lastError!
The loop runs at least once (attempt starts at 0, MAX_STRUCTURE_RETRIES is 2), so lastError is always set when we reach the throw. The ?? fallback is cosmetic. Leaving as-is to avoid adding dead code.

[tsensei]

@claude review

[claude]

Claude finished @tsensei's task in 2m 54s —— View job

---

Code Review

• Gathering context and reading changed files
• Verify fixes from previous review (Add TypeScript project scaffold #1, docs: update README with pipeline overview and ReelMistri origin #2, feat: v2 Remotion video pipeline with archetype system #4)
• Review new provider implementations and factory
• Review search resilience logic and tests
• Review CLI args, server, worker, and web UI
• Post full review feedback

---

All three targeted fixes from the previous review are correct. I found one new medium bug introduced by the #1 fix, plus a few minor nits.

---

Previous fixes verified ✅

#1 — createVerificationModel silent fallback ✅ Fixed correctly. The code now throws with a clear error when both OPENREELS_LLM_BASE_URL and ANTHROPIC_API_KEY are missing, and logs a warning when falling back to Anthropic.

#2 — Broad tool error matching ✅ Fixed. The new patterns (tools_not_supported, tools are not supported, tool_use, tool_calls, function_call, etc.) are specific. .toLowerCase() handles case variations. Full error is now logged when the fallback triggers.

#4 — searchProvider typed as string ✅ Fixed. Both CreateJobBody in server.ts:190 and JobData in worker.ts:57 now use SearchProviderKey correctly.

---

New Medium Issue

BYOK path passes the wrong API key to the Anthropic fallback

factory.ts:232 (introduced by the #1 fix):

const antKey = apiKey ?? process.env["ANTHROPIC_API_KEY"];

In the BYOK worker path (worker.ts:248-256), apiKey is resolved as:

const LLM_KEY_MAP = { "openai-compatible": "OPENREELS_LLM_API_KEY", ... };
const llmKey = keys[llmKeyName];              // ← the openai-compatible endpoint key
const verifyModel = createVerificationModel(provider, model, llmKey);

So apiKey inside createVerificationModel is the user's openai-compatible API key, not an Anthropic key. When OPENREELS_LLM_BASE_URL is missing (BYOK path) and OPENREELS_LLM_API_KEY is set, the code uses the openai-compatible key as the Anthropic key → silent auth failure from Anthropic with no useful error message. This only affects BYOK users running openai-compatible without the env var set.

The CLI path (index.ts:47) calls createVerificationModel(opts.provider, opts.verificationModel) without a key argument, so apiKey is undefined and process.env["ANTHROPIC_API_KEY"] is used correctly.

Fix — ignore apiKey and go directly to env in the Anthropic fallback branch:

case "openai-compatible": {
  const baseUrl = process.env["OPENREELS_LLM_BASE_URL"];
  if (!baseUrl) {
    const antKey = process.env["ANTHROPIC_API_KEY"];  // ← don't use apiKey here
    if (!antKey) {
      throw new Error(
        "Stock verification requires a VLM. Set OPENREELS_LLM_BASE_URL or ANTHROPIC_API_KEY, or use --no-stock-verify.",
      );
    }
    console.warn(`[openai-compatible] Using Anthropic for stock verification (VLM)...`);
    const anthropic = createAnthropic({ apiKey: antKey });
    return anthropic(model ?? "claude-sonnet-4-6");
  }

Fix this →

---

Minor Issues

Redundant as cast in worker.ts:132-134

Now that JobData.providers.searchProvider is typed as SearchProviderKey | undefined, this cast is dead code:

searchProvider: providers.searchProvider as
  | import("./schema/providers.js").SearchProviderKey
  | undefined,

Can be simplified to just providers.searchProvider.

displayName() shows raw key for new providers in web UI

web/src/pages/HomePage.tsx — DISPLAY_NAMES is missing "openrouter" and "openai-compatible". The Select trigger uses displayName(llmProvider) which falls back to the raw key string when not found. After a user selects OpenRouter, the trigger reads "openrouter" instead of "OpenRouter". Also, "ollama" in DISPLAY_NAMES is a dead entry (no such provider key exists):

const DISPLAY_NAMES: Record<string, string> = {
  // add these:
  openrouter: "OpenRouter",
  "openai-compatible": "Custom (OpenAI-compatible)",
  // remove this:
  ollama: "Ollama (Local)",  // ← dead, no longer a valid key

---

Positive notes

• Justifications for not-fixing Install manual #3, fix: voice ID, live CLI spinner, em dash prompt rule #5, fix: replace sliding-window captions with stable chunked display #6, fix: visual rendering — pan edge, text card overlap, visual type guidance #7 all hold up under scrutiny. The two-warning argument for Install manual #3 is especially sound — the validate-env warning is a pre-flight check, the factory warning is a runtime guard used by both CLI and web paths.
• The resolveSearchTools design is clean: all search routing in one place, explicit enum handling, clear auto-detect fallback with a warning.
• Tests for the tool-calling fallback path are thorough — the NoToolsLLM subclass with createSearchTools() { return {}; } accurately mirrors how OpenRouterLLM behaves.

---

One medium bug to fix (wrong key in BYOK Anthropic fallback), two minor nits (redundant cast, DISPLAY_NAMES gaps). Otherwise ready.

[tsensei]

Round 2 review response

All three findings valid. Pushed 4540010.

BYOK Anthropic fallback (medium) — Good catch. In the worker path, apiKey is the openai-compatible endpoint key from keys["OPENREELS_LLM_API_KEY"], not an Anthropic key. Changed to process.env["ANTHROPIC_API_KEY"] only, so the wrong key never reaches createAnthropic().

Redundant SearchProviderKey cast — Removed. The type fix from round 1 made it dead code.

DISPLAY_NAMES gaps — Added openrouter and openai-compatible, removed dead ollama entry. SelectValue trigger now shows proper labels.

[tsensei]

@claude review

[claude]

Claude finished @tsensei's task in 2m 35s —— View job

---

Code Review (Round 3)

• Gathering context and reading changed files
• Verify round 2 fixes (BYOK Anthropic fallback, redundant cast, DISPLAY_NAMES)
• Fresh sweep of all changed files
• Post full review feedback

---

All three round 2 fixes are correct. No new medium or high issues found. Full verification below.

---

Round 2 fixes verified ✅

BYOK Anthropic fallback ✅ Fixed correctly. factory.ts:232 now uses process.env["ANTHROPIC_API_KEY"] unconditionally in the Anthropic fallback branch — the apiKey parameter (which carries the openai-compatible endpoint key in the BYOK worker path) is no longer touched here.

Redundant SearchProviderKey cast ✅ Removed. worker.ts:132 is now a clean searchProvider: providers.searchProvider with no cast, since JobData.providers.searchProvider is typed as SearchProviderKey | undefined directly.

DISPLAY_NAMES gaps ✅ Fixed. HomePage.tsx:64-65 has openrouter: "OpenRouter" and "openai-compatible": "Custom (OpenAI-compatible)". The dead ollama entry is gone. The displayName() fallback for the SelectTrigger now resolves correctly for all valid provider keys.

---

Final sweep — nothing blocking

One nit: openai-compatible VLM default is "gpt-4o" regardless of endpoint

factory.ts:249:

return compat(model ?? "gpt-4o");

When OPENREELS_LLM_BASE_URL is set and the user hasn't specified --verification-model, VLM stock verification will request "gpt-4o" from their custom endpoint. This model won't exist on most local/custom runtimes (Ollama, vLLM, etc.), producing a confusing model-not-found error at stock verification time.

The TODOS.md P2 item (model capability detection) would eventually address this, but in the short term a log hint — "use --verification-model to set a VLM for stock verification" — in the error path would make debugging easier. Not a blocker, but worth tracking.

---

Positive notes

• The resolveSearchTools Tavily key fallback chain (keys["TAVILY_API_KEY"] ?? process.env["TAVILY_API_KEY"], factory.ts:100) is correct: BYOK keys take precedence, env var is the fallback. The explicit searchProvider === "tavily" path (line 82) passes only keys["TAVILY_API_KEY"], which is correct because @tavily/ai-sdk reads TAVILY_API_KEY from the env when no key is provided.
• The args.ts validation for openai-compatible (lines 208-213) gives immediate, clear errors before the pipeline starts — better than discovering a missing config mid-run.
• The new LLM_KEY_MAP in worker.ts:246-252 is clean and easy to extend when new providers are added. Enum completeness remains solid.

---

This is ready to merge. No new issues introduced by commit 4540010.