refactor(llm)!: atomic switch to LLMModel protocol

[Pouyanpi]

Part of the LangChain decoupling stack:

1. stack-1: canonical types (feat(types): add framework-agnostic LLM type system #1745)
2. stack-2: adapter and framework registry (feat(llm): add LangChain adapter and framework registry #1759)
3. stack-3: pipeline rewrite + caller migration (this PR)
4. stack-4: rename generate/stream to generate_async/stream_async (refactor(llm): rename generate/stream to generate_async/stream_async #1769)
5. stack-5: remove LangChain imports from core modules (refactor(llm): remove LangChain imports from core modules #1770)
6. stack-6: move LangChain implementations into integrations/langchain/ (refactor(llm): move LangChain implementations into integrations/langchain/ #1772)
7. stack-7: framework-owned provider registry (refactor(llm): framework-owned provider registry #1773)
8. stack-8: framework-agnostic test infrastructure (TODO)

Summary

Atomic switch from LangChain to the LLMModel protocol. These changes are tightly coupled (you cannot change what the initializer returns without also changing how utils.py calls it unless bridge stacks are introduced which is a mess, and we cannot change the return type without updating all callers), so they land as one PR.

How to review

This PR is split into 5 scoped commits for easier review. Please review commit-by-commit rather than the full diff:

1. rewrite initializer and llmrails for LLMModel protocol
init_llm_model() delegates to framework registry, LLMRails types self.llm as LLMModel with legacy wrapping
2. rewrite llm_call to use LLMModel.generate/stream
   the core pipeline change, removes all LangChain extraction logic
3. update integration boundary for canonical tool calls
tool_calls_to_langchain_format() at output boundary
4. migrate all callers to LLMResponse.content
   mechanical one-line changes across 35 call sites
5. restore endpoint URL in LLMCallException via provider_url -
   uses LLMModel.provider_url instead of inspecting LLM attributes

Individual commits do not pass tests on their own since the changes are interdependent. The PR as a whole passes the full test suite.

Summary by CodeRabbit

• New Features

  • Introduced framework abstraction layer enabling support for multiple LLM providers beyond LangChain, with new public APIs: get_default_framework(), register_framework(), and set_default_framework().

• Refactor

  • Replaced LangChain-specific LLM types with unified LLMModel interface throughout the codebase. Added LangChainLLMAdapter for backward compatibility with existing LangChain implementations.

[greptile-apps]

Greptile Summary

This PR performs an atomic switch from LangChain types to the new LLMModel protocol across the core pipeline, introducing LangChainLLMAdapter for backward-compat wrapping and migrating all 35+ llm_call sites to consume LLMResponse.content. The refactor is well-structured across 5 scoped commits and the overall design is clean.

• P1 — stop-sequence regression for OpenAI reasoning models: LangChainLLMAdapter._filter_reasoning_model_params only strips temperature, but the old _filter_params_for_openai_reasoning_models also stripped stop for o3, o4-mini, and gpt-5 (non-chat). stop is now passed directly to ainvoke/astream without any filtering. The Colang 1.0 generation pipeline passes stop=["\nuser ", "\nUser "]; any user targeting one of those models will hit an OpenAI API rejection.

Confidence Score: 4/5

• Safe for most users but will break API calls for anyone using o3, o4-mini, or gpt-5 (non-chat) models with Colang 1.0's standard generation pipeline.
• One P1 regression: stop-sequence filtering was explicitly present in the old code for specific OpenAI reasoning models and was not ported to the new adapter. All other changes are mechanically correct and the return-type migration to LLMResponse.content is complete across all call sites.
• nemoguardrails/integrations/langchain/llm_adapter.py — _filter_reasoning_model_params needs stop-sequence handling restored

Important Files Changed

Filename	Overview
nemoguardrails/integrations/langchain/llm_adapter.py	New LangChain adapter; drops stop-parameter filtering for OpenAI reasoning models (o3, o4-mini, gpt-5 non-chat) that was present in the old pipeline — will cause API errors for those users.
nemoguardrails/actions/llm/utils.py	Clean rewrite to LLMModel protocol; LangChain-specific extraction logic removed; main concern is that streaming path doesn't call _store_reasoning_traces/_store_tool_calls (pre-existing gap, not introduced here).
nemoguardrails/llm/models/initializer.py	Now delegates to framework registry; mode parameter correctly moved to model_kwargs and defaulted to "chat"; clean and correct.
nemoguardrails/rails/llm/llmrails.py	Typed llm as LLMModel, added _wrap_legacy_llm for backward compatibility; deprecation warning correctly uses stacklevel=3.
nemoguardrails/types.py	Well-designed canonical type system; LLMModel/LLMFramework as runtime_checkable Protocols; LLMResponse/LLMResponseChunk/ChatMessage dataclasses are complete and correct.
nemoguardrails/llm/frameworks.py	Lazy framework registry with env-var override; clean design. register_framework raises on duplicate which could trip integration tests if they don't call _reset_frameworks().
nemoguardrails/integrations/langchain/message_utils.py	New chatmessage_to_langchain_message correctly falls back to msg.content or "" for all roles; tool_calls_to_langchain_format output boundary is correct.
nemoguardrails/actions/llm/generation.py	All call sites correctly updated to .content; fire-and-forget streaming task at line 1260 correctly discards the LLMResponse.

Sequence Diagram

sequenceDiagram
    participant Caller as generation.py caller
    participant llm_call as llm_call (utils.py)
    participant Adapter as LangChainLLMAdapter
    participant LC as LangChain llm

    Caller->>llm_call: "await llm_call(llm, prompt, stop=[...], llm_params={...})"
    llm_call->>llm_call: isinstance(llm, LLMModel) check
    llm_call->>llm_call: _setup_llm_call_info / _log_prompt
    alt streaming
        llm_call->>llm_call: _stream_llm_call(model, prompt, handler, stop, llm_params)
        llm_call->>Adapter: "model.stream(prompt, stop=stop, **llm_params)"
        Adapter->>Adapter: _prepare_llm(kwargs) → filter temperature only
        Adapter->>LC: "llm.astream(messages, stop=stop)"
        LC-->>Adapter: chunks
        Adapter-->>llm_call: LLMResponseChunk stream
        llm_call-->>Caller: "LLMResponse(content=handler.completion)"
    else non-streaming
        llm_call->>Adapter: "model.generate(prompt, stop=stop, **llm_params)"
        Adapter->>Adapter: _prepare_llm(kwargs) → filter temperature only
        Adapter->>LC: "llm.ainvoke(messages, stop=stop)"
        LC-->>Adapter: AIMessage
        Adapter->>Adapter: _langchain_response_to_llm_response
        Adapter-->>llm_call: LLMResponse
        llm_call->>llm_call: _store_reasoning_traces / _log_completion / _update_token_stats
        llm_call->>llm_call: _store_tool_calls / _store_response_metadata
        llm_call-->>Caller: LLMResponse
    end

Loading

Prompt To Fix All With AI

This is a comment left during a code review.
Path: nemoguardrails/integrations/langchain/llm_adapter.py
Line: 151-169

Comment:
**`stop` parameter filtering dropped for reasoning models**

The old `_filter_params_for_openai_reasoning_models` filtered both `temperature` **and** `stop` for models that don't support them (`o3`, `o4-mini`, `gpt-5` non-chat). The new method only removes `temperature`. Now `stop` is forwarded directly to `ainvoke`/`astream` (lines 196, 208) with no stripping for these models.

NeMo Guardrails passes `stop=["\nuser ", "\nUser "]` in the Colang 1.0 generation pipeline (`generation.py` line 1264). Any user pointing `o3`, `o4-mini`, or `gpt-5` (non-chat) at that path will hit an OpenAI API error because those models reject the `stop` parameter.

The adapter needs to filter `stop` before forwarding it:

```python
_STOP_NOT_SUPPORTED_MODELS = ("o4-mini",)
_STOP_NOT_SUPPORTED_PREFIXES = ("gpt-5",)  # except gpt-5-chat-*

def _filter_reasoning_model_params(self, params: Optional[dict]) -> Optional[dict]:
    if not params:
        return params
    model_name = _infer_model_name(self._llm).lower()
    is_openai_reasoning_model = (
        model_name.startswith("o1")
        or model_name.startswith("o3")
        or (model_name.startswith("gpt-5") and "chat" not in model_name)
    )
    is_stop_not_supported = (
        (model_name.startswith("o3") and not model_name.startswith("o3-mini"))
        or model_name.startswith("o4-mini")
        or (model_name.startswith("gpt-5") and "gpt-5-chat" not in model_name)
    )
    if not is_openai_reasoning_model and not is_stop_not_supported:
        return params
    filtered = params.copy()
    if is_openai_reasoning_model:
        filtered.pop("temperature", None)
    if is_stop_not_supported:
        filtered.pop("stop", None)
    return filtered
```

And `stop` should be omitted from `ainvoke`/`astream` when it is filtered:

```python
async def generate(self, prompt, *, stop=None, **kwargs):
    filtered_stop = None if self._is_stop_not_supported() else stop
    llm = self._prepare_llm(kwargs)
    response = await llm.ainvoke(self._to_langchain_input(prompt), stop=filtered_stop)
    return _langchain_response_to_llm_response(response)
```

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (8): Last reviewed commit: "fix(llm): use direct assignment for toke..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 90.10417% with 19 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
nemoguardrails/actions/v2_x/generation.py	55.55%	8 Missing ⚠️
nemoguardrails/actions/llm/utils.py	94.73%	4 Missing ⚠️
nemoguardrails/evaluate/evaluate_moderation.py	0.00%	2 Missing ⚠️
nemoguardrails/rails/llm/llmrails.py	87.50%	2 Missing ⚠️
nemoguardrails/actions/llm/generation.py	94.44%	1 Missing ⚠️
nemoguardrails/eval/check.py	0.00%	1 Missing ⚠️
nemoguardrails/evaluate/evaluate_factcheck.py	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces a framework-agnostic LLM model abstraction (LLMModel) to replace LangChain-specific types throughout the codebase. It updates function signatures, changes llm_call() to return LLMResponse objects, adds a LangChainLLMAdapter for backward compatibility, and refactors model initialization to use a pluggable framework system with deprecation warnings for legacy LangChain inputs.

Changes

Cohort / File(s)	Summary
Package API nemoguardrails/__init__.py	Re-exports three framework-management functions (get_default_framework, register_framework, set_default_framework) and expands __all__ accordingly.
Core LLM Abstraction & Initialization nemoguardrails/llm/models/initializer.py, nemoguardrails/llm/frameworks	init_llm_model now returns framework-agnostic LLMModel instead of LangChain types; makes mode parameter optional with default "chat" and delegates to framework-based model creation.
LLM Call Utilities nemoguardrails/actions/llm/utils.py	Reworked llm_call to accept and validate LLMModel instances, returns LLMResponse instead of string; refactored response handling (reasoning, token stats, tool calls, metadata) to use strongly-typed LLMResponse/LLMResponseChunk instead of LangChain extraction paths.
Generation Actions (v1) nemoguardrails/actions/llm/generation.py	Updated all LLM parameter types from LangChain to LLMModel; changed llm_call result handling to extract .content before downstream processing in generate_user_intent, generate_next_steps, generate_bot_message, generate_value, and generate_intent_steps_message.
Generation Actions (v2.x) nemoguardrails/actions/v2_x/generation.py	Replaced LangChain type annotations with LLMModel across eight action methods; updated llm_call result handling to extract .content in generation and passthrough paths.
Guardrails Core & LLMRails nemoguardrails/guardrails/guardrails.py, nemoguardrails/rails/llm/llmrails.py	Updated Guardrails and LLMRails LLM parameter types from LangChain to LLMModel; added _wrap_legacy_llm helper to adapt deprecated raw LangChain inputs with DeprecationWarning; updated method signatures for __init__ and update_llm.
LangChain Integration nemoguardrails/integrations/langchain/llm_adapter.py, nemoguardrails/integrations/langchain/message_utils.py, nemoguardrails/integrations/langchain/runnable_rails.py	Removed temporary comment in llm_adapter.py; rewrote chatmessage_to_langchain_message/chatmessages_to_langchain_messages to use role lookup table and preserve additional fields; added tool_calls_to_langchain_format helper; updated RunnableRails._format_output to normalize tool calls via the new helper.
Library Actions (Safety, Fact-check, Moderation) nemoguardrails/library/content_safety/actions.py, nemoguardrails/library/factchecking/align_score/actions.py, nemoguardrails/evaluate/evaluate_factcheck.py, nemoguardrails/evaluate/evaluate_moderation.py, nemoguardrails/eval/check.py	Updated llms/llm parameter types from BaseLLM to LLMModel; extracted .content from llm_call() results before downstream processing (parsing, text operations, condition checks).
Library Actions (Other Checks) nemoguardrails/library/hallucination/actions.py, nemoguardrails/library/llama_guard/actions.py, nemoguardrails/library/patronusai/actions.py, nemoguardrails/library/self_check/facts/actions.py, nemoguardrails/library/self_check/input_check/actions.py, nemoguardrails/library/self_check/output_check/actions.py, nemoguardrails/library/topic_safety/actions.py	Updated llm/llms parameter types to LLMModel; consistently extracted .content from llm_call() results for downstream parsing and validation logic.
Evaluation Utilities nemoguardrails/evaluate/utils.py	Added mode="chat" keyword argument to init_llm_model call in initialize_llm().
Unit Tests (Core) tests/test_actions_llm_utils.py, tests/test_logging.py, tests/test_reasoning_trace_extraction.py	Replaced LangChain-specific test construction with LLMResponse/LLMResponseChunk and typed objects; wrapped mocked LLMs with LangChainLLMAdapter; updated assertions to check .content and new response structure; removed tests for legacy LangChain extraction and parameter-filtering logic.
Unit Tests (Action Mocks) tests/test_content_safety_actions.py, tests/test_content_safety_cache.py, tests/test_content_safety_integration.py, tests/test_llama_guard.py, tests/test_patronus_lynx.py, tests/test_topic_safety_cache.py, tests/test_topic_safety_internalevent.py	Updated test fixtures and mocks to wrap FakeLLM instances with LangChainLLMAdapter; adjusted mock return types from strings to LLMResponse where needed.
Unit Tests (Tool Calling & Passthrough) tests/test_tool_calling_passthrough_only.py, tests/test_tool_calling_utils.py, tests/test_tool_calls_event_extraction.py, tests/test_tool_output_rails.py	Wrapped mocked LLMs with LangChainLLMAdapter; updated tool-call structure expectations from flat dict shape to LangChain-compatible shape with nested function sub-dict; adjusted assertions to reference new tool-call field paths (function.name, function.arguments vs name, args).
Unit Tests (Infrastructure) tests/test_llmrails.py	Updated LLM mocking from magic-mock to concrete FakeLLM; changed assertion for injected LLM from direct equality to checking raw_llm wrapper attribute.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 39.44% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Results For Major Changes	⚠️ Warning	PR lacks documented test results for major architectural changes affecting 23 source files and breaking APIs, only claims tests pass without evidence.	Add test execution results to PR description including pass/fail counts, CI logs, and regression testing documentation to verify the major refactoring introduces no functional regressions.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'refactor(llm)!: atomic switch to LLMModel protocol' clearly and specifically describes the main change—migrating from LangChain types to an internal LLMModel protocol across the codebase.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/langchain-decouple/stack-3-pipeline-rewrite

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@nemoguardrails/rails/llm/llmrails.py`:
- Around line 121-136: In _wrap_legacy_llm, the ImportError caught when
importing LangChainLLMAdapter should be chained to the new TypeError so the
original exception context is preserved; update the except block to capture the
ImportError (e.g., "except ImportError as err:") and re-raise the TypeError
using "raise TypeError(... ) from err" when declining to proceed, referencing
the _wrap_legacy_llm function and LangChainLLMAdapter import to locate the
change.

In `@tests/test_tool_output_rails.py`:
- Around line 35-36: The fixtures build flat tool-call objects but the helpers
now expect the nested shape under function.arguments; update all mocked calls
used by self_check_tool_calls and validate_tool_parameters so each tool_call has
a "function" dict with "name" and "arguments" keys (e.g., replace top-level
"name" and "args" entries with function: {"name": <tool_name>, "arguments":
<args_dict>}) for the fixtures currently constructed in
tests/test_tool_output_rails.py (the ones referenced by self_check_tool_calls
and validate_tool_parameters), ensuring every mocked call in the happy-path and
blocking tests uses this nested schema.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9aec663d-bca9-4285-b01c-7228a00d8aa6

📥 Commits

Reviewing files that changed from the base of the PR and between 3a9cbad and a865c0b.

📒 Files selected for processing (38)

• nemoguardrails/__init__.py
• nemoguardrails/actions/llm/generation.py
• nemoguardrails/actions/llm/utils.py
• nemoguardrails/actions/v2_x/generation.py
• nemoguardrails/eval/check.py
• nemoguardrails/evaluate/evaluate_factcheck.py
• nemoguardrails/evaluate/evaluate_moderation.py
• nemoguardrails/evaluate/utils.py
• nemoguardrails/guardrails/guardrails.py
• nemoguardrails/integrations/langchain/llm_adapter.py
• nemoguardrails/integrations/langchain/message_utils.py
• nemoguardrails/integrations/langchain/runnable_rails.py
• nemoguardrails/library/content_safety/actions.py
• nemoguardrails/library/factchecking/align_score/actions.py
• nemoguardrails/library/hallucination/actions.py
• nemoguardrails/library/llama_guard/actions.py
• nemoguardrails/library/patronusai/actions.py
• nemoguardrails/library/self_check/facts/actions.py
• nemoguardrails/library/self_check/input_check/actions.py
• nemoguardrails/library/self_check/output_check/actions.py
• nemoguardrails/library/topic_safety/actions.py
• nemoguardrails/llm/models/initializer.py
• nemoguardrails/rails/llm/llmrails.py
• tests/test_actions_llm_utils.py
• tests/test_content_safety_actions.py
• tests/test_content_safety_cache.py
• tests/test_content_safety_integration.py
• tests/test_llama_guard.py
• tests/test_llmrails.py
• tests/test_logging.py
• tests/test_patronus_lynx.py
• tests/test_reasoning_trace_extraction.py
• tests/test_tool_calling_passthrough_only.py
• tests/test_tool_calling_utils.py
• tests/test_tool_calls_event_extraction.py
• tests/test_tool_output_rails.py
• tests/test_topic_safety_cache.py
• tests/test_topic_safety_internalevent.py

💤 Files with no reviewable changes (1)

• nemoguardrails/integrations/langchain/llm_adapter.py

[tgasser-nv]

Thanks for fixing the accumulation change, the rest all looks good