feat(library): namespace custom task prompts in self-check

[Pouyanpi]

Description

Follow-up to #1874. Namespace custom self-check prompt task identifiers under
self_check_input and self_check_output while preserving task-specific model
routing.

Adds Colang 1 and 2 coverage plus recorded public-API coverage for sequential,
parallel, and streaming execution. Review requested from @RobGeada.

AI Assistance

• No AI tools were used.
• AI tools were used; a human reviewed and can explain every change.

Checklist

• I've read the CONTRIBUTING guidelines.
• This PR links to a triaged issue assigned to me.
• My PR title follows the project commit convention.
• I've updated the documentation if applicable.
• I've added tests if applicable.
• I've noted any verification beyond CI and any checks I couldn't run.
• I did not update generated changelog files manually.
• I addressed all CodeRabbit, Greptile, and other review comments, or replied with why no change is needed.
• @mentions of the person or team responsible for reviewing proposed changes.

Summary by CodeRabbit

• New Features

  • Self-check rails now support multiple custom tasks with distinct prompt templates and model routing.
  • Input and output safety checks correctly preserve task-specific behavior during generation and streaming.
  • Custom task prompt validation now provides clearer missing-template errors.

• Bug Fixes

  • Improved task resolution when runtime context is unavailable or contains conflicting history.
  • Corrected sequential and parallel self-check execution and output-blocking behavior.

• Tests

  • Expanded coverage for multi-task checks, streaming, parallel execution, provider errors, and prompt validation.

[greptile-apps]

Greptile Summary

This PR namespaces custom self-check prompt task identifiers under their parent default task (e.g. check_harmful → self_check_input $variant=check_harmful) and renames the Colang flow parameter from $task to $variant, completing the multi-task self-check feature introduced in #1874. A deprecation fallback with a DeprecationWarning is provided for users who still have bare custom prompt keys.

• Core logic (utils.py): introduces get_self_check_prompt_task to resolve namespaced prompt task names with a backward-compat deprecation path for bare keys; renames SELF_CHECK_{INPUT,OUTPUT}_TASK_PARAM → SELF_CHECK_{INPUT,OUTPUT}_VARIANT_PARAM throughout.
• Config validation (config.py): replaces _validate_self_check_prompts with _validate_self_check_rail_prompts that delegates namespacing to the shared get_self_check_prompt_task helper.
• Test coverage: adds Colang 1 and 2 unit tests, recorded public-API tests, and streaming/parallel integration tests for the new namespaced task format.

Confidence Score: 5/5

Safe to merge; all logic changes are well-tested and the deprecation fallback prevents a hard break for existing prompt key formats.

The namespacing logic flows through a single shared helper (get_self_check_prompt_task) used consistently across both the validation and runtime paths. Tests cover sequential, parallel, streaming, Colang 1, Colang 2, and error propagation cases. The two flagged points are minor: a hardcoded string that should reference the shared constant, and the absence of a deprecation notice for the older $task= flow parameter.

No files require special attention beyond the two nits in utils.py and config.py.

Important Files Changed

Filename	Overview
nemoguardrails/library/self_check/utils.py	Core change: renames $task to $variant param, adds get_self_check_prompt_task for namespaced prompts with a deprecation fallback for bare task keys.
nemoguardrails/rails/llm/config.py	Refactors _validate_self_check_prompts into _validate_self_check_rail_prompts using get_self_check_prompt_task for namespacing, but hardcodes variant string instead of using SELF_CHECK_VARIANT_PARAM.
tests/test_multiple_self_check_rails.py	Updated all tests to use $variant= and namespaced prompt keys; adds new tests for bare-task deprecation and get_self_check_prompt_task namespacing.
tests/v2_x/test_multiple_self_check_rails.py	New Colang 2 coverage for multiple custom input/output self-check tasks using the $variant= syntax.
tests/recorded/rails/library/test_self_check.py	Adds recorded public-API tests covering multi-task sequential, parallel, streaming, and error propagation scenarios.
tests/integrations/langchain/test_streaming.py	Updates existing streaming test to use $variant= and adds new tests for multiple custom tasks in streaming input/output rails.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Rail config YAML] --> B[_validate_self_check_rail_prompts]
    B --> C[get_self_check_prompt_task]
    C -->|default task| D[Return default task]
    C -->|custom, new-style| E[Return namespaced key]
    C -->|custom, bare key only| F[DeprecationWarning + return bare key]
    D --> G[validate prompt exists]
    E --> G
    F --> G
    subgraph Runtime
        H[self_check_input/output action] --> I[resolve_self_check_task]
        I --> J[run_self_check_task]
        J --> K[get_self_check_llm by bare name]
        J --> L[get_self_check_prompt_task namespaced key]
        L --> M[render_task_prompt]
    end

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[Rail config YAML] --> B[_validate_self_check_rail_prompts]
    B --> C[get_self_check_prompt_task]
    C -->|default task| D[Return default task]
    C -->|custom, new-style| E[Return namespaced key]
    C -->|custom, bare key only| F[DeprecationWarning + return bare key]
    D --> G[validate prompt exists]
    E --> G
    F --> G
    subgraph Runtime
        H[self_check_input/output action] --> I[resolve_self_check_task]
        I --> J[run_self_check_task]
        J --> K[get_self_check_llm by bare name]
        J --> L[get_self_check_prompt_task namespaced key]
        L --> M[render_task_prompt]
    end

Loading

_{Reviews (3): Last reviewed commit: "refactor(self-check): align internal par..." | Re-trigger Greptile}

[coderabbitai]

📝 Walkthrough

Walkthrough

Self-check task handling now derives namespaced prompt identifiers for custom tasks. Rails configuration validation checks those identifiers, while unit, integration, recorded, streaming, and v2 tests cover multi-task execution, routing, blocking, parallelism, and provider errors.

Changes

Self-check task namespacing

Layer / File(s)	Summary
Runtime task namespacing nemoguardrails/library/self_check/utils.py	Adds shared task-parameter constants and derives namespaced prompt tasks for rendering, token settings, LLM metadata, warnings, and output parsing.
Namespaced prompt validation nemoguardrails/rails/llm/config.py	Validates default or $task-scoped self-check prompt templates and reports missing scoped templates.
Configuration and task-resolution coverage tests/test_multiple_self_check_rails.py, tests/test_rails_config.py, tests/v2_x/test_multiple_self_check_rails.py	Updates prompt bindings and tests for custom-task routing, default-task resolution, missing prompts, and sequential or parallel execution.
Recorded multi-self-check scenarios tests/recorded/rails/library/configs.py, tests/recorded/rails/library/configs/*, tests/recorded/rails/library/test_self_check.py, tests/recorded/rails/library/cassettes/test_self_check/*	Adds configurations, recorded provider interactions, and Rails tests for allowed, blocked, streaming, parallel, sequential, and invalid-model cases.
LangChain streaming coverage tests/integrations/langchain/test_streaming.py	Tests task-specific model routing and streamed input/output self-check behavior in sequential and parallel modes.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RailsConfig
  participant SelfCheckRuntime
  participant TaskSpecificLLM
  participant RailsStream
  RailsConfig->>RailsConfig: validate self_check_* $task prompt
  RailsStream->>SelfCheckRuntime: run input/output self-check
  SelfCheckRuntime->>TaskSpecificLLM: submit namespaced prompt
  TaskSpecificLLM-->>SelfCheckRuntime: return safe or unsafe
  SelfCheckRuntime-->>RailsStream: return parsed self-check result
  RailsStream-->>RailsStream: emit normal or guardrail-violation chunks

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 10.71% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Results For Major Changes	⚠️ Warning	The PR is a major feature/refactor, but the description only mentions added coverage and checkboxes; it doesn’t document actual test results or verification details.	Add a short testing section with commands run and outcomes (or CI/benchmark results if relevant) to the PR description.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: namespacing custom self-check task prompts.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch pouyanpi/self-check-prompt-namespacing

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integrations/langchain/test_streaming.py`:
- Around line 574-579: Strengthen the streaming assertion around
chat.app.stream_async by verifying that no collected chunk contains the
blocked_data_leakage marker before asserting the final error parameter. Keep the
existing error parsing and final self-check assertion intact while explicitly
enforcing that blocked content is never emitted.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 19001ae9-2b4e-40cf-9ab8-d40c28e18706

📥 Commits

Reviewing files that changed from the base of the PR and between 2283574 and 46201ff.

📒 Files selected for processing (20)

• nemoguardrails/library/self_check/utils.py
• nemoguardrails/rails/llm/config.py
• tests/integrations/langchain/test_streaming.py
• tests/recorded/rails/library/cassettes/test_self_check/test_multiple_self_check_generate_async_parallel_runs_all_tasks.yaml
• tests/recorded/rails/library/cassettes/test_self_check/test_multiple_self_check_generate_async_sequential_blocks_output.yaml
• tests/recorded/rails/library/cassettes/test_self_check/test_multiple_self_check_input_provider_error_raises.yaml
• tests/recorded/rails/library/cassettes/test_self_check/test_multiple_self_check_input_second_task_blocks.yaml
• tests/recorded/rails/library/cassettes/test_self_check/test_multiple_self_check_input_tasks_allow.yaml
• tests/recorded/rails/library/cassettes/test_self_check/test_multiple_self_check_output_rails_streaming[parallel-allows].yaml
• tests/recorded/rails/library/cassettes/test_self_check/test_multiple_self_check_output_rails_streaming[sequential-blocks].yaml
• tests/recorded/rails/library/cassettes/test_self_check/test_multiple_self_check_output_second_task_blocks.yaml
• tests/recorded/rails/library/cassettes/test_self_check/test_multiple_self_check_output_second_task_blocks_fake_main_stream.yaml
• tests/recorded/rails/library/cassettes/test_self_check/test_multiple_self_check_stream_async_runs_input_tasks.yaml
• tests/recorded/rails/library/configs.py
• tests/recorded/rails/library/configs/openai_multi_self_check/config.yml
• tests/recorded/rails/library/configs/openai_multi_self_check_invalid_model/config.yml
• tests/recorded/rails/library/test_self_check.py
• tests/test_multiple_self_check_rails.py
• tests/test_rails_config.py
• tests/v2_x/test_multiple_self_check_rails.py