fix(epp): route completion prompts in Rust EPP

[sttts]

Overview

Rust ext-proc EPP handled every request body as chat completions. This PR teaches it to route /v1/completions bodies through the completion request type so text prompts, token-ID prompts, and nvext routing hints keep their intended semantics.

Summary

• Parse completion request JSON as NvCreateCompletionRequest instead of forcing chat completion parsing.
• Preserve completion token-ID prompts and raw text prompt tokenization for routing.
• Avoid injecting synthetic nvext.token_data for batch completion prompts and prompt-embedding completions.
• Forward nvext priority, strict priority, and routing constraints for chat and completion requests.

Details

• Batch completion prompts are reduced to a representative first prompt for router scoring only until batch-aware routing has an explicit policy.
• Prompt-embedding completions route with empty token IDs, matching the backend preprocessing path where embeddings are authoritative.
• Deprecated latency_sensitivity priority hints are clamped the same way as priority.

Where should the reviewer start?

Start with deploy/inference-gateway/ext-proc/src/epp.rs, especially Router::tokenize, prepare_completion_prompt_for_routing, and the PickResult construction in pick().

Validation

• git diff --check
• Pre-commit hooks passed during commits.
• Not run locally: Rust cargo/rustfmt; this environment has no Rust toolchain installed.

Related Issues

Fixes #11041.
Related to #10872.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi sttts! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[coderabbitai]

Walkthrough

The Rust EPP router gains support for /v1/completions requests alongside existing chat-completion routing. tokenize now branches on request type, normalizes completion prompt shapes, extracts routing_constraints from nvext, and returns it as a fourth tuple element. route_prefill and route_decode accept and forward the constraints into router queries; pick() is wired accordingly. Helper extraction functions are refactored to operate on Option<&NvExt>.

Changes

EPP RoutingConstraints propagation and completion prompt support

Layer / File(s)	Summary
Helper functions: extraction and prompt normalization deploy/inference-gateway/ext-proc/src/epp.rs	Adds extract_strict_priority, extract_routing_constraints, and normalize_completion_prompt_for_routing helpers; refactors extract_priority_jump to accept Option<&NvExt>; adds imports for NvExt, routing_constraints_to_kv, and completion request types.
tokenize: completion branching and routing_constraints return deploy/inference-gateway/ext-proc/src/epp.rs	Reworks Router::tokenize to branch on completion vs chat-completion, normalize prompts via the new helper, extract all three nvext fields, and return (Vec<u32>, f64, u32, RoutingConstraints).
route_prefill / route_decode accept routing_constraints deploy/inference-gateway/ext-proc/src/epp.rs	Adds routing_constraints: RoutingConstraints parameter to both routing methods and passes it into the underlying prefill and decode router queries.
pick() hot path wired to new signatures deploy/inference-gateway/ext-proc/src/epp.rs	Destructures routing_constraints from the tokenize result and forwards it into both route_prefill and route_decode calls.
Tests for extraction and completion handling deploy/inference-gateway/ext-proc/src/epp.rs	Updates existing extraction tests to match refactored signatures; adds tests for completion nvext priority, strict priority, prompt normalization (string/array/nested), and routing_constraints lifting.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly names the main change: routing completion prompts in Rust EPP.
Linked Issues check	✅ Passed	The changes preserve completion prompt semantics, handle string-array normalization, and forward nvext constraints as required by #11041.
Out of Scope Changes check	✅ Passed	The changes stay focused on Rust EPP completion routing and related tests, with no obvious unrelated additions.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description check	✅ Passed	The PR description matches the required template and includes overview, details, review starting point, validation, and related issue links.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

deploy/inference-gateway/ext-proc/src/epp.rs (1)

529-536: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Clamp the deprecated priority alias too.

priority is clamped, but latency_sensitivity is returned as-is. A negative deprecated alias can still produce a negative priority_jump, contradicting the helper contract.

🐛 Proposed fix

             h.priority
-                .map(|p| p.max(0) as f64)
+                .map(|p| p as f64)
                 .or(h.latency_sensitivity)
+                .map(|p| p.max(0.0))

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/inference-gateway/ext-proc/src/epp.rs` around lines 529 - 536, The
`extract_priority_jump` helper is still allowing negative values through the
deprecated `latency_sensitivity` alias, so clamp that fallback the same way
`priority` is clamped. Update the `NvExt`/`agent_hints` extraction logic in
`extract_priority_jump` so both branches normalize to a non-negative `f64`,
preserving the helper’s contract regardless of which hint field is present.

🧹 Nitpick comments (1)

deploy/inference-gateway/ext-proc/src/epp.rs (1)

1218-1267: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Add coverage for the actual completion tokenize branch.

These cases stop at helper-level extraction/normalization. Add a test that calls Router::tokenize for raw text, token-ID arrays, and the chosen string-array policy so regressions in completion deserialization or gather_tokens are caught. The PR objective calls for focused coverage of completion token-ID arrays, raw text completion prompts, and the chosen string-array policy.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/inference-gateway/ext-proc/src/epp.rs` around lines 1218 - 1267, The
current tests only cover helper-level prompt normalization and hint extraction,
so add coverage for the actual completion tokenize flow in Router::tokenize.
Introduce a test that exercises raw text prompts, token-ID array prompts, and
the chosen string-array policy through Router::tokenize so completion
deserialization and gather_tokens are validated end to end. Use the existing
completion prompt helpers and Router::tokenize as the key symbols to locate the
code, and keep the assertions focused on the routed token output for each input
form.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@deploy/inference-gateway/ext-proc/src/epp.rs`:
- Around line 529-536: The `extract_priority_jump` helper is still allowing
negative values through the deprecated `latency_sensitivity` alias, so clamp
that fallback the same way `priority` is clamped. Update the
`NvExt`/`agent_hints` extraction logic in `extract_priority_jump` so both
branches normalize to a non-negative `f64`, preserving the helper’s contract
regardless of which hint field is present.

---

Nitpick comments:
In `@deploy/inference-gateway/ext-proc/src/epp.rs`:
- Around line 1218-1267: The current tests only cover helper-level prompt
normalization and hint extraction, so add coverage for the actual completion
tokenize flow in Router::tokenize. Introduce a test that exercises raw text
prompts, token-ID array prompts, and the chosen string-array policy through
Router::tokenize so completion deserialization and gather_tokens are validated
end to end. Use the existing completion prompt helpers and Router::tokenize as
the key symbols to locate the code, and keep the assertions focused on the
routed token output for each input form.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 336e29cd-d18f-4117-bae9-24610017b2da

📥 Commits

Reviewing files that changed from the base of the PR and between 59e76f4 and 18fe651.

📒 Files selected for processing (1)

• deploy/inference-gateway/ext-proc/src/epp.rs

[sttts]

Addressed the current non-inline review items:

• Clamped deprecated latency_sensitivity the same way as priority in extract_priority_jump (ac85af7bc4, formatting follow-up 0aed71504b).
• Added prompt-policy coverage for text completions, single string arrays, token-ID arrays, multi-prompt batches, token batches, routing constraints, and prompt-embeds token injection behavior.
• The requested full Router::tokenize unit coverage would require constructing Router with live KV routers/runtime/pod reflector state; I kept this PR to focused helper/unit coverage rather than adding a broad test harness in the hot-path router wrapper.

[dynamo-ops]

/ok to test c0f922e

[sttts]

/ok to test a72aff2

[sttts]

Rebased onto latest main and pushed a72aff222c6887e75bace071941774debd58da67. The existing completion token-data review fixes were carried forward, and I added coverage for preserving nvext.cache_salt on the prompt-embeds routing-only path introduced by the rebase with main. /ok to test has been posted for the new head.

[dynamo-ops]

/ok to test a72aff2

[sttts]

/ok to test 8512911

[dynamo-ops]

/ok to test 8512911

[sttts]

/ok to test 7949c59

[dynamo-ops]

/ok to test 7949c59

[sttts]

Rebase and CI follow-ups are done. Current head is 7949c59fcd9d7c35b70fe48a5579b60db8830743 on latest main. I fixed the Rust EPP CI fallout from the rebase (gather_tokens is async on main, and route_decode now groups decode routing arguments to satisfy clippy), reran the transient lychee failure, and all GitHub checks are now green. Existing inline review threads remain resolved.