[codex] Rewrite InferenceServer docs for Ray Serve and Dynamo

[lbliii]

Summary

• replace the obsolete InferenceModelConfig guide with typed Ray Serve and NVIDIA Dynamo configuration
• add runnable Ray Serve, Dynamo aggregated, and Dynamo disaggregated examples
• document model/server validation, multi-model behavior, static replicas, role overrides, router modes, KV-event constraints, multimodal routing, runtime environments, placement, lifecycle, and HAProxy ingress
• add a concrete migration from InferenceModelConfig
• update the LLM client, NeMo Data Designer, and release-note references to current typed APIs

Why

The published quickstart imported InferenceModelConfig, which no longer exists on main, and described Ray Serve as the only backend. Users could not run the examples or discover the current Dynamo serving surface.

User impact

Users can now choose RayServeModelConfig or DynamoVLLMModelConfig, size aggregated or disaggregated deployments, configure routing and runtime environments, and understand the tested architecture/dependency boundaries before starting a cluster.

Validation

• fern check — 0 errors
• fern docs broken-links — no errors in changed pages; 22 pre-existing errors remain in older API-reference pages
• parsed all 32 Python fences in the changed guides with ast.parse
• instantiated the documented Ray Serve, aggregated Dynamo, disaggregated Dynamo, role, and router configurations against current main
• focused serve/config/runtime-env unit tests — 22 passed, 1 skipped
  • excluded the repository-wide Ray fixture because this sandbox hostname does not resolve; no focused assertion failed
• git diff --check

Closes #2146

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR rewrites the Inference Server documentation to replace the removed InferenceModelConfig with the current typed RayServeModelConfig and DynamoVLLMModelConfig APIs, and propagates those import fixes into the LLM client and NeMo Data Designer guides.

• inference-server.mdx: Full rewrite covering Ray Serve quickstart, Dynamo aggregated and disaggregated serving, routing modes (including the kv_events auto-enable behavior), runtime environments, resource placement, HAProxy ingress, migration path from InferenceModelConfig, and a troubleshooting section.
• llm-client.mdx / nemo-data-designer.mdx: Import lines updated from InferenceModelConfig to RayServeModelConfig; lifecycle usage is correct.
• .secrets.baseline: Stale entries for deleted docs/ files removed; line-number reference for inference-server.mdx updated to match the rewritten file.

Confidence Score: 5/5

Documentation-only change; no production code is modified. All changed pages remove references to the deleted InferenceModelConfig class and replace them with the current typed APIs.

Every changed file is Markdown/MDX documentation or the secrets baseline. The code snippets use correct imports that match the current main-branch API. The two accuracy concerns raised in previous review threads (kv_events auto-enable behavior and DynamoRoleConfig validation attribution) are both addressed in the new text.

llm-client.mdx line 141 has a stale '(Ray Serve + vLLM)' description that was not updated to reflect the new Dynamo backend.

Important Files Changed

Filename	Overview
fern/versions/main/pages/curate-text/synthetic/inference-server.mdx	Complete rewrite replacing InferenceModelConfig with RayServeModelConfig and DynamoVLLMModelConfig; covers aggregated/disaggregated Dynamo, routing, runtime environments, placement, and migration. kv_events auto-enable and DynamoRoleConfig validation accuracy from previous threads are both addressed.
fern/versions/main/pages/curate-text/synthetic/llm-client.mdx	Import updated from InferenceModelConfig to RayServeModelConfig; the inline description on line 141 still says '(Ray Serve + vLLM)' and was not updated to reflect the new Dynamo backend.
fern/versions/main/pages/curate-text/synthetic/nemo-data-designer.mdx	Import and variable type updated from InferenceModelConfig to RayServeModelConfig; lifecycle (start/stop) is correct in the end-to-end example.
fern/versions/main/pages/about/release-notes/index.mdx	Release note updated to replace InferenceModelConfig reference with the typed RayServeModelConfig/DynamoVLLMModelConfig description.
.github/workflows/config/.secrets.baseline	Removes stale entries for deleted docs/ files and updates the line number for the existing inference-server.mdx secret reference; generated_at timestamp refreshed.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[InferenceServer] --> B{backend=}
    B -->|omitted / RayServeServerConfig| C[Ray Serve backend]
    B -->|DynamoServerConfig| D[NVIDIA Dynamo backend]

    C --> E[RayServeModelConfig]
    E --> E1[deployment_config\nautoscaling / replicas]
    E --> E2[engine_kwargs\nvLLM settings]
    E --> E3[runtime_env\nRay env per model]

    D --> F{mode}
    F -->|aggregated| G[DynamoVLLMModelConfig\nnum_replicas]
    F -->|disagg| H[DynamoVLLMModelConfig\nprefill + decode roles]

    H --> I[DynamoRoleConfig\nprefill num_replicas\nengine_kwargs override]
    H --> J[DynamoRoleConfig\ndecode num_replicas\nengine_kwargs override]

    D --> K[DynamoServerConfig\netcd / NATS / router]
    K --> L[DynamoRouterConfig\nmode / kv_events]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[InferenceServer] --> B{backend=}
    B -->|omitted / RayServeServerConfig| C[Ray Serve backend]
    B -->|DynamoServerConfig| D[NVIDIA Dynamo backend]

    C --> E[RayServeModelConfig]
    E --> E1[deployment_config\nautoscaling / replicas]
    E --> E2[engine_kwargs\nvLLM settings]
    E --> E3[runtime_env\nRay env per model]

    D --> F{mode}
    F -->|aggregated| G[DynamoVLLMModelConfig\nnum_replicas]
    F -->|disagg| H[DynamoVLLMModelConfig\nprefill + decode roles]

    H --> I[DynamoRoleConfig\nprefill num_replicas\nengine_kwargs override]
    H --> J[DynamoRoleConfig\ndecode num_replicas\nengine_kwargs override]

    D --> K[DynamoServerConfig\netcd / NATS / router]
    K --> L[DynamoRouterConfig\nmode / kv_events]

Loading

_{Reviews (6): Last reviewed commit: "Merge branch 'main' into codex/docs-infe..." | Re-trigger Greptile}

[greptile-apps]

Disaggregated mode silently overrides kv_events=False

The docs state that kv_events=False "uses approximate tree-based tracking," implying the default applies for disaggregated serving. In practice, DynamoBackend._resolve_effective_router computes kv_events = mode == "kv" and (mode_was_auto_picked or router.kv_events). When auto-routing selects "kv" for any disaggregated model (mode_was_auto_picked=True), kv_events is forced to True regardless of the DynamoRouterConfig default of kv_events=False. A user who relies on the default router config expecting tree-based tracking with disaggregated serving will actually get event-backed KV routing. The only exception is when an HMA publisher is detected and the user explicitly left kv_events=False. This auto-enable behavior should be documented here to avoid surprises.

[greptile-apps]

DynamoRoleConfig.num_replicas validation described inaccurately

The table says the constraint is "at least one prefill and one decode replica," presenting it as a DynamoRoleConfig-level rule. In the code, DynamoRoleConfig.__post_init__ only enforces >= 0; the >= 1 requirement is checked by DynamoVLLMModelConfig.__post_init__. A user can successfully construct DynamoRoleConfig(num_replicas=0) and only hit the error when that config is embedded in a DynamoVLLMModelConfig. Attributing the constraint to the model config rather than the role config is more accurate.

Suggested change

	| `num_replicas` | `int` | `1` | Number of workers for this role. Disaggregated models require at least one prefill and one decode replica. |
	| `num_replicas` | `int` | `1` | Number of workers for this role. Must be `>= 0`; `DynamoVLLMModelConfig` enforces that both prefill and decode are `>= 1` for disaggregated mode. |

[github-actions]

🌿 Preview your docs: https://nvidia-preview-codex-docs-inference-server-dynamo.docs.buildwithfern.com/nemo/curator

Here are the markdown pages you've updated:

• nemo/curator/about/release-notes
• nemo/curator/curate-text/synthetic/inference-server
• nemo/curator/curate-text/synthetic/llm-client
• nemo/curator/curate-text/synthetic/nemo-data-designer

[lbliii]

@abhinavg4 friendly ping on this docs PR for the #2118 release-docs workstream — when you have a moment, could you take a look at the InferenceServer Ray Serve and Dynamo docs?

Docs preview: https://nvidia-preview-codex-docs-inference-server-dynamo.docs.buildwithfern.com/nemo/curator

Thanks!

[praateekmahajan]

Technically both Ray Serve and Dynamo support, all these features. There isn't one reason over the other to choose Ray Serve or Dynamo except if perf indicates that for that specific workload.

We can say that with Ray Serve you can also access using Avtor handles (however in our benchmarks it hasn't proven to be beneficial)

One important nuance to add is that Ray Serve uses vLLM from base env which is 0.18 whereas Dynamo (1.1.0) during actor creation uses Ray to create a new venv which uses vLLM 0.19, this means the startup time is higher with Dynamo since it resolves much of the installations.

[praateekmahajan]

Left some comments

[lbliii]

ok to test 364bb0a

[lbliii]

ok to test 68d8f39