feat(routing): session-routing plugin layer for per-session router affinity

One plugin category (session_routing) + one CLI flag (--session-routing
<mode>, with repeatable --session-routing-opt key=value) covers every
mechanism for telling an external router which session a request belongs
to, so all of a conversation's turns re-land on the replica holding its
KV prefix. "Session" (live instance keyed by x_correlation_id), not
"conversation" (dataset template), is the affinity identity. Exactly
one mode runs per invocation; the selected plugin is instantiated once
per worker by InferenceClient and invoked at the request-serialization
chokepoint.

Built-in modes:
- dynamo_headers (preset): X-Dynamo-Session-ID plus
  X-Dynamo-Parent-Session-ID on subagent children, for a Dynamo frontend
  running --router-session-affinity-ttl-secs.
- dynamo_nvext: nvext.session_control request-body metadata -- bind
  (with timeout, option timeout_seconds, default 300) on every non-final
  turn, close on the final turn -- for Dynamo builds that implement
  session_control. Plugin values win over dataset-shipped
  session_control keys; other nvext content is preserved.
- smg_routing_key (preset): X-SMG-Routing-Key for the SGLang Model
  Gateway manual policy.
- session_id_header (preset): additive X-Session-ID, parameterless.
- identity_headers: fully generic tiered identity headers emitting
  exactly what is configured, nothing by default. Options (each a
  comma-separable name list): session (this session's correlation ID),
  parent (immediate parent's ID, omitted for roots), root (session-tree
  root's ID -- whole-tree affinity, e.g. pinning an entire agent tree to
  one replica for prefix-cache locality). Requires at least one name
  across tiers; names are validated as RFC 9110 tokens at config time
  and must be case-insensitively unique across all tiers combined.
  Each header preset is documented with its identity_headers equivalent.

Removed (superseded): --use-dynamo-conv-aware-routing /
--use-dynamo-session-control, --use-legacy-dynamo-session-control,
--dynamo-session-timeout-seconds, the dynamo_session_control module,
AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID,
AIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID, and the
base_transports env-toggle header blocks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>

Author: ajcasagrande

README.md

@@ -200,7 +200,7 @@ Log File: /home/user/aiperf/artifacts/granite4:350m-openai-chat-concurrency1/log
 | [CLI Options](docs/cli-options.md) | Complete command and option reference |
 | [Metrics Reference](docs/metrics-reference.md) | All metric definitions, formulas, and requirements |
 | [Environment Variables](docs/environment-variables.md) | All `AIPERF_*` configuration variables |
-| [Plugin System](docs/plugins/plugin-system.md) | Plugin architecture, 27 categories, creation guide |
+| [Plugin System](docs/plugins/plugin-system.md) | Plugin architecture, 31 categories, creation guide |
 | [Creating Plugins](docs/plugins/creating-your-first-plugin.md) | Step-by-step plugin tutorial |
 | [Accuracy Benchmarks](docs/accuracy/accuracy_stubs.md) | Accuracy evaluation stubs and datasets |
 | [Benchmark Modes](docs/benchmark-modes/trace-replay.md) | Trace replay and timing modes |

docs/cli-options.md

@@ -266,21 +266,15 @@ Use the legacy 'max_tokens' field instead of 'max_completion_tokens' in request
 Use server-reported token counts from API usage fields instead of client-side tokenization. When enabled, tokenizers are still loaded (needed for dataset generation) but tokenizer.encode() is not called for computing metrics. Token count fields will be None if the server does not provide usage information. For OpenAI-compatible streaming endpoints (chat/completions), stream_options.include_usage is automatically configured when this flag is enabled. Recommended whenever the AIPerf tokenizer can disagree with the server's tokenizer (e.g. unmatched tokenizer revision, vendor-specific BPE merges, or chat templates that differ from the server) — this most often shows up as an output sequence length (OSL) mismatch even when the server is honoring the request (e.g. with ignore_eos=true).
 <br/>_Flag (no value required)_
 
-#### `--use-dynamo-conv-aware-routing`, `--use-dynamo-session-control`
+#### `--session-routing` `<str>`
 
-Emit Dynamo nvext.session_control in OpenAI-compatible request bodies so Dynamo can bind all turns from the same replayed conversation lineage to the same backend worker. This is only intended for Dynamo frontends that implement session_control.
-<br/>_Flag (no value required)_
-
-#### `--use-legacy-dynamo-session-control`
-
-Emit the legacy Dynamo nvext.session_control lifecycle that released Dynamo (v1.2.x) understands: action 'open' on the first turn, session_id only on intermediate turns, and action 'close' on the final turn. Use this when the target Dynamo predates the 'bind' action (added in v1.3.0-dev); otherwise 'bind' is rejected with an HTTP 400. Requires --use-dynamo-conv-aware-routing, and the Dynamo deployment must expose a worker session_control endpoint for 'open' to take effect.
-<br/>_Flag (no value required)_
+Session-aware routing mode: stamps per-session identity on every request for router affinity. Presets: dynamo_headers (X-Dynamo-Session-ID + parent header), smg_routing_key (X-SMG-Routing-Key for the SGLang Model Gateway manual policy), session_id_header (additive X-Session-ID). Generic: identity_headers (any name(s) per identity tier -- session/parent/root -- via --session-routing-opt). Body-based: dynamo_nvext (nvext.session_control bind/close request-body metadata; --session-routing-opt timeout_seconds=N).
+<br/>_Choices: [`dynamo_headers`, `dynamo_nvext`, `smg_routing_key`, `session_id_header`, `identity_headers`]_
 
-#### `--dynamo-session-timeout-seconds` `<int>`
+#### `--session-routing-opt` `<list>`
 
-Dynamo nvext.session_control timeout in seconds when --use-dynamo-conv-aware-routing is enabled.
-<br/>_Constraints: ≥ 1_
-<br/>_Default: `300`_
+Repeatable key=value option for the selected --session-routing mode (e.g. --session-routing-opt timeout_seconds=600), validated against the plugin's Options model. Commas inside the value are passed through to the plugin (repeat the flag for multiple opts).
+<br/>_Default: `[]`_
 
 #### `--connection-reuse-strategy` `<str>`
 
@@ -1306,7 +1300,7 @@ Explore AIPerf plugins: aiperf plugins [category] [type]
 #### `--category` `<str>`
 
 Category to explore.
-<br/>_Choices: [`accumulator`, `accuracy_benchmark`, `accuracy_grader`, `analyzer`, `api_router`, `arrival_pattern`, `communication`, `communication_client`, `console_exporter`, `custom_dataset_loader`, `data_exporter`, `dataset_backing_store`, `dataset_client_store`, `dataset_composer`, `dataset_sampler`, `endpoint`, `gpu_telemetry_collector`, `plot`, `public_dataset_loader`, `ramp`, `record_processor`, `service`, `service_manager`, `stream_exporter`, `timing_strategy`, `transport`, `ui`, `url_selection_strategy`, `zmq_proxy`]_
+<br/>_Choices: [`accumulator`, `accuracy_benchmark`, `accuracy_grader`, `analyzer`, `api_router`, `arrival_pattern`, `communication`, `communication_client`, `console_exporter`, `custom_dataset_loader`, `data_exporter`, `dataset_backing_store`, `dataset_client_store`, `dataset_composer`, `dataset_sampler`, `endpoint`, `gpu_telemetry_collector`, `plot`, `public_dataset_loader`, `ramp`, `record_processor`, `service`, `service_manager`, `session_routing`, `stream_exporter`, `timing_strategy`, `transport`, `ui`, `url_selection_strategy`, `zmq_proxy`]_
 
 #### `--name` `<str>`
 

docs/environment-variables.md

@@ -135,8 +135,6 @@ HTTP client socket and connection configuration. Controls low-level socket optio
 | `AIPERF_HTTP_REQUEST_CANCELLATION_SEND_TIMEOUT` | `300.0` | ≥ 10.0, ≤ 3600.0 | Safety net timeout in seconds for waiting for HTTP request to be fully sent when request cancellation is enabled. Used as fallback when no explicit timeout is configured to prevent hanging indefinitely while waiting for the request to be written to the socket. |
 | `AIPERF_HTTP_IP_VERSION` | `'4'` | — | IP version for HTTP socket connections. Options: '4' (AF_INET, default), '6' (AF_INET6), or 'auto' (AF_UNSPEC, system chooses). |
 | `AIPERF_HTTP_TRUST_ENV` | `False` | — | Trust environment variables for HTTP client configuration. When enabled, aiohttp will read proxy settings from HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables. |
-| `AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID` | `False` | — | Also send X-Session-ID with the stable X-Correlation-ID value. Use this when an external router requires a session-affinity header. |
-| `AIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID` | `False` | — | Also send X-SMG-Routing-Key with the stable X-Correlation-ID value. Use this with the SGLang Model Gateway manual routing policy. |
 | `AIPERF_HTTP_VIDEO_POLL_INTERVAL` | `0.1` | ≥ 0.001, ≤ 10.0 | Interval in seconds between status polls for async video generation jobs. Lower values provide faster completion detection but increase server load. Applies to the aiohttp transport. |
 
 ## LOGGING

docs/plugins/plugin-system.md

@@ -100,7 +100,7 @@ for entry, cls in plugins.iter_all(PluginType.ENDPOINT):
 
 ## Plugin Categories
 
-AIPerf supports 27 plugin categories organized by function:
+AIPerf supports 31 plugin categories organized by function:
 
 ### Timing Categories
 
@@ -128,6 +128,59 @@ AIPerf supports 27 plugin categories organized by function:
 | `endpoint` | `EndpointType` | API endpoint implementations (chat, completions, embeddings, etc.) |
 | `transport` | `TransportType` | Network transport (HTTP via aiohttp) |
 
+### Session Routing Category
+
+| Category | Enum | Description |
+|----------|------|-------------|
+| `session_routing` | `SessionRoutingType` | Stamps per-session identity (headers or body metadata) onto outbound requests so an external router pins every turn of a session to one worker; selected via `--session-routing` |
+
+**Purpose.** A session-routing plugin gives an external router (SGLang Model Gateway, Dynamo, a
+generic session-affinity load balancer) the identity of the session a request belongs to, so all of
+a conversation's turns re-land on the replica holding its KV prefix. Exactly one mode runs per
+invocation; the selected plugin is instantiated once per worker by `InferenceClient` and invoked at
+the request-serialization chokepoint. The base class is `SessionRoutingBase`
+(`src/aiperf/workers/session_routing.py`); passthrough is the default (no headers, body unchanged).
+
+**Built-ins:**
+
+| Name | Class | Description |
+|------|-------|-------------|
+| `dynamo_headers` | `DynamoHeadersRouting` | Dynamo session affinity via `X-Dynamo-Session-ID`, plus `X-Dynamo-Parent-Session-ID` on subagent children. No options. |
+| `dynamo_nvext` | `DynamoNvextRouting` | Dynamo session affinity via `nvext.session_control` request-body metadata (bind on non-final turns, close on the final turn). Only for Dynamo builds that implement `session_control`. Option: `timeout_seconds` (default 300). |
+| `smg_routing_key` | `SmgRoutingKeyRouting` | SGLang Model Gateway `manual`-policy stickiness via `X-SMG-Routing-Key`. No options. |
+| `session_id_header` | `SessionIdHeaderRouting` | Preset: additive `X-Session-ID` header carrying the session's correlation ID. No options. |
+| `identity_headers` | `IdentityHeadersRouting` | Fully generic tiered identity headers: emits exactly what you configure, nothing by default. Options (each a comma-separable name list): `session` (this session's ID), `parent` (immediate parent's ID; omitted for roots), `root` (session-tree root's ID, for whole-tree affinity). Requires at least one name across tiers; names must be unique (case-insensitive) across all tiers combined, including within a single tier. |
+
+The three header presets are expressible via `identity_headers`: `session_id_header` ≡
+`session=X-Session-ID`, `smg_routing_key` ≡ `session=X-SMG-Routing-Key`, and `dynamo_headers` ≡
+`session=X-Dynamo-Session-ID` + `parent=X-Dynamo-Parent-Session-ID`. Reach for `identity_headers`
+when a router needs custom names, several headers stamped with the same value (layered routers),
+or tree-scoped affinity; reach for a preset when it already says exactly what you mean.
+
+**Options (`--session-routing-opt key=value`).** Each plugin exposes an `Options` Pydantic model
+(`extra="forbid"`, so unknown keys are rejected at startup). Repeated `--session-routing-opt`
+pairs populate it; values are coerced to the model's field types and canonicalized at config
+resolution, so downstream code always sees typed values. Commas inside a value are passed through
+to the plugin (repeat the flag for multiple opts). `--session-routing-opt` without
+`--session-routing` is an error, and parameterless modes (`dynamo_headers`, `smg_routing_key`,
+`session_id_header`) reject every opt key.
+
+**`mutates_body` and the PAYLOAD_BYTES fast path.** A plugin sets the `mutates_body` class var to
+`True` when `transform_body` changes the payload (only `dynamo_nvext` does). Body-mutating modes are
+incompatible with the verbatim PAYLOAD_BYTES mmap fast path and are gated off it at three points:
+dataset build, cache hit, and runtime. Header-only modes leave the body untouched and keep the fast
+path. `transform_body` must never mutate its input — the structured path shares cached
+`Turn.raw_payload` dicts with the dataset — so it returns a new dict.
+
+**`on_session_end` contract.** Fires strictly after the session's last worker-side activity, on
+every terminal path (final turn, cancellation, terminal context overflow, cancel-before-start). It
+is post-session cleanup only and must be idempotent (default no-op).
+
+**Stateful-plugin rule.** A stateful plugin must key its instance state on `ctx.x_correlation_id`
+only. A session tree deliberately spans workers, so tree-keyed worker state fragments across
+processes. For tree-scoped behavior, use the stateless per-request context facts
+`root_correlation_id` and `is_tree_final` instead of accumulating state.
+
 ### Processing Categories
 
 | Category | Enum | Description |

src/aiperf/common/config/config_defaults.py

@@ -48,9 +48,6 @@ class EndpointDefaults:
     CONNECTION_REUSE_STRATEGY = ConnectionReuseStrategy.POOLED
     DOWNLOAD_VIDEO_CONTENT = False
     REQUEST_CONTENT_TYPE = None
-    USE_DYNAMO_CONV_AWARE_ROUTING = False
-    USE_LEGACY_DYNAMO_SESSION_CONTROL = False
-    DYNAMO_SESSION_TIMEOUT_SECONDS = 300
     # Readiness probe defaults. Timeout 0 disables the probe (the default);
     # any positive value enables it. Interval is only consulted when the
     # probe is enabled but is validated positive so mis-configuration