Skip to content

[Rollout] Add opt-in group-scoped session affinity#2206

Open
chengcuiping wants to merge 1 commit into
THUDM:mainfrom
chengcuiping:feat/group-session-affinity
Open

[Rollout] Add opt-in group-scoped session affinity#2206
chengcuiping wants to merge 1 commit into
THUDM:mainfrom
chengcuiping:feat/group-session-affinity

Conversation

@chengcuiping

@chengcuiping chengcuiping commented Jul 14, 2026

Copy link
Copy Markdown

Summary

  • Add an opt-in group scope for automatically assigned rollout session IDs.
  • Preserve the existing per-sample behavior by default.
  • Reuse the existing X-SMG-Routing-Key path for consistent-hashing routing.

Motivation

  • Slime already preserves session affinity across turns of one sample.
  • Grouped sibling rollouts may also need a shared routing identity.
  • Previously this required callers to pre-populate the same session_id manually.

Behavior

  • sample: unchanged default, one generated ID per sample.
  • group: one generated or inherited ID per rollout group.
  • conflicting explicit IDs fail closed.
  • group scope requires consistent_hashing.
  • non-consistent-hashing routing remains unchanged.

Testing

CPU-only qualification was run against the latest upstream main after cherry-picking this PR.

Additional production-path coverage was added for:

  • generate_and_rm_group -> generate -> HTTP post
  • verifying that 3 sibling requests from the same group carry the same X-SMG-Routing-Key
  • preserving the default per-sample routing behavior for non-grouped generation

Commands:

CUDA_VISIBLE_DEVICES='' python -m compileall -q slime/rollout/sglang_rollout.py slime/utils/arguments.py tests/test_group_session_affinity.py tests/test_megatron_argument_validation.py
CUDA_VISIBLE_DEVICES='' python -m pytest tests/test_group_session_affinity.py tests/test_sample.py tests/test_megatron_argument_validation.py -m unit
CUDA_VISIBLE_DEVICES='' pre-commit run --files slime/utils/arguments.py slime/rollout/sglang_rollout.py tests/test_megatron_argument_validation.py tests/test_group_session_affinity.py docs/en/advanced/sglang-config.md docs/zh/advanced/sglang-config.md

Qualification result: 31 tests passed. git diff --check passed. Testing was CPU-only; no GPU was used.

Limitations

  • This controls routing identity only.
  • It does not implement KV pinning, eviction, transfer, TTL, or cache management.
  • It does not guarantee a cache hit or claim a throughput or other performance improvement.
  • Group affinity may reduce cross-worker load-balancing flexibility.

Related work

Related to #1469.
Builds on the consistent-hashing routing path introduced by #1588.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant