Skip to content

Add DRAM KV offloading (HiCache) to GLM-5.2 NVFP4 B300 SGLang agentic recipe #2279

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Oseltamivir merged 3 commits into from
Jul 19, 2026
+52 −7
Merged

Add DRAM KV offloading (HiCache) to GLM-5.2 NVFP4 B300 SGLang agentic recipe #2279

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b92cbed
feat: DRAM KV offloading via HiCache for all glm5.2 b300 agentic points
Oseltamivir Jul 19, 2026
9a89931
chore: fill perf-changelog pr-link for glm5.2 b300 hicache entry
Oseltamivir Jul 19, 2026
dc3a492
fix: ratio-based HiCache sizing (0.75) for DSA-family GLM-5.2
Oseltamivir Jul 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 42 additions & 5 deletions benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,13 @@ set -x
# Required env vars:
# MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION,
# EP_SIZE, DP_ATTENTION
#
# KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=hicache.

source "$(dirname "$0")/../../benchmark_lib.sh"

check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION

if [[ "$KV_OFFLOADING" != "none" ]]; then
echo "Error: KV_OFFLOADING=$KV_OFFLOADING is not supported by this recipe" >&2
exit 1
fi

if [[ -n "${SLURM_JOB_ID:-}" ]]; then
echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}"
fi
Expand All @@ -46,6 +43,45 @@ install_agentic_deps
SERVER_LOG="$RESULT_DIR/server.log"
mkdir -p "$RESULT_DIR"

CACHE_ARGS=()
if require_agentic_kv_offload_backend hicache; then
# HiCache extends RadixAttention: prefixes evicted from the HBM KV pool
# spill to a pinned host pool instead of being recomputed. On the
# 1M-context agentic corpus the live working set outgrows HBM past
# conc 8 (TP8) / 64 (DP8) and the radix hit rate collapses to <0.1
# against a ~0.97 theoretical ceiling, so every turn re-prefills its
# whole history; the host tier restores those hits at C2C bandwidth.
# GLM-5.2 is DSA/MLA-family (attention_backend=dsa): every rank holds
# complete per-token KV (169.98 GB device pool per rank, replicated on
# all 8 ranks), so host capacity is controlled through the host/device
# token-capacity ratio like the DSv4 recipe, NOT a per-rank
# --hicache-size. A GB-based size of TOTAL_CPU_DRAM_GB/TP pinned the
# whole 0.80-DRAM budget (8 x 299 GB) at init on top of 465 GB of
# weights and OOM-killed the node (run 29678598595); DSv4's own
# ratio=2 default pins 2 x 170 GB x 8 = 2.7 TB here and OOMs too
# (GLM-5.2's device pool is far larger than DSv4's). Fractional 0.75
# = ~128 GB/rank = ~1.0 TB total, matching the cluster's proven ~1 TB
# host-pool envelope; validated on-node 2026-07-19 (boot + 4.2M-token
# overflow bench forcing eviction through the DSA KV+INDEXER pools).
DEFAULT_HICACHE_RATIO=0.75
HICACHE_RATIO="${HICACHE_RATIO:-$DEFAULT_HICACHE_RATIO}"
if awk -v r="$HICACHE_RATIO" -v cap="$DEFAULT_HICACHE_RATIO" 'BEGIN { exit !(r > cap) }'; then
echo "Error: HICACHE_RATIO=$HICACHE_RATIO exceeds configured limit $DEFAULT_HICACHE_RATIO" >&2
exit 1
fi
HICACHE_WRITE_POLICY="${HICACHE_WRITE_POLICY:-write_back}"
HICACHE_IO_BACKEND="${HICACHE_IO_BACKEND:-direct}"
HICACHE_MEM_LAYOUT="${HICACHE_MEM_LAYOUT:-page_first_direct}"
echo "HiCache CPU tier: ratio=$HICACHE_RATIO, capacity=${TOTAL_CPU_DRAM_GB} GB, write_policy=$HICACHE_WRITE_POLICY, io_backend=$HICACHE_IO_BACKEND, mem_layout=$HICACHE_MEM_LAYOUT"
CACHE_ARGS=(
--enable-hierarchical-cache
--hicache-ratio "$HICACHE_RATIO"
--hicache-write-policy "$HICACHE_WRITE_POLICY"
--hicache-io-backend "$HICACHE_IO_BACKEND"
--hicache-mem-layout "$HICACHE_MEM_LAYOUT"
)
fi

# With attention-DP, front the DP ranks with sglang-router using consistent
# hashing on the AIPerf correlation id so multi-turn sessions stay on the DP
# rank that holds their radix-cache prefix.
Expand Down Expand Up @@ -136,6 +172,7 @@ SGLANG_CMD=(
--mem-fraction-static 0.85
--max-running-requests "$MAX_RUNNING_REQUESTS"
"${GRAPH_ARGS[@]}"
"${CACHE_ARGS[@]}"
--watchdog-timeout 1800
--enable-metrics
)
Expand Down
4 changes: 2 additions & 2 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7873,5 +7873,5 @@ glm5.2-fp4-b300-sglang-agentic:
agentic-coding:
- dram-utilization: 0.80
search-space:
- { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 32] }
- { tp: 8, dp-attn: true, kv-offloading: none, conc-list: [48, 64, 96, 128, 192, 256, 512], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [1, 2, 4, 8, 16, 32] }
- { tp: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [48, 64, 96, 128, 192, 256, 512], router: { name: sglang-router, version: "0.3.2" } }
8 changes: 8 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4950,3 +4950,11 @@
description:
- "Bump vLLM image to v0.25.0"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2256

- config-keys:
- glm5.2-fp4-b300-sglang-agentic
description:
- "Enable DRAM KV offloading (SGLang HiCache) on all points of both arms: DSv4-style host/device token-capacity ratio, tuned to 0.75 for GLM-5.2 (DSA/MLA-family: the 169.98 GB device pool is replicated on all 8 ranks, so ratio 0.75 = ~1.0 TB pinned host pools; ratio 2 and GB-based sizing both OOM-killed 3 TB nodes), io-backend direct, mem-layout page_first_direct, write policy write_back; validated on-node with an overflow bench forcing eviction through the DSA KV+INDEXER host pools"
- "Run 29651235293 showed the 1M-context corpus working set outgrowing the HBM KV pool past conc 8 (TP8) / conc 64 (DP8): gpu_kv_cache_usage pinned at 1.0 and the radix hit rate collapsed from a ~0.97 theoretical ceiling to 0.04-0.06, so every post-knee turn re-prefilled its full history and throughput fell together with interactivity"
Comment on lines +4953 to +4958

@claude claude Bot Jul 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 This PR's title and description are English-only, but AGENTS.md (line 7) mandates that every PR title follow the <English title> / <中文标题> format and that the body include a Chinese translation section (e.g. ## 中文说明). Please add the missing Chinese title suffix and translation section.

Extended reasoning...

AGENTS.md explicitly states: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary; ...). This applies to every PR and every issue..."

This PR's title is plain English: "Add DRAM KV offloading (HiCache) to GLM-5.2 NVFP4 B300 SGLang agentic recipe" — there is no / <中文标题> suffix. The description body contains only ## Summary and ## Validation sections, both in English, with no ## 中文说明 (or equivalent) section anywhere.

Why existing tooling doesn't catch this: the repo has a codeowner-signoff-verify.yml workflow that checks for the exact English sign-off phrase ("As a PR reviewer and CODEOWNER, I have reviewed this and have"), but that workflow verifies reviewer sign-off content, not author-side PR title/description bilingual formatting — there's no automated gate that rejects an English-only PR title or body before merge.

Step-by-step proof of the violation:

  1. Open AGENTS.md and read line 7 — it defines the exact required title format <English title> / <中文标题> and requires a Chinese translation section in the body.
  2. Read this PR's actual title: "Add DRAM KV offloading (HiCache) to GLM-5.2 NVFP4 B300 SGLang agentic recipe". Compare against the required format — there is no / <中文标题> suffix.
  3. Read this PR's description — it has ## Summary and ## Validation, both English, no Chinese section.
  4. Conclude: the required bilingual elements (title suffix + body translation section) are both absent, which is a direct, checkable violation of an explicit repo-mandated rule (not a matter of style/subjectivity).

Impact: Because this is a process/documentation compliance rule rather than something that affects the benchmark scripts, config, or changelog data added in this PR (benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang.sh, configs/nvidia-master.yaml, perf-changelog.yaml all look functionally correct), it does not cause any runtime failure, incorrect benchmark behavior, or crash. It is a compliance gap against an explicit repo convention.

Fix: rename the PR title to append / <中文标题> with an idiomatic Chinese translation, and add a ## 中文说明 section to the description mirroring the ## Summary (and optionally ## Validation) content, following the translation quality bar described later in AGENTS.md (idiomatic technical Chinese, preserving model/hardware/framework names and flags in English).

- "HiCache spills evicted prefixes to host DRAM and restores them at C2C bandwidth instead of recomputing; sizing follows the qwen3.5-fp8-b300-sglang-agentic-hicache recipe (GLM-5.2 is plain GQA: one host pool per rank, GB-based --hicache-size)"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2279
Loading