Skip to content

feat: add configurable quantized indexer cache support.#1941

Merged
phantomlei3 merged 4 commits into
xLLM-AI:mainfrom
phantomlei3:feat/indexer-cache-dtype
Jul 15, 2026
Merged

feat: add configurable quantized indexer cache support.#1941
phantomlei3 merged 4 commits into
xLLM-AI:mainfrom
phantomlei3:feat/indexer-cache-dtype

Conversation

@phantomlei3

@phantomlei3 phantomlei3 commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Description

Adds configurable INT8 quantization for indexer caches on the MLU backend via --indexer_cache_dtype. The change covers cache allocation and capacity estimation, scale tensors, distributed/RDMA transfer, worker configuration propagation, documentation, and comprehensive tests.

为 MLU 后端新增可配置的 Indexer Cache INT8 量化能力,通过 --indexer_cache_dtype 参数启用。相关修改涵盖缓存分配与容量估算、量化缩放因子、分布式/RDMA 传输、Worker 配置传递、文档及完整测试。GLM5.1 indexer int8在pd分离下支持,decode阶段在32k上下文以上有收益,prefill阶段在4k上下文以上存在收益。

Change Type

  • Bug fix
  • New feature
  • Performance improvement
  • Refactor
  • Documentation
  • Test
  • Build or CI

Pull Request Checklist

Thank you for contributing to xLLM. Before requesting review, please make sure the following items are complete.

PR Title and Commit Messages

  • The PR title and each commit message follow the xLLM commit format: <type>: <subject>.

Allowed types: feat, bugfix, docs, test, refactor, chore, style, revert, perf, model, build, release.
The subject should use clear English, start with a verb, include at least 4 words, and end with ..

Pre-commit Checks

  • I have installed pre-commit by running pip install pre-commit or an equivalent command.
  • I have installed the hooks with pre-commit install.
  • I have run pre-commit run --all-files and fixed any reported issues.

If you are unsure how to set up pre-commit, see the pre-commit documentation.

Self Review

  • I have self-reviewed the code according to .agents/skills/code-review/references/custom-code-style.md, especially code written or assisted by AI.
  • I have rebased this PR onto the latest main branch.

Build and Test Coverage

  • Tests have been added or updated as needed.
  • CUDA: python setup.py build test has passed on a CUDA machine.
  • NPU: python setup.py build test has passed on an NPU machine.
  • MLU: python setup.py build test has passed on an MLU machine.

Reviewer Notes

DongheJin
DongheJin previously approved these changes Jul 14, 2026
XuZhang99
XuZhang99 previously approved these changes Jul 14, 2026
@phantomlei3 phantomlei3 dismissed stale reviews from XuZhang99 and DongheJin via d6d3197 July 14, 2026 11:38
@phantomlei3 phantomlei3 force-pushed the feat/indexer-cache-dtype branch from d6d3197 to efe2c6a Compare July 14, 2026 11:41
XuZhang99
XuZhang99 previously approved these changes Jul 14, 2026
@phantomlei3 phantomlei3 merged commit 6489f3a into xLLM-AI:main Jul 15, 2026
15 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants