feat: add embedded-python model executor for qwen3 on cuda. by zhang-minchao · Pull Request #1891 · xLLM-AI/xllm

zhang-minchao · 2026-07-06T08:12:25Z

Description

Add an embedded-CPython model executor that runs Qwen3 inference on CUDA through the xLLM runtime, as an opt-in alternative to the native C++ model path. The C++ engine drives an in-process Python interpreter to execute the model forward pass, while all compute stays on the existing xllm_ops fused CUDA kernels, so the Python path is numerically validated against the native path (byte-identical greedy decode at both tp=1 and tp=2).

Enable it with --model_impl=python --python_model_path=<path-to-xllm>; the default native path is unchanged.

Main pieces:

C++ ↔ Python bridge — PyCausalLM + py_model_helper (pybind) let the C++ executor construct and call a Python model like any native CausalLM, sharing the KV cache, forward inputs, and attention metadata.
Python model package (xllm/python/) — Qwen3 model, layers (linear / layernorm / embedding / rotary), op dispatch, triton kernels, and a cudagraph-capable model_runner, all built on xllm_ops.
PROPERTY struct reflection (property_reflect.h + macros.h) — dependency-free field reflection behind an opt-in REFLECT_PROPERTIES, so the full ModelArgs / ParallelArgs cross the pybind boundary without hand-maintained field lists.
xllm_ops op registration — ops layer structured as compute/collectives modules; TP collectives registered as torch.library.custom_op for Dynamo capture; attention uses flashinfer Python API directly.
Graph-mode decode — decode uses manual full CUDA graph (bucketed by batch), with FlashInfer host planner metadata and device runtime metadata using the same empty-sequence representation for padded sequences; prefill runs eager for minimal TTFT.

服务启动

# Python 模型执行器 (eager)
./xllm --model /path/to/Qwen3-0.6B \
  --model_impl=python \
  --python_model_path=/path/to/xllm/xllm \
  --device_id=0 \
  --enable_graph=false

# Python 模型执行器 (decode full CUDA graph)
./xllm --model /path/to/Qwen3-0.6B \
  --model_impl=python \
  --python_model_path=/path/to/xllm/xllm \
  --device_id=0 \
  --enable_graph=false \
  --python_graph_backend=cudagraphs

Performance

Full regression on Qwen3-0.6B using two dedicated H100 GPUs (cuda:6 and
cuda:7), input length 1024, output length 256, and
CUDA_LAUNCH_BLOCKING=0. The run completed build, JSON-only Python cudagraph
startup, byte-identical C++/Python greedy parity at TP1 and TP2, and the same
cross-batch sweep for all four variants. The batch list deliberately mixes
bucket boundaries and odd values to exercise decode-graph padding.

Report generated on 2026-07-14:
log/regression/batch_sweep_20260714_175345.md.

TP=1 complete comparison (Median TTFT / TPOT, ms)

batch	bucket	pad?	cpp_eager TTFT	cpp_eager TPOT	cpp_graph TTFT	cpp_graph TPOT	py_eager TTFT	py_eager TPOT	py_graph TTFT	py_graph TPOT	py_graph vs cpp_graph TTFT	py_graph vs cpp_graph TPOT
1	1		7.302	2.902	6.205	1.955	8.566	5.249	7.499	1.928	+20.9%	-1.4%
2	2		8.721	2.954	8.360	1.892	9.573	5.314	8.531	1.882	+2.0%	-0.5%
3	4	y	11.770	2.958	11.637	2.125	11.734	5.386	12.135	2.094	+4.3%	-1.5%
5	8	y	14.027	3.013	16.983	2.029	13.989	5.458	15.157	2.042	-10.8%	+0.6%
7	8	y	21.141	3.079	17.556	2.163	14.936	5.515	14.670	2.207	-16.4%	+2.0%
8	8		24.314	3.165	25.143	2.155	25.114	5.451	25.197	2.146	+0.2%	-0.4%
13	16	y	34.719	3.360	23.631	2.538	34.988	5.762	30.559	2.489	+29.3%	-1.9%
16	16		28.872	3.297	31.570	2.810	33.539	5.755	31.001	2.565	-1.8%	-8.7%
31	32	y	45.784	3.709	45.402	3.482	44.744	6.060	45.499	3.525	+0.2%	+1.2%
47	48	y	70.691	4.528	68.503	4.214	76.332	6.443	63.107	4.283	-7.9%	+1.6%
100	112	y	124.055	7.470	124.159	7.224	127.821	7.570	131.613	7.246	+6.0%	+0.3%

TP=2 complete comparison (Median TTFT / TPOT, ms)

batch	bucket	pad?	cpp_eager TTFT	cpp_eager TPOT	cpp_graph TTFT	cpp_graph TPOT	py_eager TTFT	py_eager TPOT	py_graph TTFT	py_graph TPOT	py_graph vs cpp_graph TTFT	py_graph vs cpp_graph TPOT
1	1		8.953	4.713	9.301	4.518	15.072	11.429	15.204	4.536	+63.5%	+0.4%
2	2		10.810	4.775	11.266	4.531	14.350	11.417	15.313	4.467	+35.9%	-1.4%
3	4	y	12.701	4.802	13.210	4.306	15.089	11.421	15.709	4.432	+18.9%	+2.9%
5	8	y	20.564	4.861	15.261	4.404	15.801	11.471	16.405	4.631	+7.5%	+5.2%
7	8	y	18.141	4.897	15.632	4.485	18.196	11.454	22.172	4.724	+41.8%	+5.3%
8	8		20.633	4.935	26.476	4.437	29.357	11.485	29.045	4.737	+9.7%	+6.8%
13	16	y	28.271	5.086	37.933	5.067	35.116	11.334	49.283	5.170	+29.9%	+2.0%
16	16		32.237	5.063	34.987	5.074	34.056	11.691	35.159	5.256	+0.5%	+3.6%
31	32	y	48.867	5.464	60.212	5.564	51.107	12.052	68.112	5.889	+13.1%	+5.8%
47	48	y	63.373	5.813	75.758	6.308	70.360	12.514	71.868	6.604	-5.1%	+4.7%
100	112	y	128.817	7.661	132.780	9.150	137.056	13.685	140.100	9.421	+5.5%	+3.0%

Takeaways:

Python greedy parity is byte-identical to C++ at both TP1 and TP2.
py_graph TPOT tracks cpp_graph TPOT closely: TP1 ranges from -8.7% to
+2.0%, and TP2 from -1.4% to +6.8% across all tested batches.
py_graph substantially reduces Python decode overhead: its TPOT is much
lower than py_eager at both TP degrees because decode replays a captured CUDA
graph instead of dispatching the model from CPython every step.
Prefill remains eager: TTFT differences are workload and scheduling
sensitive and are not used as the decode-performance acceptance criterion.
Padding rows (pad?=y) validate empty-sequence padding: host planner and
device runtime metadata use repeated terminal indptr values, zero KV-length
deltas, and no dummy page allocation. All padded buckets passed at TP1 and
TP2.

Related Issues

Change Type

gemini-code-assist

Code Review

This pull request introduces a Python model executor framework that allows executing Python-defined model graphs, such as Qwen3, within xLLM's C++ worker using an embedded CPython interpreter. It adds the C++ bridge, a lightweight config reflection system, and the Python-side model, layers, op dispatching, and graph runner. Feedback on the changes focuses on strict adherence to the repository's style guide, including using project-root-relative paths for #include directives, preferring fixed-width integers and emplace_back in C++, using CHECK instead of TORCH_CHECK, ensuring complete Python type annotations, and routing Python diagnostic logs through the shared logger instead of print().

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

XuZhang99 · 2026-07-10T10:34:30Z

    fi

 # Install tvm-ffi
 RUN set -eux; \


remove tvm-ffi installation manually.

XuZhang99 · 2026-07-10T10:34:47Z

 # Install flashinfer + AOT compile
 ENV FLASHINFER_VERSION=${FLASHINFER_VERSION}
 ENV FLASHINFER_CUDA_ARCH_LIST=${FLASHINFER_CUDA_ARCH_LIST}
 RUN set -eux; \


- cpp_framework_python_model_architecture.md: document the architecture decision to add python model execution on the c++ serving framework and why python owns both the model and its ModelExecutor. - refactor_python_model_impl.md: describe the implementable refactor plan covering cross-language interface, object lifecycle, migration order and acceptance conditions, baselined on PR xLLM-AI#1891.

…ucture.

…s bridge - Delete xllm_attention_ops.cpp/h (batch_decode/prefill/update_plan torch ops were only used by the Python executor, now replaced by flashinfer Python API) - Move reshape_paged_cache registration into xllm_ops_library.cpp - layers/attention.py: use flashinfer BatchDecodeWithPagedKVCacheWrapper / BatchPrefillWithRaggedKVCacheWrapper directly instead of torch.ops.xllm_ops - Rename py_model_bridge → py_model_helper, consolidate PyDictVisitor and dtype_to_string utilities - Move py_causal_lm to models/llm/ (where other CausalLM implementations live) - Add core/kernels/xllm_torch_ops.h as device-agnostic dispatch header - Remove USE_CUDA guard from PyCausalLM compilation (device-agnostic) - Add FlashinferWorkspace skip for model_impl=python in llm_worker_impl.cpp

The backend == "fa3" change was unnecessary — main's hardware-based branching is correct for the ragged prefill path.

The uri-based backend detection was unnecessary — main's determine_attention_backend() call is correct.

Remove outdated references to deleted xllm_attention_ops.cpp, redundant argumentation, and premature NPU landing checklist. Keep only the current architecture table, minimal extension guide, and risk notes.

Add complete description of the three interaction phases (init config dict, weight loading, per-step forward args) and remove speculative PrivateUse1 references.

The variable holds attention metadata, not generic metadata. Rename for clarity and update the design doc to match.

Add model_impl field to ModelContext so model_registry.cpp no longer reads the global FLAGS_model_impl directly. The caller (llm_worker_impl) sets it before calling create_llm_model.

- initialize PyTorch NCCL TP collectives and honor JSON executor settings. - prevent KV padding overflow and remove obsolete graph batch configuration. - add config, dump, and embedded Python regression coverage.

Update the Python executor design document to match the current CUDA, TP, JSON, and decode graph implementation.

Install FlashInfer 0.6.14 after the AOT build and make the embedded Python runtime share the xLLM TVM-FFI libraries.

- cpp_framework_python_model_architecture.md: document the architecture decision to add python model execution on the c++ serving framework and why python owns both the model and its ModelExecutor. - refactor_python_model_impl.md: describe the implementable refactor plan covering cross-language interface, object lifecycle, migration order and acceptance conditions, baselined on PR xLLM-AI#1891.

…d of hardware probe. - batch_prefill: determine FA2/FA3 call schema from the URI suffix (_sm90) instead of re-evaluating hardware and mask at runtime. - flashinfer_planinfo: select short (16-param) vs long (19-param) plan args from the explicit backend string, removing the platform.h dependency.

…h runner. - Remove max_seqs_per_batch from config dict; pass it as an explicit constructor argument through PyExecutorImpl → ModelExecutor → runner. - DecodeCudaGraphRunner now takes max_model_len and pre-allocates a single shared paged_kv_indices buffer sized for the worst case, reused across all bucket entries.

…da graph. - Add DecodeGraphSharesMaximumBlockTableBufferAcrossBuckets to verify the single shared paged_kv_indices buffer is reused across buckets. - Add ModelExecutorUsesExplicitRuntimeBatchLimit to confirm the runtime batch limit flows through to the decode runner. - Update DecodeMetadataFastPath* tests for max_seqs_per_batch=4 and assert the full padded shape including kv_cu_seq_lens and qo_indptr. - Add MakeFa2PrefillParams helper to pin FA2 for piecewise prefill eager references, isolating graph capture correctness from FA2/FA3 numerical differences.

Clarify at both call sites (batch_prefill.cpp ragged_run and flashinfer_planinfo.cpp prefill plan) why the FlashInfer ABI is chosen from the loaded module/backend (fa2 19-arg vs fa3 16-arg) rather than the SM90 hardware probe, which silently broke fa2-forced-on-SM90 graph piecewise prefill. Comment only; no logic change.

…model executor. - flashinfer: map sliding window W to window_left W-1 to match the native attention convention, so SWA models attend the correct span. - linear: add optional bias to Column/RowParallelLinear (column bias sharded on the output dim; row bias replicated and added once after all-reduce) and load qkv/o_proj bias when attention_bias is set; standard Qwen3 (attention_bias=False) is unchanged. - qwen3: cast position ids to int64 once in Qwen3Model.forward instead of once per layer inside fused_qk_norm_rope; the single cast stays inside the captured decode graph so replay re-casts static_positions correctly.

zhang-minchao requested review from Clement-Wang26, DongheJin, DragonFive, JimHsiung, Kang-Meng, RobbieLeung, XuZhang99, liujinguang0125, liutongxuan, walsonyang, xiao-yu-chen, yingxudeng and yq33victor as code owners July 6, 2026 08:12

gemini-code-assist Bot reviewed Jul 6, 2026

View reviewed changes

zhang-minchao force-pushed the feat/python-model-executor branch from 7430c11 to 479ab8a Compare July 6, 2026 08:16

zhang-minchao changed the title ~~Feat/python model executor~~ feat: add embedded-python model executor for qwen3 on cuda. Jul 6, 2026

zhang-minchao force-pushed the feat/python-model-executor branch 2 times, most recently from c962bc8 to eb22c0b Compare July 6, 2026 11:25

zhang-minchao marked this pull request as draft July 6, 2026 12:22

zhang-minchao marked this pull request as ready for review July 7, 2026 19:18

zhang-minchao force-pushed the feat/python-model-executor branch from be4d1a1 to 357910e Compare July 7, 2026 19:19

XuZhang99 reviewed Jul 8, 2026

View reviewed changes

Comment thread xllm/core/framework/config/model_config.h

Comment thread xllm/core/layers/cuda/flashinfer_planinfo.cpp Outdated

Comment thread xllm/models/model_registry.cpp Outdated

XuZhang99 mentioned this pull request Jul 10, 2026

[Question]: models/llm/qwen3_5.h 必须编译 NPU 头文件，而不是 common 的 #1916

Open

XuZhang99 reviewed Jul 10, 2026

View reviewed changes

Comment thread xllm/core/kernels/cuda/cuda_ops_library.cpp

zhang-minchao force-pushed the feat/python-model-executor branch from 99a0f54 to 6f8e5e6 Compare July 10, 2026 07:23

XuZhang99 reviewed Jul 10, 2026

View reviewed changes

zhang-minchao force-pushed the feat/python-model-executor branch 2 times, most recently from 06baa0a to 584611d Compare July 13, 2026 12:58

zhang-minchao added 29 commits July 14, 2026 18:34

docs: update multi-hardware ops design doc to reflect ops layer restr…

c83c89c

…ucture.

revert: restore Platform::is_support_sm90a() check in prefill plan

c462e14

The backend == "fa3" change was unnecessary — main's hardware-based branching is correct for the ragged prefill path.

revert: restore determine_attention_backend() in batch_prefill

1268520

The uri-based backend detection was unnecessary — main's determine_attention_backend() call is correct.

docs: simplify multi-hardware ops design doc

4e60716

Remove outdated references to deleted xllm_attention_ops.cpp, redundant argumentation, and premature NPU landing checklist. Keep only the current architecture table, minimal extension guide, and risk notes.

docs: merge reshape_paged_cache into compute ops row

6ab9ce1

docs: remove speculative NPU landing section

1296cd0

docs: rewrite with C++/Python interaction details

4cbf492

Add complete description of the three interaction phases (init config dict, weight loading, per-step forward args) and remove speculative PrivateUse1 references.

refactor: rename meta -> attn_meta in PyCausalLM::forward

4906483

The variable holds attention metadata, not generic metadata. Rename for clarity and update the design doc to match.

refactor: use full name attention_metadata_dict, no abbreviation

e86582b

refactor: pass model_impl via ModelContext instead of FLAGS

2e71a16

Add model_impl field to ModelContext so model_registry.cpp no longer reads the global FLAGS_model_impl directly. The caller (llm_worker_impl) sets it before calling create_llm_model.

bugfix: fix python executor tp and graph configuration.

e58ad90

- initialize PyTorch NCCL TP collectives and honor JSON executor settings. - prevent KV padding overflow and remove obsolete graph batch configuration. - add config, dump, and embedded Python regression coverage.

refactor: rename cuda torch ops library files.

4ad1a40

Update the Python executor design document to match the current CUDA, TP, JSON, and decode graph implementation.

build: install flashinfer python runtime.

5263b48

Install FlashInfer 0.6.14 after the AOT build and make the embedded Python runtime share the xLLM TVM-FFI libraries.

docs: remove python executor design document.

e1d1a9b

refactor: add dedicated python model executor.

25699d9

test: cover decode graph metadata updates.

0243904

docs: simplify python executor refactor design.

c156834

refactor: keep python executor changes scoped to python graph path

fb25933

bugfix: release python config while holding gil.

c84c7f2

refactor: narrow cuda graph metadata scope

2c1b8f2

refactor: use empty sequences for graph padding

f258610

docs: explain empty sequence graph padding

2d0ced8

zhang-minchao force-pushed the feat/python-model-executor branch from 7fe566f to 2d0ced8 Compare July 14, 2026 10:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add embedded-python model executor for qwen3 on cuda.#1891

feat: add embedded-python model executor for qwen3 on cuda.#1891
zhang-minchao wants to merge 47 commits into
xLLM-AI:mainfrom
zhang-minchao:feat/python-model-executor

zhang-minchao commented Jul 6, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

XuZhang99 Jul 10, 2026

Uh oh!

XuZhang99 Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

zhang-minchao commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

服务启动

Performance

TP=1 complete comparison (Median TTFT / TPOT, ms)

TP=2 complete comparison (Median TTFT / TPOT, ms)

Related Issues

Change Type

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

XuZhang99 Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

XuZhang99 Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhang-minchao commented Jul 6, 2026 •

edited

Loading