From f41dc348bd54df1f355b15350e063f8dd8a2a77b Mon Sep 17 00:00:00 2001 From: zengchao0424 Date: Mon, 4 Aug 2025 16:06:53 +0800 Subject: [PATCH 1/4] update hunyuan Eagle3 benchmark --- README.md | 27 +++++++++++++++++ README_en.md | 29 +++++++++++++++++++ .../features/speculative_decoding/eagle.md | 13 +++++++++ .../speculative_decoding/benchmarks.md | 29 +++++++++++++++++++ 4 files changed, 98 insertions(+) diff --git a/README.md b/README.md index c6d8d044..3690309d 100644 --- a/README.md +++ b/README.md @@ -279,6 +279,7 @@ Qwen3系列模型的`BF16`、`FP8-Static`、`FP8-Dynamic`、`INT8-Dynamic`、`IN ### （2）投机采样 +#### Qwen3 Series Models Qwen3系列的Eagle3模型在MT-bench/HunmanEval/GSM8K/Alpaca上的加速结果如下： @@ -312,6 +313,32 @@ Qwen3系列的Eagle3模型在MT-bench/HunmanEval/GSM8K/Alpaca上的加速结果
+Hunyuan系列的Eagle3模型在MT-bench/HunmanEval/GSM8K/Alpaca上的加速结果如下： + + + + + + + + + + + + + + + + + + + + + + + + +
  MT-benchHumanEvalGSM8KAlpacaMean
TemperatureModelSpeedupτSpeedupτSpeedupτSpeedupτSpeedupτ
T=0Hunyuan-1.8B-Instruct1.97x2.902.58x3.732.61x3.711.71x2.432.22x3.19
Hunyuan-4B-Instruct1.77x2.602.64x3.352.14x3.171.72x2.572.07x2.92
Hunyuan-7B-Instruct2.22x3.583.59x5.472.96x4.681.64x2.562.60x4.07
T=1Hunyuan-1.8B-Instruct1.58x2.362.35x3.562.23x3.381.26x1.871.86x2.79
Hunyuan-4B-Instruct1.36x2.051.97x2.861.72x2.681.14x1.761.55x2.34
Hunyuan-7B-Instruct1.90x3.113.12x5.092.74x4.341.47x2.392.31x3.73
## 📝许可协议 本项目的代码依照 [License for AngelSlim](LICENSE) 协议开源。 diff --git a/README_en.md b/README_en.md index 471ece14..09fb2168 100644 --- a/README_en.md +++ b/README_en.md @@ -282,6 +282,8 @@ Benchmark results for other models with `FP8-Static`, `FP8-Dynamic`, `INT4-GPTQ` ### (2) Speculative Decoding + +#### Qwen3 Series Models Benchmark results for Qwen3 series models with `Eagle3` speculative decoding algorithm on datasets including `MT-bench`, `HunmanEval`, `GSM8K`, and `Alpaca`: @@ -315,6 +317,33 @@ Benchmark results for Qwen3 series models with `Eagle3` speculative decoding alg
+#### Hunyuan Series Models +Benchmark results for Hunyuan series models with `Eagle3` speculative decoding algorithm on datasets including `MT-bench`, `HunmanEval`, `GSM8K`, and `Alpaca`: + + + + + + + + + + + + + + + + + + + + + + + + +
  MT-benchHumanEvalGSM8KAlpacaMean
TemperatureModelSpeedupτSpeedupτSpeedupτSpeedupτSpeedupτ
T=0Hunyuan-1.8B-Instruct1.97x2.902.58x3.732.61x3.711.71x2.432.22x3.19
Hunyuan-4B-Instruct1.77x2.602.64x3.352.14x3.171.72x2.572.07x2.92
Hunyuan-7B-Instruct2.22x3.583.59x5.472.96x4.681.64x2.562.60x4.07
T=1Hunyuan-1.8B-Instruct1.58x2.362.35x3.562.23x3.381.26x1.871.86x2.79
Hunyuan-4B-Instruct1.36x2.051.97x2.861.72x2.681.14x1.761.55x2.34
Hunyuan-7B-Instruct1.90x3.113.12x5.092.74x4.341.47x2.392.31x3.73
## 📝 License diff --git a/docs/source/features/speculative_decoding/eagle.md b/docs/source/features/speculative_decoding/eagle.md index 873b7adf..4b4e5907 100644 --- a/docs/source/features/speculative_decoding/eagle.md +++ b/docs/source/features/speculative_decoding/eagle.md @@ -28,5 +28,18 @@ - `TARGET_MODEL_PATH_OR_NAME`为本地路径或模型在huggingface上的名字; - `EAGLE3_MODEL_PATH`为Eagle3模型路径或在huggingface上的名字; + +目前vllm已经支持Hunyuan-1.8B-Instruct/4B-Instruct/7B-Instruct模型的eagle3部署，你可以选择使用vllm作为推理后端快速验证Eagle3模型的加速效果。 +在已经安装正确的[vllm commit](https://github.com/vllm-project/vllm/pull/22080) 的环境中使用以下命令可以快速启动一个兼容Openai的服务，然后即可以通过本地端口进行请求了。 +- 启动兼容OpenAI格式的API服务 + + ```shell + python3 -m vllm.entrypoints.openai.api_server --tensor-parallel-size 1 \ + --port 8000 \ + --speculative_config '{"model": "AngelSlim/Hunyuan-1.8B-Instruct_eagle3", "method" : "eagle3", "draft_tensor_parallel_size" : 1, "num_speculative_tokens": 2}' --trust-remote-code \ + --model tencent/Hunyuan-1.8B-Instruct + ``` +但是由于vllm最新版本Eagle3并不支持tree attention, 因此推理验证时为chain-base推理模式。 + ## 训练及创新 Comming soon. \ No newline at end of file diff --git a/docs/source/performance/speculative_decoding/benchmarks.md b/docs/source/performance/speculative_decoding/benchmarks.md index ad33de97..63cd82c1 100644 --- a/docs/source/performance/speculative_decoding/benchmarks.md +++ b/docs/source/performance/speculative_decoding/benchmarks.md @@ -2,6 +2,8 @@ ## Eagle3 +### Qwen3 Series Models + | | | MT-bench | | HumanEval | | GSM8K | | Alpaca | | Mean | | |------------------|--------------|------------------|------------|-------------------|-------------|----------------|---------|----------------|----------|---------------|--------| | | Model | Speedup | τ | Speedup | τ | Speedup | τ | Speedup | τ | Speedup | τ | @@ -19,3 +21,30 @@ | | Qwen3-32B | 1.62x | 1.91 | 1.71x | 2.05 | 1.78x | 2.10 | 1.80x | 1.95 | 1.62x | 2.00 | | | Qwen3-30B-A3B| 1.91x | 2.46 | 2.00x | 2.64 | 1.90x | 2.53 | 1.80x | 2.32 | 1.90x | 2.48 | +### Hunyuan Series Models + + + + + + + + + + + + + + + + + + + + + + + + +
  MT-benchHumanEvalGSM8KAlpacaMean
TemperatureModelSpeedupτSpeedupτSpeedupτSpeedupτSpeedupτ
Temperature=0Hunyuan-1.8B-Instruct1.97x2.902.58x3.732.61x3.711.71x2.432.22x3.19
Hunyuan-4B-Instruct1.77x2.602.64x3.352.14x3.171.72x2.572.07x2.92
Hunyuan-7B-Instruct2.22x3.583.59x5.472.96x4.681.64x2.562.60x4.07
Temperature=1Hunyuan-1.8B-Instruct1.58x2.362.35x3.562.23x3.381.26x1.871.86x2.79
Hunyuan-4B-Instruct1.36x2.051.97x2.861.72x2.681.14x1.761.55x2.34
Hunyuan-7B-Instruct1.90x3.113.12x5.092.74x4.341.47x2.392.31x3.73
+ \ No newline at end of file From fade41e9cf6ee89a747b6058e1503ea078099b30 Mon Sep 17 00:00:00 2001 From: zengchao0424 Date: Mon, 4 Aug 2025 18:14:49 +0800 Subject: [PATCH 2/4] update hunyuan Eagle3 benchmark --- README.md | 11 ++++++++++- docs/source/features/speculative_decoding/eagle.md | 3 +++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 3690309d..a7cf449c 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ Coming soon： | [QwQ](https://huggingface.co/collections/AngelSlim/qwen3-quant-68652e26da31740739d154f8) | ✅ | ✅ | ✅ | ✅ | ✅ | ### 投机采样 -目前已开源Qwen3系列模型的Eagle3权重。 +目前已开源Qwen3和Hunyuan系列模型的Eagle3权重。 | 模型名 | Eagle3 | | ----------| ----------------- | @@ -73,6 +73,15 @@ Coming soon： | [Qwen3-32B](https://huggingface.co/AngelSlim/Qwen3-32B_eagle3) | ✅ | | [Qwen3-30B-A3B](https://huggingface.co/AngelSlim/Qwen3-a3B_eagle3) | ✅ | +| 模型名 | Eagle3 | +| ----------| ----------------- | +| [Hunyuan-1.8B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-1.8B-Instruct_eagle3) | ✅ | +| [Hunyuan-4B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-4B-Instruct_eagle3) | ✅ | +| [Hunyuan-7B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-7B-Instruct_eagle3) | ✅ | + + + + ## 🛎️如何使用 ### 安装 AngelSlim diff --git a/docs/source/features/speculative_decoding/eagle.md b/docs/source/features/speculative_decoding/eagle.md index 4b4e5907..272165c3 100644 --- a/docs/source/features/speculative_decoding/eagle.md +++ b/docs/source/features/speculative_decoding/eagle.md @@ -7,6 +7,8 @@ 其中全部数据都是在单张H20上使用pytorch推理获得。 ## 快速测试 + +### SGLang 目前sglang已经支持Qwen3-8B/14B/30B-A3B模型的eagle3部署，你可以选择使用sglang作为推理后端快速验证Eagle3模型的加速效果。 在已经安装sglang的环境中使用以下命令可以快速启动一个兼容Openai的服务，然后即可以通过本地端口进行请求了。 - 启动兼容OpenAI格式的API服务 @@ -29,6 +31,7 @@ - `EAGLE3_MODEL_PATH`为Eagle3模型路径或在huggingface上的名字; +### vLLM 目前vllm已经支持Hunyuan-1.8B-Instruct/4B-Instruct/7B-Instruct模型的eagle3部署，你可以选择使用vllm作为推理后端快速验证Eagle3模型的加速效果。 在已经安装正确的[vllm commit](https://github.com/vllm-project/vllm/pull/22080) 的环境中使用以下命令可以快速启动一个兼容Openai的服务，然后即可以通过本地端口进行请求了。 - 启动兼容OpenAI格式的API服务 From 67741602b9ab9482c38592e16f70b10a874cce4d Mon Sep 17 00:00:00 2001 From: zengchao0424 Date: Mon, 4 Aug 2025 20:11:17 +0800 Subject: [PATCH 3/4] update hunyuan Eagle3 support model list --- README.md | 17 +++++++---------- README_en.md | 12 +++++++----- 2 files changed, 14 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index a7cf449c..d62c3b85 100644 --- a/README.md +++ b/README.md @@ -62,22 +62,19 @@ Coming soon： | [QwQ](https://huggingface.co/collections/AngelSlim/qwen3-quant-68652e26da31740739d154f8) | ✅ | ✅ | ✅ | ✅ | ✅ | ### 投机采样 + +#### Eagle3 目前已开源Qwen3和Hunyuan系列模型的Eagle3权重。 -| 模型名 | Eagle3 | -| ----------| ----------------- | -| [Qwen3-1.7B](https://huggingface.co/AngelSlim/Qwen3-1.7B_eagle3) | ✅ | -| [Qwen3-4B](https://huggingface.co/AngelSlim/Qwen3-4B_eagle3) | ✅ | -| [Qwen3-8B](https://huggingface.co/AngelSlim/Qwen3-8B_eagle3) | ✅ | +| Qwen3 | | Hunyuan | | +| ----------| -------- |----------|-------- | +| [Qwen3-1.7B](https://huggingface.co/AngelSlim/Qwen3-1.7B_eagle3) | ✅ |[Hunyuan-1.8B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-1.8B-Instruct_eagle3) | ✅ | +| [Qwen3-4B](https://huggingface.co/AngelSlim/Qwen3-4B_eagle3) | ✅ |[Hunyuan-4B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-4B-Instruct_eagle3) | ✅ | +| [Qwen3-8B](https://huggingface.co/AngelSlim/Qwen3-8B_eagle3) | ✅ |[Hunyuan-7B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-7B-Instruct_eagle3) | ✅ | | [Qwen3-14B](https://huggingface.co/AngelSlim/Qwen3-14B_eagle3) | ✅ | | [Qwen3-32B](https://huggingface.co/AngelSlim/Qwen3-32B_eagle3) | ✅ | | [Qwen3-30B-A3B](https://huggingface.co/AngelSlim/Qwen3-a3B_eagle3) | ✅ | -| 模型名 | Eagle3 | -| ----------| ----------------- | -| [Hunyuan-1.8B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-1.8B-Instruct_eagle3) | ✅ | -| [Hunyuan-4B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-4B-Instruct_eagle3) | ✅ | -| [Hunyuan-7B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-7B-Instruct_eagle3) | ✅ | diff --git a/README_en.md b/README_en.md index 09fb2168..cb8c20a6 100644 --- a/README_en.md +++ b/README_en.md @@ -62,13 +62,15 @@ Currently supports the following LLMs, including Hunyuan-Dense, Hunyuan-MoE, Qwe | [QwQ](https://huggingface.co/collections/AngelSlim/qwen3-quant-68652e26da31740739d154f8) | ✅ | ✅ | ✅ | ✅ | ✅ | ### Speculative Decoding + +#### Eagle3 The Eagle3 weights for the Qwen3 series model are now available. -| Model | Eagle3 | -| ----------| ----------------- | -| [Qwen3-1.7B](https://huggingface.co/AngelSlim/Qwen3-1.7B_eagle3) | ✅ | -| [Qwen3-4B](https://huggingface.co/AngelSlim/Qwen3-4B_eagle3) | ✅ | -| [Qwen3-8B](https://huggingface.co/AngelSlim/Qwen3-8B_eagle3) | ✅ | +| Qwen3 | | Hunyuan | | +| ----------| -------- |----------|-------- | +| [Qwen3-1.7B](https://huggingface.co/AngelSlim/Qwen3-1.7B_eagle3) | ✅ |[Hunyuan-1.8B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-1.8B-Instruct_eagle3) | ✅ | +| [Qwen3-4B](https://huggingface.co/AngelSlim/Qwen3-4B_eagle3) | ✅ |[Hunyuan-4B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-4B-Instruct_eagle3) | ✅ | +| [Qwen3-8B](https://huggingface.co/AngelSlim/Qwen3-8B_eagle3) | ✅ |[Hunyuan-7B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-7B-Instruct_eagle3) | ✅ | | [Qwen3-14B](https://huggingface.co/AngelSlim/Qwen3-14B_eagle3) | ✅ | | [Qwen3-32B](https://huggingface.co/AngelSlim/Qwen3-32B_eagle3) | ✅ | | [Qwen3-30B-A3B](https://huggingface.co/AngelSlim/Qwen3-a3B_eagle3) | ✅ | From 03830bd800fbcd1d119bf3eef6b8f9849cff287e Mon Sep 17 00:00:00 2001 From: zengchao0424 Date: Mon, 4 Aug 2025 20:20:46 +0800 Subject: [PATCH 4/4] update hunyuan Eagle3 support model list --- README.md | 16 ++++++++-------- README_en.md | 16 ++++++++-------- 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index d62c3b85..b74d5a82 100644 --- a/README.md +++ b/README.md @@ -66,14 +66,14 @@ Coming soon： #### Eagle3 目前已开源Qwen3和Hunyuan系列模型的Eagle3权重。 -| Qwen3 | | Hunyuan | | -| ----------| -------- |----------|-------- | -| [Qwen3-1.7B](https://huggingface.co/AngelSlim/Qwen3-1.7B_eagle3) | ✅ |[Hunyuan-1.8B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-1.8B-Instruct_eagle3) | ✅ | -| [Qwen3-4B](https://huggingface.co/AngelSlim/Qwen3-4B_eagle3) | ✅ |[Hunyuan-4B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-4B-Instruct_eagle3) | ✅ | -| [Qwen3-8B](https://huggingface.co/AngelSlim/Qwen3-8B_eagle3) | ✅ |[Hunyuan-7B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-7B-Instruct_eagle3) | ✅ | -| [Qwen3-14B](https://huggingface.co/AngelSlim/Qwen3-14B_eagle3) | ✅ | -| [Qwen3-32B](https://huggingface.co/AngelSlim/Qwen3-32B_eagle3) | ✅ | -| [Qwen3-30B-A3B](https://huggingface.co/AngelSlim/Qwen3-a3B_eagle3) | ✅ | +| Qwen3 Models | Hunyuan Models | +| ----------|----------| +| ✅ [Qwen3-1.7B](https://huggingface.co/AngelSlim/Qwen3-1.7B_eagle3) |✅ [Hunyuan-1.8B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-1.8B-Instruct_eagle3) | +| ✅ [Qwen3-4B](https://huggingface.co/AngelSlim/Qwen3-4B_eagle3) |✅ [Hunyuan-4B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-4B-Instruct_eagle3) | +| ✅ [Qwen3-8B](https://huggingface.co/AngelSlim/Qwen3-8B_eagle3) |✅ [Hunyuan-7B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-7B-Instruct_eagle3) | +| ✅ [Qwen3-14B](https://huggingface.co/AngelSlim/Qwen3-14B_eagle3) | +| ✅ [Qwen3-32B](https://huggingface.co/AngelSlim/Qwen3-32B_eagle3) | +| ✅ [Qwen3-30B-A3B](https://huggingface.co/AngelSlim/Qwen3-a3B_eagle3) | diff --git a/README_en.md b/README_en.md index cb8c20a6..6684abfa 100644 --- a/README_en.md +++ b/README_en.md @@ -66,14 +66,14 @@ Currently supports the following LLMs, including Hunyuan-Dense, Hunyuan-MoE, Qwe #### Eagle3 The Eagle3 weights for the Qwen3 series model are now available. -| Qwen3 | | Hunyuan | | -| ----------| -------- |----------|-------- | -| [Qwen3-1.7B](https://huggingface.co/AngelSlim/Qwen3-1.7B_eagle3) | ✅ |[Hunyuan-1.8B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-1.8B-Instruct_eagle3) | ✅ | -| [Qwen3-4B](https://huggingface.co/AngelSlim/Qwen3-4B_eagle3) | ✅ |[Hunyuan-4B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-4B-Instruct_eagle3) | ✅ | -| [Qwen3-8B](https://huggingface.co/AngelSlim/Qwen3-8B_eagle3) | ✅ |[Hunyuan-7B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-7B-Instruct_eagle3) | ✅ | -| [Qwen3-14B](https://huggingface.co/AngelSlim/Qwen3-14B_eagle3) | ✅ | -| [Qwen3-32B](https://huggingface.co/AngelSlim/Qwen3-32B_eagle3) | ✅ | -| [Qwen3-30B-A3B](https://huggingface.co/AngelSlim/Qwen3-a3B_eagle3) | ✅ | +| Qwen3 Models | Hunyuan Models | +| ----------|----------| +| ✅ [Qwen3-1.7B](https://huggingface.co/AngelSlim/Qwen3-1.7B_eagle3) |✅ [Hunyuan-1.8B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-1.8B-Instruct_eagle3) | +| ✅ [Qwen3-4B](https://huggingface.co/AngelSlim/Qwen3-4B_eagle3) |✅ [Hunyuan-4B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-4B-Instruct_eagle3) | +| ✅ [Qwen3-8B](https://huggingface.co/AngelSlim/Qwen3-8B_eagle3) |✅ [Hunyuan-7B-Instruct](https://huggingface.co/AngelSlim/Hunyuan-7B-Instruct_eagle3) | +| ✅ [Qwen3-14B](https://huggingface.co/AngelSlim/Qwen3-14B_eagle3) | +| ✅ [Qwen3-32B](https://huggingface.co/AngelSlim/Qwen3-32B_eagle3) | +| ✅ [Qwen3-30B-A3B](https://huggingface.co/AngelSlim/Qwen3-a3B_eagle3) | ## 🛎️How to Use