Skip to content

Commit 857b29a

Browse files
xenovaxenova
authored
Add support for DeepSeek-V4 (#1679)
* Add support for DeepSeek-V4 * formatting * fix
1 parent 35636df commit 857b29a

9 files changed

Lines changed: 124 additions & 15 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -271,6 +271,7 @@ To find compatible models on the Hub, select the "transformers.js" library tag i
271271
1. **[DeBERTa-v2](https://huggingface.co/docs/transformers/model_doc/deberta-v2)** (from Microsoft) released with the paper [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://huggingface.co/papers/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen.
272272
1. **[Decision Transformer](https://huggingface.co/docs/transformers/model_doc/decision_transformer)** (from Berkeley/Facebook/Google) released with the paper [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://huggingface.co/papers/2106.01345) by Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch.
273273
1. **[DeepSeek-V3](https://huggingface.co/docs/transformers/model_doc/deepseek_v3)** (from DeepSeek-AI) released with the paper [DeepSeek-V3 Technical Report](https://huggingface.co/papers/2412.19437) by the DeepSeek-AI team.
274+
1. **[DeepSeek-V4](https://huggingface.co/docs/transformers/model_doc/deepseek_v4)** (from DeepSeek-AI) released with the paper [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf) by the DeepSeek-AI team.
274275
1. **[DeiT](https://huggingface.co/docs/transformers/model_doc/deit)** (from Facebook) released with the paper [Training data-efficient image transformers & distillation through attention](https://huggingface.co/papers/2012.12877) by Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Hervé Jégou.
275276
1. **[Depth Anything](https://huggingface.co/docs/transformers/main/model_doc/depth_anything)** (from University of Hong Kong and TikTok) released with the paper [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://huggingface.co/papers/2401.10891) by Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao.
276277
1. **Depth Pro** (from Apple) released with the paper [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://huggingface.co/papers/2410.02073) by Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, Vladlen Koltun.

‎packages/transformers/docs/snippets/5_supported-models.snippet‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
1. **[DeBERTa-v2](https://huggingface.co/docs/transformers/model_doc/deberta-v2)** (from Microsoft) released with the paper [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://huggingface.co/papers/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen.
3333
1. **[Decision Transformer](https://huggingface.co/docs/transformers/model_doc/decision_transformer)** (from Berkeley/Facebook/Google) released with the paper [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://huggingface.co/papers/2106.01345) by Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch.
3434
1. **[DeepSeek-V3](https://huggingface.co/docs/transformers/model_doc/deepseek_v3)** (from DeepSeek-AI) released with the paper [DeepSeek-V3 Technical Report](https://huggingface.co/papers/2412.19437) by the DeepSeek-AI team.
35+
1. **[DeepSeek-V4](https://huggingface.co/docs/transformers/model_doc/deepseek_v4)** (from DeepSeek-AI) released with the paper [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf) by the DeepSeek-AI team.
3536
1. **[DeiT](https://huggingface.co/docs/transformers/model_doc/deit)** (from Facebook) released with the paper [Training data-efficient image transformers & distillation through attention](https://huggingface.co/papers/2012.12877) by Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Hervé Jégou.
3637
1. **[Depth Anything](https://huggingface.co/docs/transformers/main/model_doc/depth_anything)** (from University of Hong Kong and TikTok) released with the paper [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://huggingface.co/papers/2401.10891) by Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao.
3738
1. **Depth Pro** (from Apple) released with the paper [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://huggingface.co/papers/2410.02073) by Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, Vladlen Koltun.

‎packages/transformers/src/configs.js‎

Lines changed: 32 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -208,11 +208,12 @@ function getNormalizedConfig(config) {
208208
break;
209209
case 'youtu':
210210
case 'deepseek_v3':
211+
case 'deepseek_v4':
211212
case 'glm_moe_dsa':
212213
case 'mistral4':
213214
mapping['num_heads'] = 'num_key_value_heads';
214215
mapping['num_layers'] = 'num_hidden_layers';
215-
mapping['dim_kv'] = 'qk_head_dim';
216+
mapping['dim_kv'] = config.model_type === 'deepseek_v4' ? 'head_dim' : 'qk_head_dim';
216217
mapping['num_attention_heads'] = 'num_attention_heads';
217218
break;
218219

@@ -332,7 +333,7 @@ export function getCacheNames(config, options) {
332333
}
333334

334335
const pkv_prefix = options?.prefix ?? 'past_key_values';
335-
const conv_prefix = pkv_prefix === 'present' ? 'present' : 'past';
336+
const cache_prefix = pkv_prefix === 'present' ? 'present' : 'past';
336337
/** @type {Set<string>} */
337338
const names = new Set();
338339

@@ -343,7 +344,7 @@ export function getCacheNames(config, options) {
343344
names.add(`${pkv_prefix}.${i}.key`);
344345
names.add(`${pkv_prefix}.${i}.value`);
345346
} else if (layer_types[i] === 'conv') {
346-
names.add(`${conv_prefix}_conv.${i}`);
347+
names.add(`${cache_prefix}_conv.${i}`);
347348
} else {
348349
throw new Error(`Unsupported layer type: ${layer_types[i]}`);
349350
}
@@ -356,8 +357,8 @@ export function getCacheNames(config, options) {
356357

357358
for (let i = 0; i < num_layers; ++i) {
358359
if (!layer_types || layer_types[i] === 'mamba') {
359-
names.add(`${conv_prefix}_conv.${i}`);
360-
names.add(`${conv_prefix}_ssm.${i}`);
360+
names.add(`${cache_prefix}_conv.${i}`);
361+
names.add(`${cache_prefix}_ssm.${i}`);
361362
}
362363
if (!layer_types || layer_types[i] === 'attention') {
363364
names.add(`${pkv_prefix}.${i}.key`);
@@ -373,13 +374,13 @@ export function getCacheNames(config, options) {
373374
names.add(`${pkv_prefix}.${i}.value`);
374375
} else if (layer_types[i] === 'linear_attention') {
375376
if (config.model_type === 'olmo_hybrid') {
376-
names.add(`${conv_prefix}_conv.${i}.key`);
377-
names.add(`${conv_prefix}_conv.${i}.value`);
378-
names.add(`${conv_prefix}_conv.${i}.query`);
377+
names.add(`${cache_prefix}_conv.${i}.key`);
378+
names.add(`${cache_prefix}_conv.${i}.value`);
379+
names.add(`${cache_prefix}_conv.${i}.query`);
379380
} else {
380-
names.add(`${conv_prefix}_conv.${i}`);
381+
names.add(`${cache_prefix}_conv.${i}`);
381382
}
382-
names.add(`${conv_prefix}_recurrent.${i}`);
383+
names.add(`${cache_prefix}_recurrent.${i}`);
383384
} else {
384385
throw new Error(`Unsupported layer type: ${layer_types[i]}`);
385386
}
@@ -398,6 +399,27 @@ export function getCacheNames(config, options) {
398399
names.add(`${pkv_prefix}.${i}.value`);
399400
}
400401
return names;
402+
} else if (config.model_type === 'deepseek_v4') {
403+
const { layer_types, num_hidden_layers } = /** @type {any} */ (config);
404+
405+
for (let i = 0; i < num_hidden_layers; ++i) {
406+
names.add(`${pkv_prefix}.${i}.key`);
407+
names.add(`${pkv_prefix}.${i}.value`);
408+
409+
const layer_type = layer_types[i];
410+
if (layer_type === 'compressed_sparse_attention') {
411+
names.add(`${cache_prefix}_compressor.${i}.kv`);
412+
names.add(`${cache_prefix}_compressor.${i}.gate`);
413+
names.add(`${cache_prefix}_indexer.${i}.kv`);
414+
names.add(`${cache_prefix}_indexer.${i}.gate`);
415+
} else if (layer_type === 'heavily_compressed_attention') {
416+
names.add(`${cache_prefix}_compressor.${i}.kv`);
417+
names.add(`${cache_prefix}_compressor.${i}.gate`);
418+
} else if (layer_type && layer_type !== 'sliding_attention') {
419+
throw new Error(`Unsupported layer type: ${layer_type}`);
420+
}
421+
}
422+
return names;
401423
} else if (['lfm2_vl', 'qwen3_5', 'qwen3_5_moe', 'voxtral_realtime'].includes(config.model_type)) {
402424
let subConfig;
403425
if (config.model_type === 'voxtral_realtime' && options?.session_name === 'audio_encoder') {

‎packages/transformers/src/models/deepseek_v4/modeling_deepseek_v4.js‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
import { PreTrainedModel } from '../modeling_utils.js';
2+
3+
export class DeepseekV4PreTrainedModel extends PreTrainedModel {}
4+
export class DeepseekV4Model extends DeepseekV4PreTrainedModel {}
5+
export class DeepseekV4ForCausalLM extends DeepseekV4PreTrainedModel {}

‎packages/transformers/src/models/modeling_utils.js‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1189,6 +1189,8 @@ export function getPastKeyValues(decoderResults, pastKeyValues) {
11891189
.replace('present_ssm', 'past_ssm') // Mamba
11901190
.replace('present_conv', 'past_conv') // LFM2
11911191
.replace('present_recurrent', 'past_recurrent') // Qwen3.5
1192+
.replace('present_compressor', 'past_compressor') // Deepseek V4
1193+
.replace('present_indexer', 'past_indexer') // Deepseek V4
11921194

11931195
// Standard cache architecture
11941196
.replace('present', 'past_key_values');

‎packages/transformers/src/models/models.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ export * from './d_fine/modeling_d_fine.js';
2727
export * from './dac/modeling_dac.js';
2828
export * from './deberta/modeling_deberta.js';
2929
export * from './deepseek_v3/modeling_deepseek_v3.js';
30+
export * from './deepseek_v4/modeling_deepseek_v4.js';
3031
export * from './deberta_v2/modeling_deberta_v2.js';
3132
export * from './decision_transformer/modeling_decision_transformer.js';
3233
export * from './deit/modeling_deit.js';

‎packages/transformers/src/models/registry.js‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,7 @@ const MODEL_MAPPING_NAMES_DECODER_ONLY = new Map([
172172
['ernie4_5', 'Ernie4_5ForCausalLM'],
173173
['starcoder2', 'Starcoder2Model'],
174174
['deepseek_v3', 'DeepseekV3Model'],
175+
['deepseek_v4', 'DeepseekV4Model'],
175176
['falcon', 'FalconModel'],
176177
['falcon_h1', 'FalconH1Model'],
177178
['nemotron_h', 'NemotronHModel'],
@@ -318,6 +319,7 @@ export const MODEL_FOR_CAUSAL_LM_MAPPING_NAMES = new Map([
318319
['ernie4_5', 'Ernie4_5ForCausalLM'],
319320
['starcoder2', 'Starcoder2ForCausalLM'],
320321
['deepseek_v3', 'DeepseekV3ForCausalLM'],
322+
['deepseek_v4', 'DeepseekV4ForCausalLM'],
321323
['falcon', 'FalconForCausalLM'],
322324
['falcon_h1', 'FalconH1ForCausalLM'],
323325
['nemotron_h', 'NemotronHForCausalLM'],

‎packages/transformers/src/models/voxtral_realtime/modeling_voxtral_realtime.js‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -53,11 +53,11 @@ function createEncoderState(model, input_features) {
5353
}
5454

5555
const padding_cls = DataTypeMap[padding_type];
56-
const enc_padding_cache = new Tensor(
57-
padding_type,
58-
new padding_cls(PADDING_CACHE_CHANNELS * CONV1_LEFT_PAD),
59-
[1, PADDING_CACHE_CHANNELS, CONV1_LEFT_PAD],
60-
);
56+
const enc_padding_cache = new Tensor(padding_type, new padding_cls(PADDING_CACHE_CHANNELS * CONV1_LEFT_PAD), [
57+
1,
58+
PADDING_CACHE_CHANNELS,
59+
CONV1_LEFT_PAD,
60+
]);
6161

6262
// Set up iterator from input_features
6363
const chunks_iter = input_features[Symbol.asyncIterator]?.() ?? input_features[Symbol.iterator]?.();

‎packages/transformers/tests/models/deepseek_v4/test_modeling_deepseek_v4.js‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
import { PreTrainedTokenizer, DeepseekV4ForCausalLM } from "../../../src/transformers.js";
2+
3+
import { MAX_MODEL_LOAD_TIME, MAX_TEST_EXECUTION_TIME, MAX_MODEL_DISPOSE_TIME, DEFAULT_MODEL_OPTIONS } from "../../init.js";
4+
5+
export default () => {
6+
describe("DeepseekV4ForCausalLM", () => {
7+
const model_id = "onnx-internal-testing/tiny-random-DeepseekV4ForCausalLM";
8+
/** @type {DeepseekV4ForCausalLM} */
9+
let model;
10+
/** @type {PreTrainedTokenizer} */
11+
let tokenizer;
12+
beforeAll(async () => {
13+
model = await DeepseekV4ForCausalLM.from_pretrained(model_id, DEFAULT_MODEL_OPTIONS);
14+
tokenizer = await PreTrainedTokenizer.from_pretrained(model_id);
15+
tokenizer.padding_side = "left";
16+
}, MAX_MODEL_LOAD_TIME);
17+
18+
it(
19+
"batch_size=1",
20+
async () => {
21+
const inputs = tokenizer("hello");
22+
const outputs = await model.generate({
23+
...inputs,
24+
max_length: 10,
25+
do_sample: false,
26+
});
27+
expect(outputs.tolist()).toEqual([[33310n, 83919n, 100040n, 101230n, 55545n, 23803n, 76294n, 119693n, 74059n, 1036n]]);
28+
},
29+
MAX_TEST_EXECUTION_TIME,
30+
);
31+
32+
it(
33+
"batch_size>1",
34+
async () => {
35+
const inputs = tokenizer(["hello", "hello world"], { padding: true });
36+
const outputs = await model.generate({
37+
...inputs,
38+
max_length: 10,
39+
do_sample: false,
40+
});
41+
expect(outputs.tolist()).toEqual([
42+
[1n, 33310n, 83919n, 100040n, 101230n, 55545n, 23803n, 76294n, 119693n, 74059n],
43+
[33310n, 2058n, 75023n, 89073n, 120077n, 49327n, 57703n, 107564n, 3563n, 126139n],
44+
]);
45+
},
46+
MAX_TEST_EXECUTION_TIME,
47+
);
48+
49+
it(
50+
"generate w/ past_compressor and past_indexer cache",
51+
async () => {
52+
const inputs = tokenizer("hello");
53+
const { past_key_values } = await model.generate({
54+
...inputs,
55+
max_new_tokens: 1,
56+
do_sample: false,
57+
return_dict_in_generate: true,
58+
});
59+
60+
expect(past_key_values.get_seq_length()).toBeGreaterThan(0);
61+
const cache_names = ["past_key_values.0.key", "past_key_values.0.value", "past_key_values.3.key", "past_key_values.3.value", "past_compressor.2.kv", "past_compressor.2.gate", "past_indexer.2.kv", "past_indexer.2.gate", "past_compressor.3.kv", "past_compressor.3.gate"];
62+
for (const name of cache_names) {
63+
expect(past_key_values[name]).toBeDefined();
64+
}
65+
66+
await past_key_values.dispose();
67+
},
68+
MAX_TEST_EXECUTION_TIME,
69+
);
70+
71+
afterAll(async () => {
72+
await model?.dispose();
73+
}, MAX_MODEL_DISPOSE_TIME);
74+
});
75+
};

0 commit comments

Comments
 (0)