Skip to content

Commit 381d0b8

Browse files
malay-nagdamalay-nagda
authored
26.06.01 perf summary (#4989)
Signed-off-by: Malay Nagda <malayn@nvidia.com>
1 parent d11a6ac commit 381d0b8

2 files changed

Lines changed: 117 additions & 68 deletions

File tree

‎docs/performance-summary-archive.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,107 @@ The performance data includes:
3939

4040
---
4141

42+
## 26.06 NeMo Container
43+
44+
### Pre-Training Performance
45+
46+
#### Model: LLAMA3.1_405B
47+
48+
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
49+
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
50+
| DGX-GB300 | 256 | FP8 | 1536 | 1 | 8192 | 4 | 8 | 1 | 4 | n/a | 1048 | 2646 |
51+
| DGX-GB300 | 256 | MXFP8 | 1536 | 1 | 8192 | 2 | 8 | 2 | 4 | n/a | 952 | 2403 |
52+
| DGX-GB300 | 256 | NVFP4 | 1536 | 1 | 8192 | 4 | 8 | 1 | 4 | n/a | 1413 | 3575 |
53+
| DGX-GB200 | 256 | FP8 | 1536 | 1 | 8192 | 4 | 16 | 1 | 4 | n/a | 843 | 2129 |
54+
| DGX-GB200 | 256 | MXFP8 | 1536 | 1 | 8192 | 4 | 16 | 1 | 8 | n/a | 783 | 1976 |
55+
| DGX-GB200 | 256 | NVFP4 | 1536 | 1 | 8192 | 4 | 16 | 1 | 8 | n/a | 1166 | 2944 |
56+
| DGX-H100 | 1024 | FP8 | 1536 | 1 | 8192 | 8 | 8 | 2 | 8 | n/a | 326 | 822 |
57+
58+
#### Model: DeepSeekV3
59+
60+
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
61+
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
62+
| DGX-GB300 | 256 | MXFP8 | 4096 | 1 | 4096 | 1 | 2 | 1 | 8 | 32 | 6338 | 1648 |
63+
| DGX-GB300 | 256 | MXFP8 | 15360 | 1 | 4096 | 1 | 2 | 1 | 8 | 32 | 6422 | 1670 |
64+
| DGX-GB200 | 256 | MXFP8 | 4096 | 1 | 4096 | 1 | 4 | 1 | 4 | 64 | 4969 | 1292 |
65+
| DGX-B300 | 256 | MXFP8 | 4096 | 2 | 4096 | 1 | 8 | 1 | n/a | 8 | 3541 | 920 |
66+
67+
#### Model: GPT OSS 120B
68+
69+
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
70+
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
71+
| DGX-GB300 | 64 | MXFP8 | 1280 | 4 | 4096 | 1 | 1 | 1 | n/a | 16 | 33166 | 1081 |
72+
| DGX-GB200 | 64 | MXFP8 | 1280 | 4 | 4096 | 1 | 1 | 1 | n/a | 64 | 28947 | 943 |
73+
| DGX-B300 | 64 | MXFP8 | 1280 | 4 | 4096 | 1 | 1 | 1 | n/a | 8 | 18534 | 604 |
74+
75+
#### Model: Qwen3_30B_a3B
76+
77+
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
78+
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
79+
| DGX-GB300 | 8 | MXFP8 | 512 | 8 | 4096 | 1 | 1 | 1 | n/a | 8 | 45275 | 1041 |
80+
| DGX-GB200 | 8 | MXFP8 | 512 | 4 | 4096 | 1 | 1 | 1 | n/a | 8 | 40706 | 936 |
81+
| DGX-B300 | 8 | MXFP8 | 512 | 8 | 4096 | 1 | 1 | 1 | n/a | 8 | 40769 | 938 |
82+
| DGX-H100 | 16 | FP8 | 1024 | 1 | 4096 | 1 | 1 | 1 | n/a | 16 | 8826 | 203 |
83+
84+
#### Model: Qwen3_235B_a22B
85+
86+
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
87+
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
88+
| DGX-GB300 | 256 | MXFP8 | 8192 | 2 | 4096 | 1 | 4 | 1 | 12 | 32 | 9015 | 1335 |
89+
| DGX-GB200 | 256 | MXFP8 | 8192 | 1 | 4096 | 1 | 8 | 1 | 3 | 32 | 7376 | 1092 |
90+
91+
#### Model: Kimi_K2
92+
93+
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
94+
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
95+
| DGX-GB300 | 256 | MXFP8 | 4096 | 2 | 4096 | 1 | 4 | 1 | 4 | 64 | 5372 | 1099 |
96+
97+
- Muon optimizer was used for pre-training Kimi-K2.
98+
99+
#### Model: Nemotron_3_Nano
100+
101+
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
102+
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
103+
| DGX-GB300 | 8 | MXFP8 | 512 | 4 | 8192 | 1 | 1 | 1 | n/a | 8 | 39749 | 885 |
104+
| DGX-GB200 | 8 | MXFP8 | 512 | 2 | 8192 | 1 | 1 | 1 | n/a | 8 | 33522 | 747 |
105+
| DGX-B300 | 8 | MXFP8 | 512 | 4 | 8192 | 1 | 1 | 1 | n/a | 8 | 37316 | 831 |
106+
| DGX-H100 | 16 | FP8 | 1024 | 1 | 8192 | 1 | 1 | 1 | n/a | 8 | 14719 | 328 |
107+
108+
#### Model: Nemotron_3_Super
109+
110+
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
111+
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
112+
| DGX-GB300 | 64 | MXFP8 | 512 | 1 | 8192 | 1 | 1 | 1 | n/a | 64 | 9652 | 817 |
113+
| DGX-GB300 | 64 | NVFP4 | 512 | 1 | 8192 | 1 | 1 | 1 | n/a | 64 | 9900 | 839 |
114+
| DGX-GB200 | 64 | MXFP8 | 512 | 1 | 8192 | 2 | 1 | 1 | n/a | 64 | 6742 | 571 |
115+
| DGX-GB200 | 64 | NVFP4 | 512 | 1 | 8192 | 2 | 1 | 1 | n/a | 64 | 6928 | 587 |
116+
| DGX-B300 | 64 | MXFP8 | 512 | 1 | 8192 | 1 | 1 | 1 | n/a | 8 | 7867 | 667 |
117+
| DGX-B300 | 64 | NVFP4 | 512 | 1 | 8192 | 1 | 1 | 1 | n/a | 8 | 8131 | 689 |
118+
119+
### SFT Performance
120+
121+
#### Model: LLAMA3_70B
122+
123+
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
124+
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
125+
| DGX-GB300 | 32 | FP8 | 32 | 1 | 4096 | 1 | 2 | 1 | 20 | n/a | 4819 | 2083 |
126+
| DGX-GB300 | 32 | MXFP8 | 32 | 1 | 4096 | 1 | 2 | 1 | 20 | n/a | 4312 | 1877 |
127+
| DGX-GB200 | 32 | FP8 | 32 | 1 | 4096 | 1 | 8 | 1 | 10 | n/a | 3864 | 1671 |
128+
| DGX-GB200 | 32 | MXFP8 | 32 | 1 | 4096 | 1 | 8 | 1 | 10 | n/a | 3593 | 1553 |
129+
| DGX-H100 | 32 | FP8 | 32 | 1 | 4096 | 4 | 4 | 1 | 5 | n/a | 1638 | 710 |
130+
131+
### LoRA Performance
132+
133+
#### Model: LLAMA3_70B
134+
135+
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
136+
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
137+
| DGX-GB300 | 8 | FP8 | 32 | 1 | 4096 | 1 | 2 | 1 | 20 | n/a | 7481 | 2086 |
138+
| DGX-GB300 | 8 | MXFP8 | 32 | 1 | 4096 | 1 | 2 | 1 | 20 | n/a | 7447 | 2072 |
139+
| DGX-GB200 | 8 | FP8 | 32 | 1 | 4096 | 1 | 2 | 1 | 20 | n/a | 6206 | 1731 |
140+
| DGX-GB200 | 8 | MXFP8 | 32 | 1 | 4096 | 1 | 4 | 1 | 20 | n/a | 5958 | 1663 |
141+
| DGX-H100 | 8 | FP8 | 32 | 1 | 4096 | 2 | 4 | 1 | 20 | n/a | 2643 | 735 |
142+
42143
## 26.04.01 NeMo Container
43144

44145
### Pre-Training Performance

‎docs/performance-summary.md‎

Lines changed: 16 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -34,108 +34,56 @@ The performance data includes:
3434

3535
---
3636

37-
## 26.06 NeMo Container
37+
## 26.06.01 NeMo Container
3838

3939
### Pre-Training Performance
4040

41-
#### Model: LLAMA3.1_405B
42-
43-
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
44-
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
45-
| DGX-GB300 | 256 | FP8 | 1536 | 1 | 8192 | 4 | 8 | 1 | 4 | n/a | 1048 | 2646 |
46-
| DGX-GB300 | 256 | MXFP8 | 1536 | 1 | 8192 | 2 | 8 | 2 | 4 | n/a | 952 | 2403 |
47-
| DGX-GB300 | 256 | NVFP4 | 1536 | 1 | 8192 | 4 | 8 | 1 | 4 | n/a | 1413 | 3575 |
48-
| DGX-GB200 | 256 | FP8 | 1536 | 1 | 8192 | 4 | 16 | 1 | 4 | n/a | 843 | 2129 |
49-
| DGX-GB200 | 256 | MXFP8 | 1536 | 1 | 8192 | 4 | 16 | 1 | 8 | n/a | 783 | 1976 |
50-
| DGX-GB200 | 256 | NVFP4 | 1536 | 1 | 8192 | 4 | 16 | 1 | 8 | n/a | 1166 | 2944 |
51-
| DGX-H100 | 1024 | FP8 | 1536 | 1 | 8192 | 8 | 8 | 2 | 8 | n/a | 326 | 822 |
52-
5341
#### Model: DeepSeekV3
5442

5543
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
5644
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
57-
| DGX-GB300 | 256 | MXFP8 | 4096 | 1 | 4096 | 1 | 2 | 1 | 8 | 32 | 6338 | 1648 |
58-
| DGX-GB300 | 256 | MXFP8 | 15360 | 1 | 4096 | 1 | 2 | 1 | 8 | 32 | 6422 | 1670 |
59-
| DGX-GB200 | 256 | MXFP8 | 4096 | 1 | 4096 | 1 | 4 | 1 | 4 | 64 | 4969 | 1292 |
60-
| DGX-B300 | 256 | MXFP8 | 4096 | 2 | 4096 | 1 | 8 | 1 | n/a | 8 | 3541 | 920 |
45+
| DGX-GB300 | 256 | MXFP8 | 4096 | 1 | 4096 | 1 | 2 | 1 | 8 | 32 | 6304 | 1640 |
46+
| DGX-GB200 | 256 | MXFP8 | 4096 | 1 | 4096 | 1 | 4 | 1 | 4 | 64 | 4928 | 1280 |
6147

6248
#### Model: GPT OSS 120B
6349

6450
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
6551
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
66-
| DGX-GB300 | 64 | MXFP8 | 1280 | 4 | 4096 | 1 | 1 | 1 | n/a | 16 | 33166 | 1081 |
67-
| DGX-GB200 | 64 | MXFP8 | 1280 | 4 | 4096 | 1 | 1 | 1 | n/a | 64 | 28947 | 943 |
68-
| DGX-B300 | 64 | MXFP8 | 1280 | 4 | 4096 | 1 | 1 | 1 | n/a | 8 | 18534 | 604 |
52+
| DGX-GB300 | 64 | MXFP8 | 1280 | 4 | 4096 | 1 | 1 | 1 | n/a | 16 | 20288 | 661[^gpt-oss-note] |
53+
| DGX-GB200 | 64 | MXFP8 | 1280 | 4 | 4096 | 1 | 1 | 1 | n/a | 64 | 18304 | 597[^gpt-oss-note] |
6954

7055
#### Model: Qwen3_30B_a3B
7156

7257
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
7358
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
74-
| DGX-GB300 | 8 | MXFP8 | 512 | 8 | 4096 | 1 | 1 | 1 | n/a | 8 | 45275 | 1041 |
75-
| DGX-GB200 | 8 | MXFP8 | 512 | 4 | 4096 | 1 | 1 | 1 | n/a | 8 | 40706 | 936 |
76-
| DGX-B300 | 8 | MXFP8 | 512 | 8 | 4096 | 1 | 1 | 1 | n/a | 8 | 40769 | 938 |
77-
| DGX-H100 | 16 | FP8 | 1024 | 1 | 4096 | 1 | 1 | 1 | n/a | 16 | 8826 | 203 |
59+
| DGX-GB300 | 8 | MXFP8 | 512 | 8 | 4096 | 1 | 1 | 1 | n/a | 8 | 44544 | 1029 |
60+
| DGX-GB200 | 8 | MXFP8 | 512 | 4 | 4096 | 1 | 1 | 1 | n/a | 8 | 40960 | 937 |
7861

7962
#### Model: Qwen3_235B_a22B
8063

8164
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
8265
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
83-
| DGX-GB300 | 256 | MXFP8 | 8192 | 2 | 4096 | 1 | 4 | 1 | 12 | 32 | 9015 | 1335 |
84-
| DGX-GB200 | 256 | MXFP8 | 8192 | 1 | 4096 | 1 | 8 | 1 | 3 | 32 | 7376 | 1092 |
85-
86-
#### Model: Kimi_K2
87-
88-
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
89-
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
90-
| DGX-GB300 | 256 | MXFP8 | 4096 | 2 | 4096 | 1 | 4 | 1 | 4 | 64 | 5372 | 1099 |
91-
92-
- Muon optimizer was used for pre-training Kimi-K2.
66+
| DGX-GB300 | 256 | MXFP8 | 8192 | 2 | 4096 | 1 | 4 | 1 | 12 | 32 | 9008 | 1333 |
67+
| DGX-GB200 | 256 | MXFP8 | 8192 | 1 | 4096 | 1 | 8 | 1 | 3 | 32 | 7360 | 1089 |
9368

9469
#### Model: Nemotron_3_Nano
9570

9671
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
9772
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
98-
| DGX-GB300 | 8 | MXFP8 | 512 | 4 | 8192 | 1 | 1 | 1 | n/a | 8 | 39749 | 885 |
99-
| DGX-GB200 | 8 | MXFP8 | 512 | 2 | 8192 | 1 | 1 | 1 | n/a | 8 | 33522 | 747 |
100-
| DGX-B300 | 8 | MXFP8 | 512 | 4 | 8192 | 1 | 1 | 1 | n/a | 8 | 37316 | 831 |
101-
| DGX-H100 | 16 | FP8 | 1024 | 1 | 8192 | 1 | 1 | 1 | n/a | 8 | 14719 | 328 |
73+
| DGX-GB300 | 8 | MXFP8 | 512 | 4 | 8192 | 1 | 1 | 1 | n/a | 8 | 40960 | 905 |
74+
| DGX-GB200 | 8 | MXFP8 | 512 | 2 | 8192 | 1 | 1 | 1 | n/a | 8 | 34816 | 776 |
10275

10376
#### Model: Nemotron_3_Super
10477

10578
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
10679
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
107-
| DGX-GB300 | 64 | MXFP8 | 512 | 1 | 8192 | 1 | 1 | 1 | n/a | 64 | 9652 | 817 |
108-
| DGX-GB300 | 64 | NVFP4 | 512 | 1 | 8192 | 1 | 1 | 1 | n/a | 64 | 9900 | 839 |
109-
| DGX-GB200 | 64 | MXFP8 | 512 | 1 | 8192 | 2 | 1 | 1 | n/a | 64 | 6742 | 571 |
110-
| DGX-GB200 | 64 | NVFP4 | 512 | 1 | 8192 | 2 | 1 | 1 | n/a | 64 | 6928 | 587 |
111-
| DGX-B300 | 64 | MXFP8 | 512 | 1 | 8192 | 1 | 1 | 1 | n/a | 8 | 7867 | 667 |
112-
| DGX-B300 | 64 | NVFP4 | 512 | 1 | 8192 | 1 | 1 | 1 | n/a | 8 | 8131 | 689 |
113-
114-
### SFT Performance
115-
116-
#### Model: LLAMA3_70B
117-
118-
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
119-
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
120-
| DGX-GB300 | 32 | FP8 | 32 | 1 | 4096 | 1 | 2 | 1 | 20 | n/a | 4819 | 2083 |
121-
| DGX-GB300 | 32 | MXFP8 | 32 | 1 | 4096 | 1 | 2 | 1 | 20 | n/a | 4312 | 1877 |
122-
| DGX-GB200 | 32 | FP8 | 32 | 1 | 4096 | 1 | 8 | 1 | 10 | n/a | 3864 | 1671 |
123-
| DGX-GB200 | 32 | MXFP8 | 32 | 1 | 4096 | 1 | 8 | 1 | 10 | n/a | 3593 | 1553 |
124-
| DGX-H100 | 32 | FP8 | 32 | 1 | 4096 | 4 | 4 | 1 | 5 | n/a | 1638 | 710 |
125-
126-
### LoRA Performance
127-
128-
#### Model: LLAMA3_70B
129-
130-
| System | #-GPUs | Precision | GBS | MBS | Sequence Length | TP | PP | CP | VP | EP | Tokens / sec / GPU | Model TFLOP / sec / GPU |
131-
|--------|--------|-----------|-----|-----|-----------------|----|----|----|----|----|-----------------------|-------------------------|
132-
| DGX-GB300 | 8 | FP8 | 32 | 1 | 4096 | 1 | 2 | 1 | 20 | n/a | 7481 | 2086 |
133-
| DGX-GB300 | 8 | MXFP8 | 32 | 1 | 4096 | 1 | 2 | 1 | 20 | n/a | 7447 | 2072 |
134-
| DGX-GB200 | 8 | FP8 | 32 | 1 | 4096 | 1 | 2 | 1 | 20 | n/a | 6206 | 1731 |
135-
| DGX-GB200 | 8 | MXFP8 | 32 | 1 | 4096 | 1 | 4 | 1 | 20 | n/a | 5958 | 1663 |
136-
| DGX-H100 | 8 | FP8 | 32 | 1 | 4096 | 2 | 4 | 1 | 20 | n/a | 2643 | 735 |
80+
| DGX-GB300 | 64 | MXFP8 | 512 | 1 | 8192 | 1 | 1 | 1 | n/a | 64 | 9728 | 827 |
81+
| DGX-GB300 | 64 | NVFP4 | 512 | 1 | 8192 | 1 | 1 | 1 | n/a | 64 | 9984 | 845 |
82+
| DGX-GB200 | 64 | MXFP8 | 512 | 1 | 8192 | 2 | 1 | 1 | n/a | 64 | 7040 | 598 |
83+
| DGX-GB200 | 64 | NVFP4 | 512 | 1 | 8192 | 2 | 1 | 1 | n/a | 64 | 7040 | 598 |
13784

13885
[^moe-training-note]: In MoE training benchmarks, we force-balance the token distribution among experts and all benchmarks are token-dropless.
86+
[^gpt-oss-note]: Performance regression in GPT-OSS 120B pre-training is a [known issue](https://github.com/NVIDIA/cudnn-frontend/issues/256). It will be resolved with the next container release.
13987

14088
## Archive
14189

0 commit comments

Comments
 (0)