Skip to content

Commit 5e0c29e

Browse files
committed
[Common] Document sequence aux loss column expansion
Signed-off-by: Harry Zhou <hhanyu@nvidia.com>
1 parent 230efa9 commit 5e0c29e

1 file changed

Lines changed: 1 addition & 0 deletions

File tree

transformer_engine/common/fused_router/fused_moe_aux_loss.cu

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ void fused_moe_aux_loss_forward_kernel_launcher(const DataType* probs,
8989
cudaStream_t stream) {
9090
NVTE_CHECK(num_cols > 0, "num_cols must be positive, got ", num_cols);
9191
NVTE_CHECK(num_experts > 0, "num_experts must be positive, got ", num_experts);
92+
// Sequence aux loss batches independent sequences along the expert dimension.
9293
NVTE_CHECK(num_cols % num_experts == 0, "Number of input columns (", num_cols,
9394
") must be a multiple of number of experts (", num_experts, ").");
9495

0 commit comments

Comments
 (0)