Skip to content

Commit 7ecbc69

Browse files
authored
fix(fsdp2): avoid double-dividing tokens_per_gpu by sp_size (#158)
1 parent b928344 commit 7ecbc69

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

src/lmms_engine/train/fsdp2/fsdp2_trainer.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -669,7 +669,7 @@ def calculate_training_metrics(
669669
total_tokens += total_seq_len.item()
670670

671671
tokens_per_second = total_seq_len.item() / delta_time
672-
tokens_per_gpu = tokens_per_second / sp_size / world_size
672+
tokens_per_gpu = tokens_per_second / world_size
673673

674674
# Log total tokens and total tokens per second
675675
metrics["train/total_tokens"] = TrainUtilities.format_tokens(total_tokens)

0 commit comments

Comments
 (0)