Skip to content

Commit 41e5775

Browse files
authored
fix: remove the wrong attention mask updating (Tencent#161)
1 parent b35c62a commit 41e5775

1 file changed

Lines changed: 0 additions & 6 deletions

File tree

angelslim/compressor/speculative/train/trainer/eagle3_trainer.py

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -209,12 +209,6 @@ def draft_model_training_time_test(
209209
target_logits = padding(target_logits, left=False)
210210
loss_mask = padding(loss_mask, left=False)
211211

212-
# Update attention mask to prevent attending to future positions
213-
ind = torch.arange(seq_length, device=attention_mask.device)
214-
attention_mask[:, :, ind[idx:], ind[: seq_length - idx]] = torch.finfo(
215-
attention_mask.dtype
216-
).min
217-
218212
# Step 8: Compute weighted loss
219213
ploss_weight = [0.8**i for i in range(len(plosses))]
220214
ploss = sum([ploss_weight[i] * plosses[i] for i in range(len(plosses))])

0 commit comments

Comments
 (0)