Skip to content

Commit e4d97a3

Browse files
PerryZhang01zgplvyou
authored andcommitted
[fix](gpt-oss): fix quark quantized model in moe bias (#787)
Co-authored-by: perzhang <perzhang@amd.com>
1 parent 8ddcb7d commit e4d97a3

1 file changed

Lines changed: 2 additions & 2 deletions

File tree

atom/model_ops/moe.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -756,7 +756,7 @@ def create_weights(
756756

757757
if layer.has_bias:
758758
w13_bias = atom_parameter(
759-
torch.empty(
759+
torch.zeros(
760760
num_experts,
761761
2 * intermediate_size_per_partition_after_pad,
762762
dtype=torch.bfloat16,
@@ -793,7 +793,7 @@ def create_weights(
793793

794794
if layer.has_bias:
795795
w2_bias = atom_parameter(
796-
torch.empty(
796+
torch.zeros(
797797
num_experts,
798798
hidden_size,
799799
dtype=torch.bfloat16,

0 commit comments

Comments
 (0)