Skip to content

Commit eeb0154

Browse files
committed
Optimize mHC for expansion rate 4 using convex combination of permutations and add enable_mhc_k4_shortcut feature gate
1 parent 4d9f390 commit eeb0154

5 files changed

Lines changed: 385 additions & 102 deletions

File tree

src/maxtext/configs/base.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1216,6 +1216,8 @@ force_q_layout: false
12161216
mhc_expansion_rate: 1
12171217
# The number of iterations for the Sinkhorn-Knopp algorithm.
12181218
sinkhorn_iterations: 20
1219+
# Whether to enable the permutation-based convex combination shortcut when mhc_expansion_rate is 4.
1220+
enable_mhc_k4_shortcut: True
12191221

12201222
################################## DeepSeek Engram ##################################
12211223
# Indices of transformer layers where Engram are integrated; leave empty [] to disable.

0 commit comments

Comments
 (0)