Skip to content

Commit 8236e1e

Browse files
Correct the tokamax.RaggedDotGroupSizes configs
The key generated for looking up the tokamax autotune cache in maxtext was a list, `"group_sizes":[1024, 1024, ...]`, while in the cache file, it is `"group_sizes":{"num_groups":256,"total_size":262144}`, causing the autotune cache miss. Change it to `inputs.shape[0]` only will produce the key that has `"group_sizes":{"num_groups":256,"total_size":262144}` PiperOrigin-RevId: 948385403
1 parent 15f281b commit 8236e1e

1 file changed

Lines changed: 1 addition & 2 deletions

File tree

src/maxtext/layers/moe.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1401,10 +1401,9 @@ def get_tokamax_group_sizes(group_sizes, inputs, kernel):
14011401
elif self.config.attention == "vllm_rpa":
14021402
return group_sizes
14031403
else:
1404-
num_groups = group_sizes.shape[0]
14051404
return tokamax.RaggedDotGroupSizes(
14061405
group_sizes,
1407-
(inputs.shape[0] // num_groups,) * num_groups,
1406+
inputs.shape[0],
14081407
)
14091408

14101409
def get_quantization_dtypes():

0 commit comments

Comments
 (0)