Measure routing mismatch for Qwen models#4060
Open
xuefgu wants to merge 1 commit into
Open
Conversation
xuefgu
force-pushed
the
xfgu-routing-mismatch
branch
from
2139c05 to
533473f
Compare
xuefgu
changed the title
xuefgu
force-pushed
the
xfgu-routing-mismatch
branch
3 times, most recently
from
718e421 to
db2b48f
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
xuefgu
force-pushed
the
xfgu-routing-mismatch
branch
12 times, most recently
from
7dafd25 to
fc983eb
Compare
xuefgu
requested review from
RissyRan,
darisoy,
gagika,
jesselu-google,
jiangjy1982,
michelle-yooh,
parambole,
richjames0,
shralex,
shuningjin and
suexu1025
as code owners
xuefgu
requested review from
A9isha,
NicoGrande,
NuojCheng,
SurbhiJainUSC,
abhinavclemson,
aireenmei,
bvandermoon,
dipannita08,
hengtaoguo,
igorts-git,
khatwanimohit and
vipannalla
as code owners
xuefgu
force-pushed
the
xfgu-routing-mismatch
branch
14 times, most recently
from
3c2748a to
61fd64d
Compare
igorts-git
approved these changes
Jun 16, 2026
xuefgu
force-pushed
the
xfgu-routing-mismatch
branch
from
61fd64d to
1b071c3
Compare
Calculate the mismatch rate between model-calculated routing and externally supplied forced_routed_experts. This is limited to Qwen3 and Qwen3.5 models in training mode. The mismatch rate is logged as 'learning/routing_mismatch_rate'.
xuefgu
force-pushed
the
xfgu-routing-mismatch
branch
from
1b071c3 to
58e4ddc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR implements MoE routing mismatch measurement during training. It compares the model's natural routing decisions with forced_routed_experts passed to the model call, and logs the average mismatch rate.
Crucially, this PR only measures the mismatch and does not force the routing. The model's natural routing is still used for the actual computation.
The changes are only scoped to Qwen3 and Qwen3.5. More model support should be added at a later time.
Tests
Unit tests.
Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.