chore: bump transformers to 5.14.1#3055
Open
hallerite wants to merge 6 commits into
Open
Conversation
Bump transformers 5.6.2 -> 5.14.1 (newest, ahead of the 7-day cooldown via exclude-newer-package). Only transitive change is safetensors 0.7.0 -> 0.8.0. Two custom models needed compat fixes for API changes across the jump: - afmoe: transformers dropped the `cache_position` kwarg from `create_causal_mask`; remove it from `mask_kwargs` (matches laguna/gpt_oss). - nemotron_h: the shared Mamba2 mixer (via Zamba2MambaMixer) now reads `config.layer_types`. Add a read-only `layer_types` property that derives the canonical-remapped schedule from `layers_block_type`, keeping our own naming intact -- a plain attribute_map alias corrupts `layers_block_type` because the base config remaps legacy names and writes them back. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
hallerite
marked this pull request as ready for review
July 16, 2026 12:28
# Conflicts: # uv.lock
The merge's hand-resolved lock kept the right package versions but retained main's resolver metadata for transformers (override + requires-dist pinned ==5.6.2, exclude-newer-package lacked the transformers exemption). That mismatch made 'uv sync --locked' re-resolve and fail. Regenerated with 'uv lock': 0 package-version changes, only transformers==5.14.1 metadata + the exclusion fingerprint corrected.
…5.13 transformers 5.13 made _LazyAutoMapping.register() a silent no-op for configs defined under transformers.* (a guard against remote code hijacking native configs). That silently dropped prime-rl's overrides of Llama/Qwen3/GPT-OSS, so supports_custom_impl returned False and training fell back to the slower HF implementation -- a ~17% throughput regression (and higher multi-GPU peak memory) caught by benchmark_regression. Register via the mapping's _extra_content directly (what register() does past the guard) so native-config overrides stick. Also make the nemotron_h remap_legacy_layer_types import lazy so importing the models package doesn't hard-require a transformers version with it. Verified: Qwen3-0.6B RL benchmark throughput restored 6503 -> 7946 tok/s (baseline ~7900).
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 0dc2539. Configure here.
# Conflicts: # src/prime_rl/trainer/models/afmoe/modeling_afmoe.py
Commit 0dc2539 rewrote the causal-LM registration block but dropped the Qwen3_5TextConfig -> Qwen3_5ForCausalLM entry, silently falling back to the HF implementation for dense Qwen3.5 -- the same regression class the commit was fixing. The AutoConfig.register call is not restored: qwen3_5_text is a native transformers config, already registered (same as GptOss). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Branch is merged with
main(256809b). The merge kept package versions identical to main; only thetransformers→5.14.1 pin + theexclude-newer-packageexemption differ inpyproject.toml/uv.lock.Bumps
transformers5.6.2 → 5.14.1 (the newest release, pulled ahead of the workspace 7-day cooldown viaexclude-newer-package). The only transitive change from the bump itself issafetensors0.7.0 → 0.8.0; all dependents' constraints are satisfied (tightest:vllm>=5.5.3,verifiers>=4.56.2).The internal-API surface prime-rl vendors (checkpoint conversion mappings, Qwen3.5 internals,
auto_factory, rope, hub_kernels, the_compat.pyshims) is intact across the jump. Three fixes were needed for genuine API/behavior changes.Fixes
create_causal_mask()no longer acceptscache_position. After merging main's seq_lens refactor (feat(trainer): use seq_lens as the packed-sample boundary contract #3032) the call site in the model is gone entirely, so the fix now only lives in the vendored HF test reference (tests/unit/train/models/afmoe_hf_modeling).Zamba2MambaMixer) now readsconfig.layer_types, which our customNemotronHConfigdidn't expose. Added a read-onlylayer_typesproperty returning the canonical-remapped schedule derived fromlayers_block_type(a plainattribute_mapalias corruptslayers_block_typebecause the base config remaps legacy names and writes them back). Theremap_legacy_layer_typesimport is lazy so importing the models package doesn't hard-require a newer transformers._LazyAutoMapping.register()a silent no-op for configs defined undertransformers.*(a guard against remote code hijacking native configs). That silently dropped prime-rl's overrides of Llama / Qwen3 / GPT-OSS, sosupports_custom_impl()returned False and training fell back to the slower HF implementation — a ~17% throughput regression (and higher multi-GPU peak memory) thatbenchmark_regressioncaught. Fixed by registering native-config overrides via the mapping's_extra_contentdirectly (exactly whatregister()does past the guard).Qwen3_5TextConfig → Qwen3_5ForCausalLMentry, silently falling back to the HF implementation for dense Qwen3.5 — restored. TheAutoConfig.register("qwen3_5_text", ...)call is intentionally not restored: it's a native transformers config, already registered (same rationale as GPT-OSS).Verification
Post-merge, on a 2× RTX PRO 6000 (workstation Blackwell, sm_120) box:
CPU unit suite: 409 passed (the
test_load_configsexamples that need Hub taskset packages fail identically onmainin this environment — not bump-related).GPU model suite (
pytest tests/unit/train/models -m gpu, excludingtest_nemotron_h.py): 40 passed, 3 skipped. Twotest_nemotron_hcases hard-crash the process on this GPU arch — verified byte-for-byte identical crash point on amainworktree with transformers 5.6.2, so environmental and pre-existing, not from this bump.supports_custom_impl()verified True for Llama / Qwen3 / Qwen3.5-dense / GPT-OSS configs on 5.14.1.Benchmark root-caused + fixed locally (Qwen3-0.6B RL, 1-GPU, seq 65536), bisected to transformers 5.13.0:
Notes / out of scope
torchtitan.expert_parallelarg-count mismatch — transformers-independent (torchtitan is git-pinned), pre-existing, untested path.mamba-ssmis a non-default dependency group dropped byuv sync --all-extras; unrelated to this bump.test_qwen3_moegrad check flaked once at the tolerance margin (max grad diff 1024 vsatol=1000, ~2 bf16 ulps at that magnitude) during a full-suite run; passes deterministically standalone and on re-run, on both main and this branch.🤖 Generated with Claude Code