Skip to content

chore: bump transformers to 5.14.1#3055

Open
hallerite wants to merge 6 commits into
mainfrom
chore/bump-transformers
Open

chore: bump transformers to 5.14.1#3055
hallerite wants to merge 6 commits into
mainfrom
chore/bump-transformers

Conversation

@hallerite

@hallerite hallerite commented Jul 16, 2026

Copy link
Copy Markdown
Member

Summary

Branch is merged with main (256809b). The merge kept package versions identical to main; only the transformers→5.14.1 pin + the exclude-newer-package exemption differ in pyproject.toml/uv.lock.

Bumps transformers 5.6.2 → 5.14.1 (the newest release, pulled ahead of the workspace 7-day cooldown via exclude-newer-package). The only transitive change from the bump itself is safetensors 0.7.0 → 0.8.0; all dependents' constraints are satisfied (tightest: vllm>=5.5.3, verifiers>=4.56.2).

The internal-API surface prime-rl vendors (checkpoint conversion mappings, Qwen3.5 internals, auto_factory, rope, hub_kernels, the _compat.py shims) is intact across the jump. Three fixes were needed for genuine API/behavior changes.

Fixes

  • afmoecreate_causal_mask() no longer accepts cache_position. After merging main's seq_lens refactor (feat(trainer): use seq_lens as the packed-sample boundary contract #3032) the call site in the model is gone entirely, so the fix now only lives in the vendored HF test reference (tests/unit/train/models/afmoe_hf_modeling).
  • nemotron_h — the shared Mamba2 mixer (via Zamba2MambaMixer) now reads config.layer_types, which our custom NemotronHConfig didn't expose. Added a read-only layer_types property returning the canonical-remapped schedule derived from layers_block_type (a plain attribute_map alias corrupts layers_block_type because the base config remaps legacy names and writes them back). The remap_legacy_layer_types import is lazy so importing the models package doesn't hard-require a newer transformers.
  • custom-impl registration (perf) — transformers 5.13 made _LazyAutoMapping.register() a silent no-op for configs defined under transformers.* (a guard against remote code hijacking native configs). That silently dropped prime-rl's overrides of Llama / Qwen3 / GPT-OSS, so supports_custom_impl() returned False and training fell back to the slower HF implementation — a ~17% throughput regression (and higher multi-GPU peak memory) that benchmark_regression caught. Fixed by registering native-config overrides via the mapping's _extra_content directly (exactly what register() does past the guard).
    • Follow-up (4a52ee9, also flagged by Bugbot): the registration rewrite had accidentally dropped the Qwen3_5TextConfig → Qwen3_5ForCausalLM entry, silently falling back to the HF implementation for dense Qwen3.5 — restored. The AutoConfig.register("qwen3_5_text", ...) call is intentionally not restored: it's a native transformers config, already registered (same rationale as GPT-OSS).

Verification

Post-merge, on a 2× RTX PRO 6000 (workstation Blackwell, sm_120) box:

  • CPU unit suite: 409 passed (the test_load_configs examples that need Hub taskset packages fail identically on main in this environment — not bump-related).

  • GPU model suite (pytest tests/unit/train/models -m gpu, excluding test_nemotron_h.py): 40 passed, 3 skipped. Two test_nemotron_h cases hard-crash the process on this GPU arch — verified byte-for-byte identical crash point on a main worktree with transformers 5.6.2, so environmental and pre-existing, not from this bump.

  • supports_custom_impl() verified True for Llama / Qwen3 / Qwen3.5-dense / GPT-OSS configs on 5.14.1.

  • Benchmark root-caused + fixed locally (Qwen3-0.6B RL, 1-GPU, seq 65536), bisected to transformers 5.13.0:

    transformers throughput step time custom impl
    5.6.2 (baseline) 7878 tok/s 16.65s
    5.14.1 (before fix) 6503 tok/s 20.18s ❌ HF fallback
    5.14.1 (after fix) 7946 tok/s 16.51s

Notes / out of scope

  • gpt_oss grouped-MM single-process forward fails on a torchtitan.expert_parallel arg-count mismatch — transformers-independent (torchtitan is git-pinned), pre-existing, untested path.
  • mamba-ssm is a non-default dependency group dropped by uv sync --all-extras; unrelated to this bump.
  • test_qwen3_moe grad check flaked once at the tolerance margin (max grad diff 1024 vs atol=1000, ~2 bf16 ulps at that magnitude) during a full-suite run; passes deterministically standalone and on re-run, on both main and this branch.

🤖 Generated with Claude Code

Bump transformers 5.6.2 -> 5.14.1 (newest, ahead of the 7-day cooldown via
exclude-newer-package). Only transitive change is safetensors 0.7.0 -> 0.8.0.

Two custom models needed compat fixes for API changes across the jump:

- afmoe: transformers dropped the `cache_position` kwarg from
  `create_causal_mask`; remove it from `mask_kwargs` (matches laguna/gpt_oss).
- nemotron_h: the shared Mamba2 mixer (via Zamba2MambaMixer) now reads
  `config.layer_types`. Add a read-only `layer_types` property that derives the
  canonical-remapped schedule from `layers_block_type`, keeping our own naming
  intact -- a plain attribute_map alias corrupts `layers_block_type` because the
  base config remaps legacy names and writes them back.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@hallerite
hallerite marked this pull request as ready for review July 16, 2026 12:28
The merge's hand-resolved lock kept the right package versions but retained
main's resolver metadata for transformers (override + requires-dist pinned
==5.6.2, exclude-newer-package lacked the transformers exemption). That
mismatch made 'uv sync --locked' re-resolve and fail. Regenerated with
'uv lock': 0 package-version changes, only transformers==5.14.1 metadata +
the exclusion fingerprint corrected.
…5.13

transformers 5.13 made _LazyAutoMapping.register() a silent no-op for configs defined
under transformers.* (a guard against remote code hijacking native configs). That
silently dropped prime-rl's overrides of Llama/Qwen3/GPT-OSS, so supports_custom_impl
returned False and training fell back to the slower HF implementation -- a ~17%
throughput regression (and higher multi-GPU peak memory) caught by benchmark_regression.

Register via the mapping's _extra_content directly (what register() does past the guard)
so native-config overrides stick. Also make the nemotron_h remap_legacy_layer_types import
lazy so importing the models package doesn't hard-require a transformers version with it.

Verified: Qwen3-0.6B RL benchmark throughput restored 6503 -> 7946 tok/s (baseline ~7900).

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0dc2539. Configure here.

Comment thread src/prime_rl/trainer/models/__init__.py
hallerite and others added 2 commits July 20, 2026 14:09
# Conflicts:
#	src/prime_rl/trainer/models/afmoe/modeling_afmoe.py
Commit 0dc2539 rewrote the causal-LM registration block but dropped the
Qwen3_5TextConfig -> Qwen3_5ForCausalLM entry, silently falling back to the
HF implementation for dense Qwen3.5 -- the same regression class the commit
was fixing. The AutoConfig.register call is not restored: qwen3_5_text is a
native transformers config, already registered (same as GptOss).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant