Skip to content

feat(model): add model support verification cards#4909

Merged
yaoyu-33 merged 100 commits into
mainfrom
yuya/mb-575-model-support-cards
Jul 21, 2026
Merged

feat(model): add model support verification cards#4909
yaoyu-33 merged 100 commits into
mainfrom
yuya/mb-575-model-support-cards

Conversation

@yaoyu-33

@yaoyu-33 yaoyu-33 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds concise, agent-readable model verification cards under examples/model_verification_cards/ for four variants:

  • Qwen/Qwen3-8B
  • Qwen/Qwen3-30B-A3B
  • moonshotai/Moonlight-16B-A3B
  • nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

It also adds the repo-local create-model-verification-card skill, its deterministic validator, and a helper for exact-length, byte-stable HF inference verification.

The cards distinguish bounded convergence evidence from benchmark-only performance evidence, pin immutable Hugging Face and dataset revisions, preserve recipe-owned GBS/MBS/LR contracts, require saved post-setup runtime configuration, and keep private executor wiring out of tracked cards and PR text.

Card contract

  • Keep all twelve core items; include pretrain_performance only when the exact variant has a canonical public performance recipe.
  • Start untuned cards with the required sanity-check disclaimer; scope tuned claims to exact pretrain_performance.<hardware> leaves.
  • Use portable public launchers and short logical work/... paths.
  • Pin immutable model, tokenizer, and dataset revisions plus split, seeds, and packing alignment.
  • Keep recipe GBS/MBS/LR unchanged in verification commands.
  • Record initial/final loss plus final-ten mean step time and model TFLOP/s/GPU for every verified training item.
  • Nest hardware-dependent training and SFT export evidence under public accelerator keys (for example, pretrain.H100), with resume/export dependencies resolved only within the same hardware target.
  • Require exact-length, byte-stable inference output from two executions.
  • Verify manual forward correlation with an exact next-token match and cosine similarity at least 0.99; maximum and mean absolute logit differences remain report-only diagnostics.
  • Resume directly from the uninterrupted bounded-pretrain step-50 checkpoint to step 100 in a distinct output root.
  • Treat submission or a short smoke as evidence-in-progress, not verification.

Current verification status

The branch remains a draft while the final clean-contract resume/export audits finish. Current head: fbd1ad6a4.

Variant Tracked card Completed evidence awaiting final audit/card promotion Remaining runtime gate
Qwen3-8B 10/12 Bounded source checkpoints Direct checkpoint resume and frozen audit
Qwen3-30B-A3B 10/13 Full SFT, strict post-SFT export/inference, PEFT, supervised audit Frozen export audit and card promotion
Moonlight-16B-A3B 7/12 Full SFT and PEFT Direct resume, corrected post-SFT export/inference, and frozen audits
Nemotron 3 Nano 4B 12/12 Complete None

The three long-context items are verified from one coherent clean-run evidence cohort:

Variant Loss Final-ten step time Final-ten TFLOP/s/GPU Packing Actual supervised tokens
Qwen3-8B 1.611622 -> 1.490618 71668.780 ms 34.080 99.97% 3,444,917
Qwen3-30B-A3B 1.645009 -> 1.468103 142663.710 ms 26.120 99.28% 13,573,663
Moonlight-16B-A3B 1.306440 -> 1.228019 60599.310 ms 40.840 99.69% 13,765,732

Each completed exactly 20 optimizer steps with finite losses, zero skipped/NaN iterations, a persisted runtime config, and independently reconstructed packed-sample accounting.

GPU export config correction

The final commit fixes GPU export for checkpoints whose training-time config differs from the original Hugging Face config. Export retains the existing state-backed reference wrapper and shard map, but replaces its config with the checkpoint-derived Hugging Face config before saving. This preserves strict/distributed streaming behavior while making serialized config shapes agree with the checkpoint tensors.

The focused GPU-export tests pass 10/10 on both macOS and the fixed Linux container. Repository-wide pre-commit passes in that Linux container. The current public docs, secrets, and DCO checks are green.

Revision hardening and validation

  • Hugging Face model and tokenizer revisions are persisted by the affected recipes instead of relying on mutable defaults.
  • The revision-hardening patch passed 184 targeted tests and repository-wide pre-commit.
  • The current long-context/card update passed all three card validators, tokenizer unit tests, Ruff checks, the CI-equivalent secrets scan, and repository-wide pre-commit.
  • Two public immutable revision literals triggered the generic high-entropy detector; narrow inline allowlist annotations scope those known false positives to the exact literals.
  • git diff --check passes, and independent evidence reviews reproduced every promoted metric and supervised-token count.

Remaining work

The final bounded-pretrain direct resumes and corrected Moonlight export are queued from frozen, independently reviewed launchers. Execution state alone does not promote an item: each remains unverified until its exact optimizer-step set, finite-loss/no-skip gates, saved runtime config, checkpoint or adapter artifacts, packing/supervised-token evidence, resume sentinels, strict load, deterministic inference, and frozen evidence audit all pass. The cards, PR status, and Linear issues will be promoted only after those gates complete.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 added docs-only With great power comes great responsibility. docs Documentation-only updates or documentation debt area:model Model implementations and HF bridge logic labels Jul 16, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 16, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 changed the title docs(model): draft model support card with Qwen3 example docs(model): add minimal Qwen3 support card draft Jul 16, 2026
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Comment thread examples/model_verification_cards/qwen3-8b/card.yaml
Comment thread examples/model_verification_cards/qwen3-8b/card.yaml
Comment thread examples/model_verification_cards/qwen3-8b/card.yaml
yaoyu-33 added 8 commits July 16, 2026 15:51
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 changed the title docs(model): add minimal Qwen3 support card draft docs(model): add verified Qwen3 support card Jul 17, 2026
yaoyu-33 added 4 commits July 16, 2026 18:29
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…port-cards

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 changed the title docs(model): add verified Qwen3 support card docs(model): add Qwen3 support card Jul 17, 2026
yaoyu-33 added 5 commits July 16, 2026 22:29
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 changed the title docs(model): add Qwen3 support card docs(model): add Qwen3 support card and verification skill Jul 17, 2026
Comment thread model_cards/qwen3-8b/card.yaml Outdated
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
yaoyu-33 marked this pull request as ready for review July 21, 2026 21:12
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/nvskills-ci

the deferred MCore post-init logic.
"""
if self.mamba_chunk_size < 1:
raise ValueError("mamba_chunk_size must be at least 1.")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new mamba_chunk_size < 1 guard in finalize() has no test. test_hybrid_provider.py covers the default (128) and the applied non-default (256), but never asserts that a value < 1 raises ValueError. Consider adding a small pytest.raises(ValueError) case to lock this in.

@claude

claude Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Review — model support verification cards

Light review focused on correctness, tests, and docs. Overall a large but well-structured PR with strong test coverage. No blocking bugs found.

Verified as correct

  • GPU export config swap (scripts/conversion/gpu_backend.py:378): bridge.hf_pretrained.config = checkpoint_config_bridge.hf_pretrained looks suspicious, but from_auto_config returns a config-only bridge whose .hf_pretrained is a bare PretrainedConfig (not a wrapper), which is exactly what the .config setter expects. Matches the existing idiom at auto_bridge.py:994. Correct.
  • DeepSeek/Moonlight defaults removal (deepseek_v3_bridge.py): dropping hardcoded moe_aux_loss_coeff=0.0001 and seq_length=4096 for config-derived values is covered by the new test_provider_bridge_preserves_model_specific_context_and_aux_loss, and the inv_freq rewrite is covered by the two rewritten export tests.
  • Sampler seed propagation (loaders.py/samplers.py): the seed/random_seed/rng.seed fallback is sound; MegatronPretrainingBatchSampler already accepts seed. Covered by loader + sampler tests.
  • compare.py / generate_text.py inference-context refactor: good coverage in test_compare_mask_handling.py.

Minor test gaps (non-blocking)

  • hybrid_provider.py finalize() now raises ValueError when mamba_chunk_size < 1, but no test asserts the raise (inline comment left).
  • run_conversion.py export branch calls resolve_hf_model_revision to rewrite args.hf_model; only the import branch is tested at the dispatch level. The helper itself is unit-tested in test_utils.py, so low-risk.
  • tokenizer._is_local_tokenizer_path: the ./ ../ ~-prefixed and is_absolute() string branches, plus the empty/whitespace revision ValueError, are not directly exercised.

Docs

  • docs/models/qwen/qwen.md and the fern .mdx copy are byte-identical and consistent with the updated topologies (30B-A3B pretrain/SFT EP=16/16 GPUs, PEFT TP=4/EP=4/4 GPUs). The recipe-recommender eval was updated to match. Good.

Suggested test cases

  • tests/unit_tests/models/hybrid/test_hybrid_provider.py — assert HybridModelProvider(..., mamba_chunk_size=0).finalize() raises ValueError
  • tests/unit_tests/conversion/launcher/test_run_conversion.py — export-branch dispatch test verifying args.hf_model is rewritten via resolve_hf_model_revision
  • tests/unit_tests/training/test_tokenizer.py — cases for _is_local_tokenizer_path on ./ ../ ~-prefixed and absolute-path strings, and the empty/whitespace revision ValueError

No perf tests impacted. The touched perf_recipes/qwen/h100/qwen3_moe.py only re-bases qwen3_30b_a3b_pretrain_16gpu_h100_bf16_config onto the new library recipe; there is no scripts/performance/configs/ tree in this repo. The affected perf configs are exercised by test_perf_config_integration.py::test_qwen3_30b_flat_perf_config_instantiation and tests/unit_tests/recipes/test_qwen_recipes.py (covering qwen3_30b_a3b_pretrain_16gpu_h100_bf16_config and qwen3_30b_a3b_pretrain_16gpu_h100_fp8cs_config).

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/nvskills-ci

yaoyu-33 added 4 commits July 21, 2026 15:50
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/nvskills-ci

@yaoyu-33 yaoyu-33 added full-test-suite high-complexity Harder to merge: prone to conflicts and needs additional test coverage needs-review PR is ready for code review and waiting on a reviewer labels Jul 21, 2026
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/nvskills-ci

@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test 52c5214

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model implementations and HF bridge logic feature New capabilities, enhancements, or enablement work full-test-suite high-complexity Harder to merge: prone to conflicts and needs additional test coverage needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants