Skip to content

[EXPERIMENT][WIP] Add SmolLM3 support for OpenVINO export#8

Closed
mlukasze wants to merge 2 commits into
mainfrom
enable/HuggingFaceTB-SmolLM3-3B
Closed

[EXPERIMENT][WIP] Add SmolLM3 support for OpenVINO export#8
mlukasze wants to merge 2 commits into
mainfrom
enable/HuggingFaceTB-SmolLM3-3B

Conversation

@mlukasze

@mlukasze mlukasze commented Jun 1, 2026

Copy link
Copy Markdown
Owner

⚠️ AUTOMATICALLY GENERATED BY OMEGA AGENT — REQUIRES HUMAN REVIEW ⚠️
This PR was created by an AI agent as part of automated model enablement.
A human maintainer must review and approve it before it can be considered for merge.
Do NOT merge without human review and sign-off.

Summary

This PR adds OpenVINO export and inference support for SmolLM3 (HuggingFaceTB/SmolLM3-3B), and fixes a bug in OVModelForCausalLM that caused generation drift with recent transformers (>=5.x).

Changes

1. Add SmolLM3 config (optimum/exporters/openvino/model_configs.py)

Added SmolLM3OpenVINOConfig for OpenVINO export support:

  • Inherits from TextDecoderWithPositionIdsOnnxConfig (position_ids support)
  • Uses MistralDummyPastKeyValuesGenerator (GQA: 4 KV heads vs 16 attention heads)
  • Uses OVDecoderModelPatcher for SDPA patching

Without this change, export fails with:

sdpa_mask_without_vmap() missing 1 required positional argument: 'cache_position'\n```\n\n### 2. Fix position_ids double-append (`optimum/intel/openvino/modeling_decoder.py`)\n`_update_model_kwargs_for_generation()` was double-appending `position_ids` with transformers 5.x, shifting token positions during autoregressive decoding and causing generation drift.\n\nFixed by removing the extra append and relying on transformers `GenerationMixin` update logic.\n\n### 3. Regression test (`tests/openvino/test_decoder.py`)\nAdded test to verify `OVModelForCausalLM` does not mutate position_ids beyond the superclass result.\n\n## WWB Accuracy Results\n\n| Format | Score | Deviation | Status |\n|--------|-------|-----------|--------|\n| FP32 CPU | 1.0000 | 0.00% | ✅ PASS |\n| INT8 CPU | 0.9651 | 3.49% | ✅ PASS |\n| INT4 CPU | 0.9398 | 6.02% | ❌ FAIL (above 5% threshold) |\n\n- FP32 achieves perfect score after fix (was 12.67% deviation before)\n- INT4 quantization loss is inherent; not related to the decoder bug\n\n## Related\n- GitHub issue: intel-sandbox/applications.ai.openvino.meat#912\n- Jira: CVS-183434

Co-authored-by: hf-dependantbot-rollout[bot] <285970069+hf-dependantbot-rollout[bot]@users.noreply.github.com>
@mlukasze mlukasze added the experimental Experimental work in progress label Jun 1, 2026
@mlukasze mlukasze changed the title Add SmolLM3 support for OpenVINO export [EXPERIMENT][WIP] Add SmolLM3 support for OpenVINO export Jun 1, 2026
@mlukasze mlukasze added the do-not-merge Do not merge yet label Jun 1, 2026
…ition_ids double-append

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mlukasze

mlukasze commented Jun 2, 2026

Copy link
Copy Markdown
Owner Author

Superseded by upstream PR: huggingface#1761

@mlukasze mlukasze closed this Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge Do not merge yet experimental Experimental work in progress

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants