Skip to content

[EXPERIMENT][WIP] Add SmolLM3 support for OpenVINO export and fix position_ids double-append#1761

Closed
mlukasze wants to merge 1 commit into
huggingface:mainfrom
mlukasze:enable/HuggingFaceTB-SmolLM3-3B
Closed

[EXPERIMENT][WIP] Add SmolLM3 support for OpenVINO export and fix position_ids double-append#1761
mlukasze wants to merge 1 commit into
huggingface:mainfrom
mlukasze:enable/HuggingFaceTB-SmolLM3-3B

Conversation

@mlukasze

@mlukasze mlukasze commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

⚠️ AUTOMATICALLY GENERATED BY OMEGA AGENT — REQUIRES HUMAN REVIEW ⚠️
This PR was created by an AI agent as part of automated model enablement.
A human maintainer must review and approve it before it can be considered for merge.
Do NOT merge without human review and sign-off.

Summary

This PR adds OpenVINO export and inference support for SmolLM3 (HuggingFaceTB/SmolLM3-3B), and fixes a bug in OVModelForCausalLM that caused generation drift with recent transformers (>=5.x).

Changes

1. Add SmolLM3 config (optimum/exporters/openvino/model_configs.py)

Added SmolLM3OpenVINOConfig for OpenVINO export support:

  • Inherits from TextDecoderWithPositionIdsOnnxConfig (position_ids support)
  • Uses MistralDummyPastKeyValuesGenerator (GQA: 4 KV heads vs 16 attention heads)
  • Uses OVDecoderModelPatcher for SDPA patching

Without this change, export fails with:

sdpa_mask_without_vmap() missing 1 required positional argument: 'cache_position'

2. Fix position_ids double-append (optimum/intel/openvino/modeling_decoder.py)

_update_model_kwargs_for_generation() was double-appending position_ids with transformers 5.x, shifting token positions during autoregressive decoding and causing generation drift.

Fixed by removing the extra append and relying on transformers GenerationMixin update logic.

3. Regression test (tests/openvino/test_decoder.py)

Added test to verify OVModelForCausalLM does not mutate position_ids beyond the superclass result.

4. Export coverage and docs

  • Extended the existing OpenVINO exporter test with a SmolLM3 text-generation-with-past case
  • Added the tiny SmolLM3 test model to tests/openvino/utils_tests.py
  • Updated the OpenVINO supported models documentation

WWB Accuracy Results

Format Score Deviation Status
FP32 CPU 1.0000 0.00% ✅ PASS
INT8 CPU 0.9651 3.49% ✅ PASS
INT4 CPU 0.9398 6.02% ❌ FAIL (above 5% threshold)
  • FP32 achieves perfect score after fix (was 12.67% deviation before)
  • INT4 quantization loss is inherent; not related to the decoder bug

…ition_ids double-append

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mlukasze

Copy link
Copy Markdown
Contributor Author

product-grade PR has been merged here: #1715

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants