add HyperClovaX Vision by jp1924 · Pull Request #44314 · huggingface/transformers

jp1924 · 2026-02-27T02:01:28Z

What does this PR do?

Hello, Transformers team!

I submitted a PR to add naver-hyperclovax/HyperCLOVAX-SEED-Think-32B (hereafter HCX), developed by the Korean IT company Naver while executing the government's national AI model project.

The HCX code was written based on Transformer 4.52.4, leading to the following issues:

Being based on an outdated Transformer model prevents the application of the latest training optimization techniques supported by Transformer 5.0.0 (e.g., sequence parallelism).
The use of some deprecated code or features may cause unexpected bugs in the latest Transformer version.
The modeling code was overly complex, reducing debugging and development convenience. Additionally, experimental code used during model creation remained untouched.

Moving to Transformer 5.0.0 significantly improved the readability and development convenience of the modeling code. We aim to leverage this to add the HCX model to transformers.

TODO list

Add docstrings

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@zucchini-nlp @yonigozlan @molbap

jp1924 · 2026-02-27T02:06:37Z

When I ran the test in my environment, hcx test ran normally without any failures.
The process for adding a model will likely proceed like this:

First, fix all the comments you guys added in the modeling part.
(Optional) Add modular code.
Once the modeling code work is nearly complete, write the docstrings.
Then apply the style.

I think the work will proceed in this manner?

jp1924 · 2026-02-27T07:33:48Z

Hmm... In my environment, torch_compilable_check passes normally, but it fails in this test. I think I need to look into this issue further.

…iteration and using cache

…tion file

…mapping

…orConditionalGeneration

…d processor classes

…template application

…ing in HyperClovaXProcessor

…on classes

ArthurZucker · 2026-03-02T09:19:45Z

Yep don't worry we'll fix this one on main!

zucchini-nlp

Hey @jp1924, thanks a lot for the PR!

I think we need to move all the code to modular file first, since there is a lot of copied stuff from other models. I left you comments below about where exactly can copy from. Ping me for another review when the modular is ready

…clarity

…deo processor classes.

… to HCXVisionConfig and remove unnecessary code.

…XVisionV2Config

zucchini-nlp · 2026-04-10T10:22:04Z

BTW, I didn't forget about this PR, coming to it right after the hyperclovax lm is merged :)

jp1924 · 2026-04-10T11:38:19Z

@zucchini-nlp @bigshanedogg
Sounds good — doc-strings and tests are all done on my end. I was just putting together a summary of what changed to make the review easier. Let me know when it's merged!

…h sequence classification support

zucchini-nlp · 2026-05-07T18:32:56Z

The text backbone was merged just today, so we can resume this PR now. We'd need to use its own lm backbone and the vision backbone stays as discussed earlier

Whenever you have time @jp1924 feel free to rebase, no rush :)

github-actions · 2026-05-11T07:59:11Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, hyperclovax_vision_v2, qwen2_5_vl

github-actions · 2026-05-11T08:19:24Z

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=44314&sha=23a8be

jp1924 · 2026-05-11T08:24:01Z

@zucchini-nlp @bigshanedogg Could you please take a look for a code review? I'm getting an AssertionError: tensor(False) is not true : Found non-zero attention weights for padding token at batch 1, sequence position 0 in paligemma, but it doesn't seem to be related to this PR

zucchini-nlp

Niiice. giving an early approval

Some nits to fix, especially since we added Exaone4.5 which seems to be the best candidate to copy from in modular. And we can iterate with a core maintainter of the rest

zucchini-nlp · 2026-05-11T09:48:38Z

+
+# HyperCLOVAX Vision V2
+
+HyperCLOVAX Vision V2 is a multimodal vision-language model developed by NAVER. It combines the HyperCLOVAX language model backbone — based on the [Granite](./granite) architecture with optional post-norm (Peri-LN) layers for MuP scaling — with a [Qwen2.5-VL](./qwen2_5_vl) vision encoder. The model supports text, image, and video inputs and is capable of chain-of-thought reasoning via built-in thinking tokens (`<think>...</think>`).


imo we can say that the lm is HypexClovax now

zucchini-nlp · 2026-05-11T09:49:10Z

+You can find the original HyperCLOVAX-SEED-Think-32B checkpoint on the [naver-hyperclovax/HyperCLOVAX-SEED-Think-32B](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Think-32B) page.
+
+> [!TIP]
+> The `model_type` in the released checkpoint's `config.json` is `"vlm"`, while the Transformers implementation registers this model as `"hyperclovax_vision_v2"`. Due to this mismatch, loading via `AutoModel` or `AutoModelForCausalLM` is not supported. Use the model class directly as shown in the examples below.


lets see if we can do anything to this with Naver team before merging

zucchini-nlp · 2026-05-11T09:49:44Z

+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    attn_implementation="sdpa",


nit, delete dtype and attn_implementation, they are loaded from config

zucchini-nlp · 2026-05-11T09:51:54Z

    "deepseek_ocr2",
    "fuyu",
    "h2ovl_chat",
-    "hyperclovax_vlm",


shouldn't we keep the entry with 'vlm'?

zucchini-nlp · 2026-05-11T09:52:57Z

+            model_type = self.vision_config.get("model_type", "qwen2_5_vl_vision")
+            model_type = "qwen2_5_vl_vision" if model_type == "qwen2_5_vl" else model_type


tiny comment that this is for BC/any other reason would be nice

zucchini-nlp · 2026-05-11T10:02:47Z

+@auto_docstring
+class HCXVisionV2ForConditionalGeneration(HCXVisionV2PreTrainedModel, GenerationMixin):
+    accepts_loss_kwargs = False
+    _tied_weights_keys = {"lm_head.weight": "model.language_model.embed_tokens.weight"}


same here, copy from exaone and i beleve we can delete identical or similar methods

zucchini-nlp · 2026-05-11T10:03:40Z

+            WeightRenaming(r"^model.mm_projector", "model.projector"),
+            WeightRenaming(r"^model.language_model.model.layers", "model.language_model.layers"),
+            WeightRenaming(r"^model.language_model.model.embed_tokens", "model.language_model.embed_tokens"),
+            WeightRenaming(r"^model.language_model.model.norm", "model.language_model.norm"),


or model\.language_model\.model: model.language_model ?

zucchini-nlp · 2026-05-11T10:05:15Z

+            self.assertIsNotNone(outputs)
+
+    def test_reverse_loading_mapping(self, check_keys_were_modified=True):
+        # Conversion happens only for the `ConditionalGeneration` model, not the base model


I think this was fixed last Friday and we allow adding conversion per class now, rebasing should help

zucchini-nlp · 2026-05-11T10:05:52Z

+    @unittest.skip("Loading nested configs with overwritten `kwargs` isn't supported yet, FIXME @raushan.")
+    def test_load_with_mismatched_shapes(self):
+        pass
+


Fixed maybe as well, I merged smth else last Friday 😆

zucchini-nlp · 2026-05-11T10:06:15Z

+@require_vision
+@require_torch
+@require_torchvision
+class HCXVisionV2ProcessorTest(ProcessorTesterMixin, unittest.TestCase):
+    processor_class = HCXVisionV2Processor


not needed, the whole file. We will use the same processor as exaone

jp1924 added 3 commits February 27, 2026 01:44

add: hcx vision model

647ac9d

add: modeling test

f420da6

Merge branch 'main' into feat/hcx-seed-32b

3aeac2e

jp1924 changed the title ~~add HyperCLOVAX Vision~~ add HyperClovaX Vision Feb 27, 2026

jp1924 added 3 commits February 27, 2026 04:32

fix: update vision model selection logic for HCXVisionModel

c3f0231

fix: prepare_inputs_for_generation to handle first iteration logic

d231dff

fix: IndentationError:

e67eaf7

jp1924 added 12 commits March 1, 2026 19:31

Merge branch 'main' into feat/hcx-seed-32b

a33087b

fix: lint

26ad483

fix: update conditional checks for model type comparisons

7df5a6d

fix: set pixel_values and pixel_values_videos to None when not first …

8638c8b

…iteration and using cache

fix: remove unnecessary coding declaration from HyperClovaX configura…

f459f54

…tion file

fix: streamline imports in test_modeling_hyperclovax_vision.py

2fff572

fix: add hyperclovax model type to SPECIAL_MODEL_TYPE_TO_MODULE_NAME …

0212ef5

…mapping

fix: add docstrings for image and video grid dimensions in HCXVisionF…

eb222b7

…orConditionalGeneration

fix: update docstrings for parameters in HyperClovaX configuration an…

781c956

…d processor classes

test: add unit tests for HyperClovaXProcessor functionality and chat …

c59d586

…template application

fix: update image processing logic and enhance multimodal token handl…

21816a2

…ing in HyperClovaXProcessor

fix: update parameter types and docstrings in HyperClovaX configurati…

843f314

…on classes

zucchini-nlp reviewed Mar 2, 2026

View reviewed changes

jp1924 added 7 commits March 3, 2026 09:42

Merge branch 'main' into feat/hcx-seed-32b

bd62dfa

Merge branch 'main' into feat/hcx-seed-32b

05ff2e9

'models/auto' change

a509a21

remove image processor

82b81f0

Refactor HyperClovaX configuration classes and update type hints for …

ab921aa

…clarity

Change the class name to HCXVisionV2Processor and update the image/vi…

6add4b7

…deo processor classes.

Refactor HCXVisionModel and related classes: rename HyperClovaXConfig…

23040f4

… to HCXVisionConfig and remove unnecessary code.

jp1924 added 7 commits April 6, 2026 16:32

fix: update config type in HCXVisionV2ForSequenceClassification to HC…

2589d01

…XVisionV2Config

config & processor to modular

3d9662d

Merge branch 'main' into feat/hcx-seed-32b

bb5769f

fix test-error

361d35f

Merge branch 'main' into feat/hcx-seed-32b

8d873c3

update processor test

ef7b049

remove unused import

8f085c8

jp1924 added 5 commits April 10, 2026 20:38

Merge branch 'main' into feat/hcx-seed-32b

9000231

fix: syntax error

c2bc144

feat: add forward method for HCXVisionV2ForSequenceClassification wit…

83748fb

…h sequence classification support

fix date

dbea6b5

Merge branch 'main' into feat/hcx-seed-32b

9437fb8

This was referenced Apr 29, 2026

Cumulative feature and defect updates from recent Transformers PRs evalstate/transformers#42

Open

Cumulative defect fixes from recent Transformers PRs evalstate/transformers#43

Open

bigshanedogg mentioned this pull request May 7, 2026

Add HyperCLOVAX SEED Think 14B #44956

Merged

6 tasks

jp1924 added 5 commits May 11, 2026 14:28

Merge branch 'main' into feat/hcx-seed-32b

7fb344a

remove hypercloax & fix test error

41b1a00

Merge branch 'main' into feat/hcx-seed-32b

a56fd5c

fix image_newline failures

01c4964

fix test_transformers_specific_model_import failures

3db1711

jp1924 added 2 commits May 11, 2026 16:59

rm hyperclovax docs

dcbc59e

refactor: simplify import statements in configuration_auto.py

23a8be5

jp1924 requested a review from zucchini-nlp May 11, 2026 08:24

zucchini-nlp approved these changes May 11, 2026

View reviewed changes


		# HyperCLOVAX Vision V2

		HyperCLOVAX Vision V2 is a multimodal vision-language model developed by NAVER. It combines the HyperCLOVAX language model backbone — based on the [Granite](./granite) architecture with optional post-norm (Peri-LN) layers for MuP scaling — with a [Qwen2.5-VL](./qwen2_5_vl) vision encoder. The model supports text, image, and video inputs and is capable of chain-of-thought reasoning via built-in thinking tokens (`<think>...</think>`).

		model_type = self.vision_config.get("model_type", "qwen2_5_vl_vision")
		model_type = "qwen2_5_vl_vision" if model_type == "qwen2_5_vl" else model_type

Conversation

jp1924 commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

TODO list

Before submitting

Who can review?

Uh oh!

jp1924 commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jp1924 commented Feb 27, 2026

Uh oh!

ArthurZucker commented Mar 2, 2026

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zucchini-nlp commented Apr 10, 2026

Uh oh!

jp1924 commented Apr 10, 2026

Uh oh!

zucchini-nlp commented May 7, 2026

Uh oh!

github-actions Bot commented May 11, 2026

Uh oh!

github-actions Bot commented May 11, 2026

Uh oh!

jp1924 commented May 11, 2026

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jp1924 commented Feb 27, 2026 •

edited

Loading

jp1924 commented Feb 27, 2026 •

edited

Loading