feat: implementation of multimodal runner by NorbertKlockiewicz · Pull Request #892 · software-mansion/react-native-executorch

NorbertKlockiewicz · 2026-03-02T09:12:48Z

Description

Adds vision/multimodal support to useLLM: load a VLM by passing capabilities: ['vision'], then use sendMessage(text, { imagePath }) to send messages with images. Under the hood this introduces a pluggable encoder architecture (IEncoder / VisionEncoder), a dedicated MultimodalRunner, and a refactored BaseLLMRunner with cleaner ownership and shared state. Also exposes getVisualTokenCount() JSI method for accurate token counting with images. No changes to the text-only path.

Introduces a breaking change?

Yes
No

Type of change

Bug fix (change which fixes an issue)
New feature (change which adds functionality)
Documentation update (improves or adds clarity to existing documentation)
Other (chores, tests, code style improvements etc.)

Tested on

iOS
Android

Testing instructions

Run the llm example app, select multimodal llm screen. Select an image and prompt the model.

Screenshots

Related issues

Checklist

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation accordingly
My changes generate no new warnings

Additional notes

NorbertKlockiewicz · 2026-03-10T09:50:40Z

Documentation
Tests
hugging face

API reference will be generated after the PR is approved

msluszniak · 2026-03-10T11:05:56Z

On huggingface, you can add information using which version of executorch was model exported.

chmjkb · 2026-03-10T11:54:03Z

  public async generate(
    messages: Message[],
-    tools?: LLMTool[]
+    tools?: LLMTool[],
+    imagePaths?: string[]


I'm not sure I understand this:
Why are we passing imagePaths if the Message type includes a mediaPath member? It seems like the user needs to pass the same things twice

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… EOS IDs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…g cache Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…run_tests.sh

…kenCount JSI Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… runner classes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ad image shape from model metadata Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…mage_token from config Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chmjkb

good job 🥳

Co-authored-by: Jakub Chmura <92989966+chmjkb@users.noreply.github.com>

benITo47

Great changes overall! Thanks!

NorbertKlockiewicz force-pushed the @nk/lfm-vlm branch from 3d22a30 to 0ac42b4 Compare March 2, 2026 09:15

NorbertKlockiewicz changed the title ~~feat: initial implementation of multimodal runner with lfm vlm~~ feat implementation of multimodal runner Mar 5, 2026

NorbertKlockiewicz marked this pull request as ready for review March 5, 2026 16:19

NorbertKlockiewicz requested review from benITo47, chmjkb and msluszniak March 5, 2026 16:20

msluszniak changed the title ~~feat implementation of multimodal runner~~ feat: implementation of multimodal runner Mar 5, 2026

msluszniak assigned NorbertKlockiewicz Mar 5, 2026

msluszniak added the feature PRs that implement a new feature label Mar 5, 2026

This was linked to issues Mar 6, 2026

Add support for LFM2.5-VL-1.6B #813

Closed

Add VLM support #552

Closed

NorbertKlockiewicz added this to the v0.8.0 milestone Mar 6, 2026

chmjkb removed this from the v0.8.0 milestone Mar 6, 2026

chmjkb requested changes Mar 6, 2026

View reviewed changes

benITo47 requested changes Mar 6, 2026

View reviewed changes

Comment thread packages/react-native-executorch/common/rnexecutorch/models/llm/LLM.cpp

benITo47 reviewed Mar 6, 2026

View reviewed changes

Comment thread packages/react-native-executorch/src/constants/modelUrls.ts Outdated

benITo47 reviewed Mar 6, 2026

View reviewed changes

Comment thread packages/react-native-executorch/common/rnexecutorch/models/llm/LLM.cpp

msluszniak requested changes Mar 9, 2026

View reviewed changes

NorbertKlockiewicz requested review from benITo47, chmjkb and msluszniak March 10, 2026 09:49

NorbertKlockiewicz force-pushed the @nk/lfm-vlm branch from 09954c9 to 8e8a304 Compare March 10, 2026 10:44

This comment was marked as outdated.

Sign in to view

chmjkb requested changes Mar 10, 2026

View reviewed changes

NorbertKlockiewicz force-pushed the @nk/lfm-vlm branch from 271100f to 0eee8b6 Compare March 10, 2026 13:30

msluszniak self-requested a review March 10, 2026 13:43

NorbertKlockiewicz requested a review from chmjkb March 10, 2026 13:59

msluszniak approved these changes Mar 10, 2026

View reviewed changes

NorbertKlockiewicz and others added 23 commits March 11, 2026 11:35

feat: add MultimodalRunner with plug-in encoder map

88e8443

feat: wire capabilities through LLM.cpp, delete UnifiedRunner

bddff5a

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat: forward capabilities from LLMController to native

6a86444

feat: add logging, fix metadata application, fix module ownership and…

60dbd0f

… EOS IDs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

refactor: replace Image class with ImagePath + VisionEncoder embeddin…

f489d45

…g cache Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test: add TextRunnerTests and VLMTests suites, register in CMake and …

21f5f59

…run_tests.sh

refactor: unify multimodal/text paths in sendMessage, add getVisualTo…

0790ea9

…kenCount JSI Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

refactor: replace example namespace with rnexecutorch::llm::runner in…

9cf417b

… runner classes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

refactor: collapse BaseLLMRunner constructor, deduplicate eos_ids, re…

f6d369d

…ad image shape from model metadata Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

refactor: comments etc.

84e0b65

fix: cap VLM generation tokens, propagate encoder load errors, pass i…

2517431

…mage_token from config Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

revert: remove TextRunnerTests and VLMTests suites

1acc7a0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

refactor: unify namespaces

caaa456

fix: address PR review comments for VLM support

7da0875

fix: use & instead of *

56778d7

fix: requested changes

47bfeaf

docs: write an instruction for using llm with vision capabilities

1bc22b0

chore: point to swmansion org on huggingface

c1b785d

tests: add tests for new runner

ac1ba44

feat: add missing changes to LLMModule

4a82046

feat: requested changes

20ce02a

fix: remove audioPath left after rebase

b0fe6d3

fix: comment, throw when no image tag

4c8b0ff

NorbertKlockiewicz force-pushed the @nk/lfm-vlm branch from 63f27b8 to 4c8b0ff Compare March 11, 2026 10:36

chmjkb approved these changes Mar 11, 2026

View reviewed changes

Comment thread packages/react-native-executorch/src/constants/modelUrls.ts Outdated

Update packages/react-native-executorch/src/constants/modelUrls.ts

6e68958

Co-authored-by: Jakub Chmura <92989966+chmjkb@users.noreply.github.com>

NorbertKlockiewicz enabled auto-merge (squash) March 11, 2026 11:09

benITo47 approved these changes Mar 11, 2026

View reviewed changes

NorbertKlockiewicz merged commit ce065d2 into main Mar 11, 2026
5 checks passed

NorbertKlockiewicz deleted the @nk/lfm-vlm branch March 11, 2026 11:12

Conversation

NorbertKlockiewicz commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Introduces a breaking change?

Type of change

Tested on

Testing instructions

Screenshots

Related issues

Checklist

Additional notes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NorbertKlockiewicz commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msluszniak commented Mar 10, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chmjkb Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

chmjkb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

benITo47 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

NorbertKlockiewicz commented Mar 2, 2026 •

edited

Loading

NorbertKlockiewicz commented Mar 10, 2026 •

edited

Loading