Skip to content

Navigation Menu

Appearance settings

View all features
- BY COMPANY SIZE
  Enterprises
  Small and medium teams
  Startups
  Nonprofits
- BY USE CASE
  App Modernization
  DevSecOps
  DevOps
  CI/CD
  View all use cases
- BY INDUSTRY
  Healthcare
  Financial services
  Manufacturing
  Government
  View all industries
View all solutions
- EXPLORE BY TOPIC
  AI
  Software Development
  DevOps
  Security
  View all topics
- EXPLORE BY TYPE
  Customer stories
  Events & webinars
  Ebooks & reports
  Business insights
  GitHub Skills
- SUPPORT & SERVICES
  Documentation
  Customer support
  Community forum
  Trust center
  Partners
View all resources
- COMMUNITY
  GitHub SponsorsFund open source developers
- PROGRAMS
  Security Lab
  Maintainer Community
  Accelerator
  GitHub Stars
  Archive Program
- REPOSITORIES
  Topics
  Trending
  Collections
- ENTERPRISE SOLUTIONS
  Enterprise platformAI-powered developer platform
- AVAILABLE ADD-ONS
  GitHub Advanced SecurityEnterprise-grade security features
  Copilot for BusinessEnterprise-grade AI features
  Premium SupportEnterprise-grade 24/7 support
Pricing

Search code, repositories, users, issues, pull requests...

Search

Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Saved searches

Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our documentation.

Appearance settings

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

software-mansion / react-native-executorch Public

Notifications You must be signed in to change notification settings
Fork 69
Star 1.4k

Code
Issues 57
Pull requests 11
Discussions
Actions
Projects
Security and quality
Insights

Additional navigation options

Code
Issues
Pull requests
Discussions
Actions
Projects
Security and quality
Insights

feat: implementation of multimodal runner#892

Merged

NorbertKlockiewicz merged 58 commits intomainsoftware-mansion/react-native-executorch:mainfrom

@nk/lfm-vlmsoftware-mansion/react-native-executorch:@nk/lfm-vlmCopy head branch name to clipboard

Mar 11, 2026

Conversation Commits58 (58)Checks Files changed

Merged

feat: implementation of multimodal runner#892
NorbertKlockiewicz merged 58 commits intomainfrom
@nk/lfm-vlm

Commits

Commits on Mar 11, 2026

feat: initial implementation of multimodal runner with lfm vlm
NorbertKlockiewicz
committed
feat: unified LLM runner for text-only and multimodal PTEs

NorbertKlockiewicz
and
claude
committed
feat: add conversational VLM demo with multimodal/text-only support and fix token generation bugs

NorbertKlockiewicz
and
claude
committed
fix: default UnifiedRunner temperature to 0.8 and topp to 0.9
NorbertKlockiewicz
committed
feat: add NativeMessage struct and JSI conversion for message history
NorbertKlockiewicz
committed
feat: declare generateMultimodal on LLM and register JSI binding
NorbertKlockiewicz
committed
fix: remove redundant unordered_map and vector includes from LLM.h
NorbertKlockiewicz
committed
feat: implement generateMultimodal with per-turn chat template and image cache

NorbertKlockiewicz
and
claude
committed
feat: add mediaPath to Message, remove sendMessageWithImage from LLMType
NorbertKlockiewicz
committed
feat: replace sendMessageWithImage with sendMessage(msg, mediaPath?) using generateMultimodal

NorbertKlockiewicz
and
claude
committed
fix: use updatedHistory for multimodal routing, remove redundant reset before generateMultimodal
NorbertKlockiewicz
committed
fix: skip system messages in generateMultimodal, clear imageUri after send
NorbertKlockiewicz
committed
feat: show image thumbnail in user message bubble when mediaPath is set
NorbertKlockiewicz
committed
fix: use resizeMode contain so full image is always visible in message bubble
NorbertKlockiewicz
committed
refactor: derive isMultimodal from load param, unify load branches, remove tokenizerConfig guard

NorbertKlockiewicz
and
claude
committed
refactor: remove isMultimodal flag, inline generateMultimodal into sendMessage

NorbertKlockiewicz
and
claude
committed
fix: make tokenizerConfigSource required throughout load pipeline

NorbertKlockiewicz
and
claude
committed
fix: prepend system prompt to multimodal history before generateMultimodal

NorbertKlockiewicz
and
claude
committed
refactor: unify generate — Jinja renders prompt+<image> tokens in JS, C++ splits on placeholder

NorbertKlockiewicz
and
claude
committed
fix: collect imagePaths from messageHistoryWithPrompt, not full history

NorbertKlockiewicz
and
claude
committed
fix: typing
NorbertKlockiewicz
committed
feat: correctly calculate image tokens
NorbertKlockiewicz
committed
fix: add missing import
NorbertKlockiewicz
committed
fix: fall back to max_seq_len when model doesn't export max_context_len
NorbertKlockiewicz
committed
fix: address code review — error on image/placeholder mismatch, remove double reset, fix max_context_len fallback, require tokenizerConfigSource, pass tools in multimodal branch, capture callback b…
NorbertKlockiewicz
committed
feat: dynamic sendMessage type based on flag
NorbertKlockiewicz
committed
fix: model stopping generation in the middle of its answer
NorbertKlockiewicz
committed
feat: add LLMCapability type and parameterize LLMTypeMultimodal
NorbertKlockiewicz
committed
feat: update sendMessage to accept typed media object
NorbertKlockiewicz
committed
feat: add LFM2_VL_1_6B and LFM2_VL_1_6B_QUANTIZED model constants
NorbertKlockiewicz
committed
feat: add IEncoder interface and VisionEncoder

NorbertKlockiewicz
and
claude
committed
fix: address vision_encoder quality review issues
NorbertKlockiewicz
committed
feat: add BaseLLMRunner with shared state and load()

NorbertKlockiewicz
and
claude
committed
feat: add TextRunner

NorbertKlockiewicz
and
claude
committed
feat: add MultimodalRunner with plug-in encoder map
NorbertKlockiewicz
committed
feat: wire capabilities through LLM.cpp, delete UnifiedRunner

NorbertKlockiewicz
and
claude
committed
feat: forward capabilities from LLMController to native
NorbertKlockiewicz
committed
feat: add logging, fix metadata application, fix module ownership and EOS IDs

NorbertKlockiewicz
and
claude
committed
refactor: replace Image class with ImagePath + VisionEncoder embedding cache

NorbertKlockiewicz
and
claude
committed
test: add TextRunnerTests and VLMTests suites, register in CMake and run_tests.sh
NorbertKlockiewicz
committed
refactor: unify multimodal/text paths in sendMessage, add getVisualTokenCount JSI

NorbertKlockiewicz
and
claude
committed
refactor: replace example namespace with rnexecutorch::llm::runner in runner classes

NorbertKlockiewicz
and
claude
committed
refactor: collapse BaseLLMRunner constructor, deduplicate eos_ids, read image shape from model metadata

NorbertKlockiewicz
and
claude
committed
refactor: comments etc.
NorbertKlockiewicz
committed
fix: cap VLM generation tokens, propagate encoder load errors, pass image_token from config

NorbertKlockiewicz
and
claude
committed
revert: remove TextRunnerTests and VLMTests suites

NorbertKlockiewicz
and
claude
committed
refactor: unify namespaces
NorbertKlockiewicz
committed
fix: address PR review comments for VLM support
NorbertKlockiewicz
committed
fix: use & instead of *
NorbertKlockiewicz
committed
fix: requested changes
NorbertKlockiewicz
committed
docs: write an instruction for using llm with vision capabilities
NorbertKlockiewicz
committed
chore: point to swmansion org on huggingface
NorbertKlockiewicz
committed
tests: add tests for new runner
NorbertKlockiewicz
committed
feat: add missing changes to LLMModule
NorbertKlockiewicz
committed
feat: requested changes
NorbertKlockiewicz
committed
fix: remove audioPath left after rebase
NorbertKlockiewicz
committed
fix: comment, throw when no image tag
NorbertKlockiewicz
committed
Update packages/react-native-executorch/src/constants/modelUrls.ts

NorbertKlockiewicz
and
chmjkb
authored

Footer

© 2026 GitHub, Inc.

Footer navigation

Terms
Privacy
Security
Status
Community
Docs
Contact

You can’t perform that action at this time.