react-native-litert-lm

High-performance on-device LLM inference for React Native, powered by LiteRT-LM and Nitro Modules. Optimized for Gemma 4 and other on-device language models.

Features

🚀 Native Swift Bridge (iOS) — Bypasses Swift actor deadlocks (User Rule #1) via direct C FFI dispatched on a serial dev.litert.engine background queue.
🤖 Stateless Kotlin Bridge (Android) — Fully conforms to HybridLiteRTLMSpec using direct JSI memory access.
⚡ Zero-Copy Multimodal API — Native-owned ArrayBuffer mapping straight to FFI inputs for image/audio data without copy overhead (complying with User Rule #2).
🧠 Speculative Decoding — Active multi-token prediction support with pre-flight model capability validation.
🛠️ Function / Tool Calling — Native JSON-encoded schema specification support for structured outputs.
🏎️ GPU Acceleration — Metal (iOS), OpenCL GPU delegate (Android, Pixel devices).
🔄 Streaming Support — Non-blocking token-by-token callbacks.
📊 Real Memory Tracking — OS-level memory metrics (RSS, native heap, available memory) via native APIs (complying with User Rule #3).
📥 Automatic Model Download — Downloads models from URL with progress tracking and local caching.

Demo

Gemma 4 E2B running on-device on a Samsung Galaxy S22 (Snapdragon 8 Gen 1, 4 GB RAM) — CPU backend, streaming inference.

demo.mp4

Installation

npm install react-native-litert-lm react-native-nitro-modules

Expo

Add to your app.json:

{
  "expo": {
    "plugins": ["react-native-litert-lm"],
    "android": {
      "minSdkVersion": 26
    }
  }
}

Then create a development build:

npx expo prebuild
npx expo run:android  # Android
npx expo run:ios      # iOS

Note: Only ARM devices/simulators are supported. x86_64 Android emulators are not supported.

Bare React Native

# Android
cd android && ./gradlew clean

# iOS
cd ios && pod install

Example App

The example/ directory contains a fully functional test app with a dark-themed diagnostic UI that demonstrates:

Model downloading with progress tracking
Text inference (blocking and streaming)
Multi-turn conversation with context retention
Performance benchmarking (tokens/sec, latency)
Real-time memory tracking
Speculative decoding & tool calling settings toggles
Zero-copy multimodal inference loading images/audio directly into ArrayBuffers

Running the Example

Build the library (compiles TypeScript to lib/):
```
npm run build
```
Install example dependencies:
```
cd example
npm install
```

Create a development build and run:

npx expo prebuild --clean
npx expo run:android  # Android
npx expo run:ios      # iOS (pre-linked with CLiteRTLM.xcframework)

Note: If you change native code (Swift/Kotlin), you must run npx expo prebuild --clean again before rebuilding.

Model Management

LiteRT-LM models (like Gemma 4) are large files (1–4 GB) and cannot be bundled into your app binary. They are downloaded at runtime.

Automatic Downloading

Pass an HTTPS URL to useModel() or loadModel() — the library handles the rest:

Progress tracking — real-time download percentage via callbacks
Local caching — downloaded models are cached and reused across app launches
- Android: files/models/ (app-private)
- iOS: Library/Caches/litert_models/ (survives app relaunch; reclaimable by iOS under storage pressure)
HTTPS enforcement — only secure URLs are accepted

Manual Downloading

If you need custom control over downloads (e.g., authentication headers for private model hosting, resumable downloads, or custom caching), use your preferred HTTP client and pass the local file path:

import { fetch } from "expo/fetch";
import { File, Paths } from "expo-file-system";
import { useModel } from "react-native-litert-lm";

const MODEL_URL = "https://example.com/private-model.litertlm";

// Download with custom headers using expo/fetch
const response = await fetch(MODEL_URL, {
  headers: { Authorization: `Bearer ${token}` },
});
const modelFile = new File(Paths.cache, "my-model.litertlm");
modelFile.write(await response.bytes());

// Pass the local path — no download occurs
const { model, isReady } = useModel(modelFile.uri, { backend: "cpu" });

Usage

React Hook (Recommended)

The useModel hook manages the full model lifecycle: downloading, loading, inference, and cleanup.

import { useModel, GEMMA_4_E2B_IT } from "react-native-litert-lm";
import { Platform } from "react-native";

function App() {
  const {
    model,
    isReady,
    downloadProgress,
    error,
    load,          // Manually trigger load
    deleteModel,   // Delete cached model file
    memorySummary, // Auto-updated memory stats (if tracking enabled)
  } = useModel(GEMMA_4_E2B_IT, {
    backend: 'cpu',
    autoLoad: true, // Default: true. Set false to load manually via load().
    systemPrompt: "You are a helpful assistant.",
    enableMemoryTracking: true,
  });

  if (!isReady) {
    return <Text>Loading... {Math.round(downloadProgress * 100)}%</Text>;
  }

  const generate = async () => {
    const response = await model.sendMessage("Hello!");
    console.log(response);
  };

  return <Button title="Generate" onPress={generate} />;
}

Manual Usage

import { createLLM } from "react-native-litert-lm";

const llm = createLLM();

// Load a model from URL (auto-downloads) or local path
await llm.loadModel("https://example.com/model.litertlm", {
  backend: "gpu",
  systemPrompt: "You are a helpful assistant.",
});

// Generate a response
const response = await llm.sendMessage("What is the capital of France?");
console.log(response);

// Clean up
llm.close();

Streaming Generation

llm.sendMessageAsync("Tell me a story", (token, done) => {
  process.stdout.write(token);
  if (done) console.log("\n--- Done ---");
});

Multimodal (Image / Audio) & Zero-Copy Buffers

Multimodal features are fully supported via standard file paths or high-performance zero-copy ArrayBuffer objects:

1. Zero-Copy Multimodal Messages (Recommended)

This API uses Nitro Modules' native-backed ArrayBuffer directly mapped to native memory buffers, avoiding any base64 heap copying overhead (User Rule #2):

import { checkMultimodalSupport } from "react-native-litert-lm";

const warning = checkMultimodalSupport();
if (warning) {
  console.warn(warning); // Experimental or unsupported on current platform (e.g. iOS simulator)
} else {
  // Read local assets or files straight into ArrayBuffers using fetch
  const response = await fetch(Image.resolveAssetSource(require("./test.jpeg")).uri);
  const imageBuffer = await response.arrayBuffer();

  const reply = await llm.sendMultimodalMessage([
    { type: "image", imageBuffer },
    { type: "text", text: "Describe what is in this image." }
  ]);
  console.log(reply);
}

2. Path-Based Multimodal Messages

// Image input
const response = await llm.sendMessageWithImage(
  "What's in this image?",
  "/path/to/image.jpg",
);

// Audio input
const transcription = await llm.sendMessageWithAudio(
  "Transcribe this audio",
  "/path/to/audio.wav",
);

Speculative Decoding & Tools

1. Speculative Decoding (MTP)

Enable speculative decoding in LLMConfig to accelerate inference using multi-token prediction when supported by your model:

const { model } = useModel(GEMMA_4_E2B_IT, {
  enableSpeculativeDecoding: true,
});

2. Function / Tool Calling

Inject tools as an array of definitions, specifying parameter validation using standard JSON schema format:

const { model } = useModel(GEMMA_4_E2B_IT, {
  tools: [
    {
      name: "get_current_weather",
      description: "Get the current weather for a location",
      parametersJson: JSON.stringify({
        type: "object",
        properties: {
          location: { type: "string", description: "The city and state, e.g. San Francisco, CA" },
          unit: { type: "string", enum: ["celsius", "fahrenheit"] }
        },
        required: ["location"]
      })
    }
  ]
});

Performance Stats

const stats = llm.getStats();
console.log(`Generated ${stats.completionTokens} tokens`);
console.log(`Speed: ${stats.tokensPerSecond.toFixed(1)} tokens/sec`);
console.log(`Time to first token: ${stats.timeToFirstToken.toFixed(0)} ms`);

Note: Stats are available for both sync (sendMessage) and streaming (sendMessageAsync) on both platforms. iOS uses real benchmark data from the C API; Android uses heuristic token counts with precise timing.

Memory Tracking

The library provides real OS-level memory data — no estimation. It reads directly from mach_task_basic_info (iOS) and Debug.getNativeHeapAllocatedSize() + /proc/self/status (Android).

Direct Memory Query

const usage = llm.getMemoryUsage();
console.log(
  `Native heap: ${(usage.nativeHeapBytes / 1024 / 1024).toFixed(1)} MB`,
);
console.log(`RSS: ${(usage.residentBytes / 1024 / 1024).toFixed(1)} MB`);
console.log(
  `Available: ${(usage.availableMemoryBytes / 1024 / 1024).toFixed(1)} MB`,
);
console.log(`Low memory: ${usage.isLowMemory}`);

Automatic Tracking with Native Buffers

Enable memory tracking to automatically record snapshots in a native-backed ArrayBuffer after every inference call:

const llm = createLLM({
  enableMemoryTracking: true,
  maxMemorySnapshots: 256,
});

await llm.loadModel("/path/to/model.litertlm", { backend: "cpu" });
await llm.sendMessage("Hello!");

const summary = llm.memoryTracker!.getSummary();
console.log(
  `Peak RSS: ${(summary.peakResidentBytes / 1024 / 1024).toFixed(1)} MB`,
);
console.log(
  `RSS Delta: ${(summary.residentDeltaBytes / 1024 / 1024).toFixed(1)} MB`,
);

Using `useModel` with Memory Tracking

const { model, isReady, memorySummary } = useModel(modelUrl, {
  enableMemoryTracking: true,
  maxMemorySnapshots: 100,
});

// memorySummary auto-updates after each inference call
if (memorySummary) {
  console.log(`Current RSS: ${memorySummary.currentResidentBytes}`);
  console.log(`Peak RSS: ${memorySummary.peakResidentBytes}`);
}

Standalone Memory Tracker

import {
  createMemoryTracker,
  createNativeBuffer,
} from "react-native-litert-lm";

const tracker = createMemoryTracker(100);

tracker.record({
  timestamp: Date.now(),
  nativeHeapBytes: 50_000_000,
  residentBytes: 200_000_000,
  availableMemoryBytes: 4_000_000_000,
});

// Access the underlying native buffer (zero-copy transfer to native code)
const buffer = tracker.getNativeBuffer();

Supported Models

All exported model URLs are public — no authentication required. Pass them directly to useModel() or loadModel() for automatic downloading with progress tracking and local caching.

Constant	Model	Size	Min RAM	Source
`GEMMA_4_E2B_IT`	Gemma 4 E2B (Multimodal, IT)	2.58 GB	4 GB+	HuggingFace
`GEMMA_4_E4B_IT`	Gemma 4 E4B (Higher Quality)	3.65 GB	6 GB+	HuggingFace
`GEMMA_3N_E2B_IT_INT4`	Gemma 3n E2B (Int4, Multimodal)	~1.3 GB	4 GB+	litert.dev

Recommended: Use GEMMA_4_E2B_IT for most use cases — multimodal (text + vision + audio) and the best quality-to-size ratio.

iOS Note: Models larger than ~2 GB require the com.apple.developer.kernel.extended-virtual-addressing entitlement. See iOS Entitlements below. Gemma 3n E2B (~1.3 GB) works without it.

Other compatible models (download .litertlm files manually from HuggingFace):

Model	Size	Min RAM	Notes
Gemma 3 1B	~1 GB	4 GB+	Smallest, fastest
Phi-4 Mini	~2 GB	4 GB+	Microsoft's small LLM
Qwen 2.5 1.5B	~1.5 GB	4 GB+	Multilingual

API Reference

`createLLM(options?): LiteRTLM`

Creates a new LLM inference engine instance.

options.enableMemoryTracking — enable automatic memory snapshot recording
options.maxMemorySnapshots — max number of snapshots to retain (default: 256)

`loadModel(path, config?): Promise<void>`

Loads a model from a local path or HTTPS URL.

Parameter	Type	Default	Description
`path`	`string`	—	Absolute path to `.litertlm` or HTTPS URL
`config.backend`	`string`	`'cpu'`	`'cpu'`, `'gpu'`, or `'npu'`
`config.systemPrompt`	`string`	—	System prompt for the model
`config.temperature`	`number`	`0.7`	Sampling temperature
`config.topK`	`number`	`40`	Top-K sampling
`config.topP`	`number`	`0.95`	Top-P (nucleus) sampling
`config.maxTokens`	`number`	`1024`	Maximum generation length

Backend Options

Backend	Engine	Speed	Notes
`'cpu'`	CPU inference	Slowest	Always available on all devices
`'gpu'`	Metal (iOS) / OpenCL (Android)	Fast	iOS: always available. Android: requires OpenCL (Pixel only, not Samsung/Qualcomm)
`'npu'`	NPU / Neural Engine	Fastest	Requires supported hardware; experimental

iOS: Both 'cpu' and 'gpu' (Metal) are supported. The engine automatically tries fallback backend combinations if the primary one fails.

Android GPU: The GPU backend requires OpenCL, which is not available on most Samsung and Qualcomm devices. Use checkBackendSupport('gpu') to check before loading. The engine will throw a clear error if GPU is unsupported.

`sendMessage(message): Promise<string>`

Runs inference synchronously on a background thread. Returns the complete response.

`sendMessageAsync(message, callback)`

Streaming generation. Callback signature: (token: string, isDone: boolean) => void.

`sendMessageWithImage(message, imagePath): Promise<string>`

Send a message with an image (for vision models like Gemma 4 E2B).

`sendMessageWithAudio(message, audioPath): Promise<string>`

Send a message with audio (for audio-capable models like Gemma 4 E2B).

`getStats(): GenerationStats`

Returns performance metrics from the last inference call.

`getMemoryUsage(): MemoryUsage`

Returns real OS-level memory usage.

`getHistory(): Message[]`

Returns the conversation history.

`resetConversation()`

Clears conversation context and starts a fresh session.

`close()`

Releases all native resources. Call when the model is no longer needed.

`deleteModel(fileName): Promise<void>`

Deletes a cached model file from the app's local storage.

Utility Functions

import {
  checkBackendSupport,
  checkMultimodalSupport,
  getRecommendedBackend,
} from "react-native-litert-lm";

// Check if GPU is supported on this device
const gpuWarning = checkBackendSupport("gpu");

// Check NPU support
const npuWarning = checkBackendSupport("npu"); // string | undefined

// Check multimodal support
const mmError = checkMultimodalSupport(); // string | undefined

// Get recommended backend
const backend = getRecommendedBackend(); // 'cpu'

Requirements

Dependency	Version
React Native	0.76+
react-native-nitro-modules	0.35.0+
Android API	26+ (ARM64)
iOS	15.0+ (ARM64)
LiteRT-LM Engine	0.12.0

Platform Support

Platform	Status	Architecture	Backends
Android	✅ Ready	arm64-v8a	CPU (all devices), GPU (OpenCL devices only), NPU
iOS	✅ Ready	arm64	CPU, GPU (Metal — always available)

iOS Feature Matrix

Feature	Status	Notes
Text inference (blocking)	✅	Direct FFI using `dev.litert.engine` background queue
Text inference (streaming)	✅	Token-by-token callbacks
CPU inference	✅	Safe fallback default
GPU inference (Metal/MPS)	✅	Supported via `backend: 'gpu'`
Model download with progress	✅	URLSession-based, cached in `Caches/`
Memory tracking	✅	Real-time Resident Set Size (RSS) tracking
Multi-turn conversation	✅	Context retained across turns
Multimodal (image/audio)	✅	Zero-copy `ArrayBuffer` mapping to FFI input buffers
Speculative Decoding	✅	Dynamic capabilities check during model pre-load
Function / Tool Calling	✅	Supported via JSON-encoded schema specification

iOS Entitlements

Models larger than ~2 GB (like Gemma 4 E2B at 2.58 GB) require the Extended Virtual Addressing entitlement on iOS physical devices. Without it, iOS limits virtual memory to ~2 GB and the app will be killed by Jetsam.

Add to your app's .entitlements file:

<key>com.apple.developer.kernel.extended-virtual-addressing</key>
<true/>

Note: This entitlement requires a paid Apple Developer account ($99/year). Gemma 3n E2B (~1.3 GB) works without it.

iOS FFI Architecture & Integration

The library uses a highly optimized Swift Direct-FFI bridge that links directly with the pre-compiled C library CLiteRTLM.xcframework.

Key Design Commitments

JSI Thread Safety (User Rule #1):
- The JSI/JS thread must never be blocked by native synchronous lock-waiting operations.
- We dispatch all FFI calls to a serial background dev.litert.engine queue, executing callbacks asynchronously to prevent deadlocking JSI execution.
Zero-Copy Memory Pipelines (User Rule #2):
- Enforce the use of Nitro Modules' ArrayBuffer directly referencing native memory pointers (ArrayBuffer.data) when processing heavy media assets like images or audio.
Manual FFI Resource Management (User Rule #3):
- Raw pointers (LiteRtLmEngine*, LiteRtLmConversation*) are manually allocated and strictly deallocated inside Swift deinit and close() destructors to guarantee 0% memory leaks during prolonged inference sessions.

Architecture Topology

┌──────────────────────────────────────────────────────────┐
│  React Native (TypeScript / JavaScript)                  │
├──────────────────────────────────────────────────────────┤
│  Nitro Modules JSI Bindings (`HybridLiteRTLMSpec`)       │
├─────────────────────────────┬────────────────────────────┤
│  Android (Kotlin)           │  iOS (Swift Direct FFI)    │
│  `HybridLiteRTLM.kt`        │  `HybridLiteRTLM.swift`    │
│  `litertlm-android` AAR     │  `CLiteRTLM.xcframework`   │
└─────────────────────────────┴────────────────────────────┘

Android Bridging

Conforms fully to HybridLiteRTLMSpec using Kotlin.
Incorporates Proguard keep rules to prevent dynamic JSI/JNI code stripping.
Declares <uses-native-library android:name="libOpenCL.so" android:required="false" /> to load dynamic OpenCL for GPU delegate acceleration on Android 12+ without throwing platform installer exceptions.

iOS Bridging

Entirely written in native Swift (HybridLiteRTLM.swift) calling direct FFI.
Avoids the upstream Swift SDK actor lock-blocking deadlocks by utilizing low-level C functions directly.
Implements custom getMemoryUsage that queries the OS directly via mach_task_basic_info to get precise real-time Resident Set Size (RSS) metrics.

Testing

The library includes a comprehensive multi-tier unit testing suite designed to run quickly on host machines (CI runners or local development environments) without requiring a physical test device.

1. JavaScript / TypeScript Layer (Jest)

The JS/TS layer uses Jest to validate the useModel hook, download progress callbacks, URL query scrubbing, file storage helpers, and the zero-copy native memory tracker buffer allocations.

Setup & Mocking: Includes an active stub (src/__mocks__/react-native-nitro-modules.ts) that mocks the Nitro Modules HybridObject architecture.
How to run:
```
npm run test
```

2. Android Kotlin Layer (Robolectric)

The Android layer uses local JUnit Robolectric tests to run Android code on the JVM, sandboxing OS dependencies. It validates HTTPS schema constraints, path traversal mitigations, and initial telemetry states.

Setup & Mocking: Uses a local shadow Promise implementation to test thread-asynchronous errors.

How to run:

cd example/android
./gradlew :react-native-litert-lm:testDebugUnitTest

3. iOS Swift Layer (XCTest)

The iOS layer leverages native XCTests integrated directly into CocoaPods via standard development test specs. It verifies FFI path traversal blocking, non-HTTPS download blocks, automatic deinit cleanup, and Mach-based telemetry bounds.

How to run:

Boot your preferred iOS simulator (e.g., iPhone 16 running iOS 18.6).

Run the tests using xcodebuild:

cd example/ios
xcodebuild test -workspace LLMTest.xcworkspace -scheme react-native-litert-lm-Unit-Tests -sdk iphonesimulator -destination 'platform=iOS Simulator,name=iPhone 16'

Security & Sanitization Protections Checked

Every test run automatically asserts:

Defense in depth for download boundaries: Blocks non-HTTPS schemes at both JS model factory and low-level native layers.
Path Traversal protections: Prevents directory traversal attacks (.., /, \) in download and deletion APIs.
Telemetry sanity: Ensures zero-leak memory usage telemetry boundaries stay strictly linear.

License

The code in this repository is licensed under the MIT License.

⚠️ AI Model Disclaimer

This library is an execution engine for on-device LLMs. The AI models themselves are not distributed with this package and have their own licenses:

Gemma (Google): Gemma Terms of Use
Llama 3 (Meta): Llama 3.2 Community License
Qwen (Alibaba): Apache 2.0
Phi (Microsoft): MIT License

By downloading and using these models, you agree to their respective licenses and acceptable use policies. The author of react-native-litert-lm takes no responsibility for model outputs or applications built with them.

Name		Name	Last commit message	Last commit date
Latest commit History 43 Commits
android		android
build		build
coverage		coverage
cpp		cpp
example		example
ios		ios
scripts		scripts
src		src
.gitignore		.gitignore
.npmignore		.npmignore
.npmrc		.npmrc
.release-it.json		.release-it.json
LICENSE		LICENSE
README.md		README.md
app.plugin.js		app.plugin.js
jest.config.js		jest.config.js
nitro.json		nitro.json
package-lock.json		package-lock.json
package.json		package.json
react-native-litert-lm.podspec		react-native-litert-lm.podspec
react-native.config.js		react-native.config.js
tsconfig.json		tsconfig.json

Folders and files

Latest commit

History

Repository files navigation

react-native-litert-lm

Features

Demo

Installation

Expo

Bare React Native

Example App

Running the Example

Model Management

Automatic Downloading

Manual Downloading

Usage

React Hook (Recommended)

Manual Usage

Streaming Generation

Multimodal (Image / Audio) & Zero-Copy Buffers

1. Zero-Copy Multimodal Messages (Recommended)

2. Path-Based Multimodal Messages

Speculative Decoding & Tools

1. Speculative Decoding (MTP)

2. Function / Tool Calling

Performance Stats

Memory Tracking

Direct Memory Query

Automatic Tracking with Native Buffers

Using useModel with Memory Tracking

Standalone Memory Tracker

Supported Models

API Reference

createLLM(options?): LiteRTLM

loadModel(path, config?): Promise<void>

Backend Options

sendMessage(message): Promise<string>

sendMessageAsync(message, callback)

sendMessageWithImage(message, imagePath): Promise<string>

sendMessageWithAudio(message, audioPath): Promise<string>

getStats(): GenerationStats

getMemoryUsage(): MemoryUsage

getHistory(): Message[]

resetConversation()

close()

deleteModel(fileName): Promise<void>

Utility Functions

Requirements

Platform Support

iOS Feature Matrix

iOS Entitlements

iOS FFI Architecture & Integration

Key Design Commitments

Architecture Topology

Android Bridging

iOS Bridging

Testing

1. JavaScript / TypeScript Layer (Jest)

2. Android Kotlin Layer (Robolectric)

3. iOS Swift Layer (XCTest)

Security & Sanitization Protections Checked

License

⚠️ AI Model Disclaimer

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 14

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Using `useModel` with Memory Tracking

`createLLM(options?): LiteRTLM`

`loadModel(path, config?): Promise<void>`

`sendMessage(message): Promise<string>`

`sendMessageAsync(message, callback)`

`sendMessageWithImage(message, imagePath): Promise<string>`

`sendMessageWithAudio(message, audioPath): Promise<string>`

`getStats(): GenerationStats`

`getMemoryUsage(): MemoryUsage`

`getHistory(): Message[]`

`resetConversation()`

`close()`

`deleteModel(fileName): Promise<void>`

Packages