Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ For more information on loading resources, take a look at [loading models](../..
- `error` - An error object if the model failed to load or encountered a runtime error.
- `downloadProgress` - A value between 0 and 1 representing the download progress of the model binary.
- `forward` - A function to run inference on an image.
- `getAvailableInputSizes` - Returns the available input sizes for the loaded model, or `undefined` if the model accepts only a single input size. Use this to populate UI controls for selecting the input resolution.
- `runOnFrame` - A synchronous worklet function for real-time VisionCamera frame processing. Returns `null` until the model is ready. See [VisionCamera Integration](./visioncamera-integration.md) for usage.
Comment thread
msluszniak marked this conversation as resolved.
Outdated

## Running the model

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ import { useModuleFactory } from '../useModuleFactory';
* React hook for managing an Instance Segmentation model instance.
* @typeParam C - A {@link InstanceSegmentationModelSources} config specifying which model to load.
* @param props - Configuration object containing `model` config and optional `preventLoad` flag.
* @returns An object with model state (`error`, `isReady`, `isGenerating`, `downloadProgress`) and a typed `forward` function.
* @returns An object with model state (`error`, `isReady`, `isGenerating`, `downloadProgress`), a typed `forward` function, `getAvailableInputSizes` helper, and a `runOnFrame` worklet for VisionCamera integration.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

im fine with this to unblock this PR but I think this comment doesn't add much value - it's just restating what the code already expresses.

* @example
* ```ts
* const { isReady, isGenerating, forward, error, downloadProgress } =
* const { isReady, isGenerating, forward, error, downloadProgress, getAvailableInputSizes, runOnFrame } =
* useInstanceSegmentation({
* model: {
* modelName: 'yolo26n-seg',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -276,15 +276,14 @@ export class InstanceSegmentationModule<
* Override runOnFrame to add label mapping for VisionCamera integration.
* The parent's runOnFrame returns raw native results with class indices;
* this override maps them to label strings and provides an options-based API.
* @returns A worklet function for VisionCamera frame processing, or null if the model is not loaded.
* @returns A worklet function for VisionCamera frame processing.
* @throws {RnExecutorchError} If the underlying native worklet is unavailable (should not occur on a loaded module).
*/
override get runOnFrame():
| ((
frame: Frame,
isFrontCamera: boolean,
options?: InstanceSegmentationOptions<ResolveLabels<T>>
) => SegmentedInstance<ResolveLabels<T>>[])
| null {
override get runOnFrame(): (
frame: Frame,
isFrontCamera: boolean,
options?: InstanceSegmentationOptions<ResolveLabels<T>>
) => SegmentedInstance<ResolveLabels<T>>[] {
const baseRunOnFrame = super.runOnFrame;
if (!baseRunOnFrame) {
throw new RnExecutorchError(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -144,15 +144,14 @@ export class ObjectDetectionModule<

/**
* Override runOnFrame to provide an options-based API for VisionCamera integration.
* @returns A worklet function for frame processing, or null if the model is not loaded.
* @returns A worklet function for frame processing.
* @throws {RnExecutorchError} If the underlying native worklet is unavailable (should not occur on a loaded module).
*/
override get runOnFrame():
| ((
frame: any,
isFrontCamera: boolean,
options?: ObjectDetectionOptions<ResolveLabels<T>>
) => Detection<ResolveLabels<T>>[])
| null {
override get runOnFrame(): (
frame: any,
Comment thread
msluszniak marked this conversation as resolved.
Outdated
isFrontCamera: boolean,
options?: ObjectDetectionOptions<ResolveLabels<T>>
) => Detection<ResolveLabels<T>>[] {
const baseRunOnFrame = super.runOnFrame;
if (!baseRunOnFrame) {
throw new RnExecutorchError(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ export abstract class VisionModule<TOutput> extends BaseModule {
/**
* Synchronous worklet function for real-time VisionCamera frame processing.
*
* Only available after the model is loaded. Returns null if not loaded.
* Only available after the model is loaded.
*
* **Use this for VisionCamera frame processing in worklets.**
* For async processing, use `forward()` instead.
Expand All @@ -55,9 +55,10 @@ export abstract class VisionModule<TOutput> extends BaseModule {
* }
* });
* ```
* @returns A worklet function for frame processing, or null if the model is not loaded.
* @returns A worklet function for frame processing.
* @throws {RnExecutorchError} If the model is not loaded.
*/
get runOnFrame(): ((frame: Frame, ...args: any[]) => TOutput) | null {
get runOnFrame(): (frame: Frame, ...args: any[]) => TOutput {
if (!this.nativeModule) {
throw new RnExecutorchError(
RnExecutorchErrorCode.ModuleNotLoaded,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,9 @@ export interface InstanceSegmentationType<L extends LabelEnum> {
* **Use this for VisionCamera frame processing in worklets.**
* For async processing, use `forward()` instead.
*
* Available after model is loaded (`isReady: true`).
* `null` until the model is ready (`isReady: true`). The property itself is
* `null` when the model has not loaded yet — the function always returns an
* array (never `null`) once called.
* @param frame - VisionCamera Frame object
* @param isFrontCamera - Whether the front camera is active (for mirroring correction).
* @param options - Optional configuration for the segmentation process.
Expand Down
8 changes: 4 additions & 4 deletions packages/react-native-executorch/src/types/objectDetection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ export { CocoLabel };
/**
* Represents a bounding box for a detected object in an image.
* @category Types
* @property {number} x1 - The x-coordinate of the bottom-left corner of the bounding box.
* @property {number} y1 - The y-coordinate of the bottom-left corner of the bounding box.
* @property {number} x2 - The x-coordinate of the top-right corner of the bounding box.
* @property {number} y2 - The y-coordinate of the top-right corner of the bounding box.
* @property {number} x1 - The x-coordinate of the top-left corner of the bounding box.
* @property {number} y1 - The y-coordinate of the top-left corner of the bounding box.
* @property {number} x2 - The x-coordinate of the bottom-right corner of the bounding box.
* @property {number} y2 - The y-coordinate of the bottom-right corner of the bounding box.
*/
export interface Bbox {
x1: number;
Expand Down
Loading