Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/computer-vision/components/ImageWithMasks.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ function createMaskImage(
{
width: dstW,
height: dstH,
alphaType: AlphaType.Premul,
alphaType: AlphaType.Unpremul,
colorType: ColorType.RGBA_8888,
},
data,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ For more information on loading resources, take a look at [loading models](../..
- `error` - An error object if the model failed to load or encountered a runtime error.
- `downloadProgress` - A value between 0 and 1 representing the download progress of the model binary.
- `forward` - A function to run inference on an image.
- `getAvailableInputSizes` - Returns the available input sizes for the loaded model, or `undefined` if the model accepts only a single input size. Use this to populate UI controls for selecting the input resolution.
- `runOnFrame` - A synchronous worklet function for real-time VisionCamera frame processing. Returns `null` until the model is ready. See [VisionCamera Integration](./visioncamera-integration.md) for usage.
Comment thread
msluszniak marked this conversation as resolved.
Outdated

## Running the model

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ import { useModuleFactory } from '../useModuleFactory';
* React hook for managing an Instance Segmentation model instance.
* @typeParam C - A {@link InstanceSegmentationModelSources} config specifying which model to load.
* @param props - Configuration object containing `model` config and optional `preventLoad` flag.
* @returns An object with model state (`error`, `isReady`, `isGenerating`, `downloadProgress`) and a typed `forward` function.
* @returns An object with model state (`error`, `isReady`, `isGenerating`, `downloadProgress`), a typed `forward` function, `getAvailableInputSizes` helper, and a `runOnFrame` worklet for VisionCamera integration.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

im fine with this to unblock this PR but I think this comment doesn't add much value - it's just restating what the code already expresses.

* @example
* ```ts
* const { isReady, isGenerating, forward, error, downloadProgress } =
* const { isReady, isGenerating, forward, error, downloadProgress, getAvailableInputSizes, runOnFrame } =
* useInstanceSegmentation({
* model: {
* modelName: 'yolo26n-seg',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -276,15 +276,14 @@ export class InstanceSegmentationModule<
* Override runOnFrame to add label mapping for VisionCamera integration.
* The parent's runOnFrame returns raw native results with class indices;
* this override maps them to label strings and provides an options-based API.
* @returns A worklet function for VisionCamera frame processing, or null if the model is not loaded.
* @returns A worklet function for VisionCamera frame processing.
* @throws {RnExecutorchError} If the underlying native worklet is unavailable (should not occur on a loaded module).
*/
override get runOnFrame():
| ((
frame: Frame,
isFrontCamera: boolean,
options?: InstanceSegmentationOptions<ResolveLabels<T>>
) => SegmentedInstance<ResolveLabels<T>>[])
| null {
override get runOnFrame(): (
frame: Frame,
isFrontCamera: boolean,
options?: InstanceSegmentationOptions<ResolveLabels<T>>
) => SegmentedInstance<ResolveLabels<T>>[] {
const baseRunOnFrame = super.runOnFrame;
if (!baseRunOnFrame) {
throw new RnExecutorchError(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -144,15 +144,14 @@ export class ObjectDetectionModule<

/**
* Override runOnFrame to provide an options-based API for VisionCamera integration.
* @returns A worklet function for frame processing, or null if the model is not loaded.
* @returns A worklet function for frame processing.
* @throws {RnExecutorchError} If the underlying native worklet is unavailable (should not occur on a loaded module).
*/
override get runOnFrame():
| ((
frame: any,
isFrontCamera: boolean,
options?: ObjectDetectionOptions<ResolveLabels<T>>
) => Detection<ResolveLabels<T>>[])
| null {
override get runOnFrame(): (
frame: any,
Comment thread
msluszniak marked this conversation as resolved.
Outdated
isFrontCamera: boolean,
options?: ObjectDetectionOptions<ResolveLabels<T>>
) => Detection<ResolveLabels<T>>[] {
const baseRunOnFrame = super.runOnFrame;
if (!baseRunOnFrame) {
throw new RnExecutorchError(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ export abstract class VisionModule<TOutput> extends BaseModule {
/**
* Synchronous worklet function for real-time VisionCamera frame processing.
*
* Only available after the model is loaded. Returns null if not loaded.
* Only available after the model is loaded.
*
* **Use this for VisionCamera frame processing in worklets.**
* For async processing, use `forward()` instead.
Expand All @@ -55,9 +55,10 @@ export abstract class VisionModule<TOutput> extends BaseModule {
* }
* });
* ```
* @returns A worklet function for frame processing, or null if the model is not loaded.
* @returns A worklet function for frame processing.
* @throws {RnExecutorchError} If the model is not loaded.
*/
get runOnFrame(): ((frame: Frame, ...args: any[]) => TOutput) | null {
get runOnFrame(): (frame: Frame, ...args: any[]) => TOutput {
if (!this.nativeModule) {
throw new RnExecutorchError(
RnExecutorchErrorCode.ModuleNotLoaded,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,9 @@ export interface InstanceSegmentationType<L extends LabelEnum> {
* **Use this for VisionCamera frame processing in worklets.**
* For async processing, use `forward()` instead.
*
* Available after model is loaded (`isReady: true`).
* `null` until the model is ready (`isReady: true`). The property itself is
* `null` when the model has not loaded yet — the function always returns an
* array (never `null`) once called.
* @param frame - VisionCamera Frame object
* @param isFrontCamera - Whether the front camera is active (for mirroring correction).
* @param options - Optional configuration for the segmentation process.
Expand Down
8 changes: 4 additions & 4 deletions packages/react-native-executorch/src/types/objectDetection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ export { CocoLabel };
/**
* Represents a bounding box for a detected object in an image.
* @category Types
* @property {number} x1 - The x-coordinate of the bottom-left corner of the bounding box.
* @property {number} y1 - The y-coordinate of the bottom-left corner of the bounding box.
* @property {number} x2 - The x-coordinate of the top-right corner of the bounding box.
* @property {number} y2 - The y-coordinate of the top-right corner of the bounding box.
* @property {number} x1 - The x-coordinate of the top-left corner of the bounding box.
* @property {number} y1 - The y-coordinate of the top-left corner of the bounding box.
* @property {number} x2 - The x-coordinate of the bottom-right corner of the bounding box.
* @property {number} y2 - The y-coordinate of the bottom-right corner of the bounding box.
*/
export interface Bbox {
x1: number;
Expand Down
Loading