Skip to content

ObjectDetectionModule.md

Latest commit

 

History

History
414 lines (230 loc) · 26.2 KB

ObjectDetectionModule.md

File metadata and controls

414 lines (230 loc) · 26.2 KB

Class: ObjectDetectionModule<T>

Defined in: modules/computer_vision/ObjectDetectionModule.ts:86

Generic object detection module with type-safe label maps.

Extends

  • VisionLabeledModule<Detection<ResolveLabels<T>>[], ResolveLabels<T>>

Type Parameters

T

T extends ObjectDetectionModelName | LabelEnum

Either a built-in model name (e.g. 'ssdlite-320-mobilenet-v3-large') or a custom LabelEnum label map.

Properties

generateFromFrame()

generateFromFrame: (frameData, ...args) => any

Defined in: modules/BaseModule.ts:53

Process a camera frame directly for real-time inference.

This method is bound to a native JSI function after calling load(), making it worklet-compatible and safe to call from VisionCamera's frame processor thread.

Performance characteristics:

  • Zero-copy path: When using frame.getNativeBuffer() from VisionCamera v5, frame data is accessed directly without copying (fastest, recommended).
  • Copy path: When using frame.toArrayBuffer(), pixel data is copied from native to JS, then accessed from native code (slower, fallback).

Usage with VisionCamera:

const frameOutput = useFrameOutput({
  pixelFormat: 'rgb',
  onFrame(frame) {
    'worklet';
    // Zero-copy approach (recommended)
    const nativeBuffer = frame.getNativeBuffer();
    const result = model.generateFromFrame(
      { nativeBuffer: nativeBuffer.pointer, width: frame.width, height: frame.height },
      ...args
    );
    nativeBuffer.release();
    frame.dispose();
  }
});

Parameters

frameData

Frame

Frame data object with either nativeBuffer (zero-copy) or data (ArrayBuffer)

args

...any[]

Additional model-specific arguments (e.g., threshold, options)

Returns

any

Model-specific output (e.g., detections, classifications, embeddings)

See

Frame for frame data format details

Inherited from

VisionLabeledModule.generateFromFrame

labelMap

protected readonly labelMap: ResolveLabels

Defined in: modules/computer_vision/VisionLabeledModule.ts:42

Inherited from

VisionLabeledModule.labelMap

nativeModule

nativeModule: any = null

Defined in: modules/BaseModule.ts:16

Internal

Native module instance (JSI Host Object)

Inherited from

VisionLabeledModule.nativeModule

Accessors

runOnFrame

Get Signature

get runOnFrame(): (frame, isFrontCamera, options?) => Detection<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>[]

Defined in: modules/computer_vision/ObjectDetectionModule.ts:155

Override runOnFrame to provide an options-based API for VisionCamera integration.

Throws

If the underlying native worklet is unavailable (should not occur on a loaded module).

Returns

A worklet function for frame processing.

(frame, isFrontCamera, options?): Detection<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>[]

Parameters
frame

Frame

isFrontCamera

boolean

options?

ObjectDetectionOptions<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>

Returns

Detection<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>[]

Overrides

VisionLabeledModule.runOnFrame

Methods

delete()

delete(): void

Defined in: modules/BaseModule.ts:81

Unloads the model from memory and releases native resources.

Always call this method when you're done with a model to prevent memory leaks.

Returns

void

Inherited from

VisionLabeledModule.delete

forward()

forward(input, options?): Promise<Detection<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>[]>

Defined in: modules/computer_vision/ObjectDetectionModule.ts:247

Executes the model's forward pass to detect objects within the provided image.

Supports two input types:

  1. String path/URI: File path, URL, or Base64-encoded string
  2. PixelData: Raw pixel data from image libraries (e.g., NitroImage)

Parameters

input

A string image source (file path, URI, or Base64) or a PixelData object.

string | PixelData

options?

ObjectDetectionOptions<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>

Optional configuration for detection inference. Includes detectionThreshold, inputSize, and classesOfInterest.

Returns

Promise<Detection<ResolveLabels<T, { rf-detr-nano: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: { normMean: Triple<number>; normStd: Triple<number>; }; }; ssdlite-320-mobilenet-v3-large: { availableInputSizes: undefined; defaultDetectionThreshold: 0.7; defaultInputSize: undefined; defaultIouThreshold: 0.55; labelMap: typeof CocoLabel; preprocessorConfig: undefined; }; yolo26l: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26m: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26n: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26s: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; yolo26x: { availableInputSizes: readonly [384, 512, 640]; defaultDetectionThreshold: number; defaultInputSize: number; defaultIouThreshold: number; labelMap: typeof CocoLabelYolo; preprocessorConfig: undefined; }; }>>[]>

A Promise resolving to an array of Detection objects.

Throws

If the model is not loaded or if an invalid inputSize is provided.

Example

const detections = await model.forward('path/to/image.jpg', {
  detectionThreshold: 0.7,
  inputSize: 640,  // For YOLO models
  classesOfInterest: ['PERSON', 'CAR'],
});

Overrides

VisionLabeledModule.forward

forwardET()

protected forwardET(inputTensor): Promise<TensorPtr[]>

Defined in: modules/BaseModule.ts:62

Internal

Runs the model's forward method with the given input tensors. It returns the output tensors that mimic the structure of output from ExecuTorch.

Parameters

inputTensor

TensorPtr[]

Array of input tensors.

Returns

Promise<TensorPtr[]>

Array of output tensors.

Inherited from

VisionLabeledModule.forwardET

getAvailableInputSizes()

getAvailableInputSizes(): readonly number[] | undefined

Defined in: modules/computer_vision/ObjectDetectionModule.ts:146

Returns the available input sizes for this model, or undefined if the model accepts any size.

Returns

readonly number[] | undefined

An array of available input sizes, or undefined if not constrained.

Example

const sizes = model.getAvailableInputSizes(); // [384, 512, 640] for YOLO models, or undefined for RF-DETR

getInputShape()

getInputShape(methodName, index): Promise<number[]>

Defined in: modules/BaseModule.ts:72

Gets the input shape for a given method and index.

Parameters

methodName

string

method name

index

number

index of the argument which shape is requested

Returns

Promise<number[]>

The input shape as an array of numbers.

Inherited from

VisionLabeledModule.getInputShape

fromCustomModel()

static fromCustomModel<L>(modelSource, config, onDownloadProgress?): Promise<ObjectDetectionModule<L>>

Defined in: modules/computer_vision/ObjectDetectionModule.ts:340

Creates an object detection instance with a user-provided model binary and label map. Use this when working with a custom-exported model that is not one of the built-in presets. Internally uses 'custom' as the model name for telemetry unless overridden.

Required model contract

The .pte model binary must expose a single forward method with the following interface:

Input: one float32 tensor of shape [1, 3, H, W] — a single RGB image, values in [0, 1] after optional per-channel normalization (pixel − mean) / std. H and W are read from the model's declared input shape at load time.

Outputs: exactly three float32 tensors, in this order:

  1. Bounding boxes — flat [4·N] array of (x1, y1, x2, y2) coordinates in model-input pixel space, repeated for N detections.
  2. Confidence scores — flat [N] array of values in [0, 1].
  3. Class indices — flat [N] array of float32-encoded integer class indices (0-based, matching the order of entries in your labelMap).

Preprocessing (resize → normalize) and postprocessing (coordinate rescaling, threshold filtering, NMS) are handled by the native runtime — your model only needs to produce the raw detections above.

Type Parameters

L

L extends Readonly<Record<string, string | number>>

Parameters

modelSource

ResourceSource

A fetchable resource pointing to the model binary.

config

ObjectDetectionConfig<L>

A ObjectDetectionConfig object with the label map and optional preprocessing parameters.

onDownloadProgress?

(progress) => void

Optional callback to monitor download progress, receiving a value between 0 and 1.

Returns

Promise<ObjectDetectionModule<L>>

A Promise resolving to an ObjectDetectionModule instance typed to the provided label map.

fromModelName()

static fromModelName<C>(namedSources, onDownloadProgress?): Promise<ObjectDetectionModule<ModelNameOf<C>>>

Defined in: modules/computer_vision/ObjectDetectionModule.ts:106

Creates an object detection instance for a built-in model.

Type Parameters

C

C extends ObjectDetectionModelSources

Parameters

namedSources

C

A ObjectDetectionModelSources object specifying which model to load and where to fetch it from.

onDownloadProgress?

(progress) => void

Optional callback to monitor download progress, receiving a value between 0 and 1.

Returns

Promise<ObjectDetectionModule<ModelNameOf<C>>>

A Promise resolving to an ObjectDetectionModule instance typed to the chosen model's label map.