Skip to content

Commit e283275

Browse files
benITo47benITo47
committed
Add docs
1 parent 23b085d commit e283275

File tree

4 files changed

+260
-12
lines changed

4 files changed

+260
-12
lines changed

‎apps/computer-vision/components/ImageWithMasks.tsx‎

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -108,17 +108,24 @@ export default function ImageWithMasks({
108108
{instances.length > 0 && (
109109
<View style={styles.overlay}>
110110
<Canvas style={styles.canvas}>
111-
{maskImages.map((maskImg, idx) => (
112-
<SkiaImage
113-
key={`mask-${idx}`}
114-
image={maskImg}
115-
fit="contain"
116-
x={0}
117-
y={0}
118-
width={layout.width}
119-
height={layout.height}
120-
/>
121-
))}
111+
{maskImages.map((maskImg, idx) => {
112+
const inst = instances[idx];
113+
const mx = inst.bbox.x1 * scale + offsetX;
114+
const my = inst.bbox.y1 * scale + offsetY;
115+
const mw = (inst.bbox.x2 - inst.bbox.x1) * scale;
116+
const mh = (inst.bbox.y2 - inst.bbox.y1) * scale;
117+
return (
118+
<SkiaImage
119+
key={`mask-${idx}`}
120+
image={maskImg}
121+
fit="fill"
122+
x={mx}
123+
y={my}
124+
width={mw}
125+
height={mh}
126+
/>
127+
);
128+
})}
122129

123130
{instances.map((instance, idx) => {
124131
const color = INSTANCE_COLORS[idx % INSTANCE_COLORS.length];

‎docs/docs/02-benchmarks/memory-usage.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ Data presented in the following sections is based on inference with non-resized
9696
## Instance Segmentation
9797

9898
:::warning
99-
Data presented in the following sections is based on inference with forward_1024 method.
99+
Data presented in the following sections is based on inference with forward_640 method.
100100
:::
101101

102102
| Model | Android (XNNPACK) [MB] | iOS (XNNPACK) [MB] |

‎docs/docs/03-hooks/02-computer-vision/useInstanceSegmentation.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
---
2+
title: useInstanceSegmentation
3+
---
4+
5+
Instance segmentation is a computer vision technique that detects individual objects within an image and produces a per-pixel segmentation mask for each one. Unlike object detection (which only returns bounding boxes), instance segmentation provides precise object boundaries. React Native ExecuTorch offers a dedicated hook `useInstanceSegmentation` for this task.
6+
7+
:::warning
8+
It is recommended to use models provided by us, which are available at our [Hugging Face repository](https://huggingface.co/collections/software-mansion/instance-segmentation-68d0ea936cd0906843cbba7d).
9+
:::
10+
11+
## API Reference
12+
13+
- For detailed API Reference for `useInstanceSegmentation` see: [`useInstanceSegmentation` API Reference](../../06-api-reference/functions/useInstanceSegmentation.md).
14+
15+
## High Level Overview
16+
17+
```typescript
18+
import { useInstanceSegmentation } from 'react-native-executorch';
19+
20+
const model = useInstanceSegmentation({
21+
model: {
22+
modelName: 'yolo26n-seg',
23+
modelSource: 'https://huggingface.co/.../yolo26n-seg.pte',
24+
},
25+
});
26+
27+
const imageUri = 'file:///Users/.../photo.jpg';
28+
29+
try {
30+
const instances = await model.forward(imageUri);
31+
// instances is an array of SegmentedInstance objects
32+
} catch (error) {
33+
console.error(error);
34+
}
35+
```
36+
37+
### Arguments
38+
39+
`useInstanceSegmentation` takes [`InstanceSegmentationProps`](../../06-api-reference/interfaces/InstanceSegmentationProps.md) that consists of:
40+
41+
- `model` - An object containing:
42+
- `modelName` - The name of a built-in model. See [`InstanceSegmentationModelSources`](../../06-api-reference/interfaces/InstanceSegmentationProps.md) for the list of supported models.
43+
- `modelSource` - The location of the model binary (a URL or a bundled resource).
44+
- An optional flag [`preventLoad`](../../06-api-reference/interfaces/InstanceSegmentationProps.md#preventload) which prevents auto-loading of the model.
45+
46+
The hook is generic over the model config — TypeScript automatically infers the correct label type based on the `modelName` you provide. No explicit generic parameter is needed.
47+
48+
For more information on loading resources, take a look at [loading models](../../01-fundamentals/02-loading-models.md) page.
49+
50+
### Returns
51+
52+
`useInstanceSegmentation` returns an [`InstanceSegmentationType`](../../06-api-reference/interfaces/InstanceSegmentationType.md) object containing:
53+
54+
- `isReady` - Whether the model is loaded and ready to process images.
55+
- `isGenerating` - Whether the model is currently processing an image.
56+
- `error` - An error object if the model failed to load or encountered a runtime error.
57+
- `downloadProgress` - A value between 0 and 1 representing the download progress of the model binary.
58+
- `forward` - A function to run inference on an image.
59+
60+
## Running the model
61+
62+
To run the model, use the [`forward`](../../06-api-reference/interfaces/InstanceSegmentationType.md#forward) method. It accepts two arguments:
63+
64+
- `imageSource` (required) - The image to process. Can be a remote URL, a local file URI, or a base64-encoded image (whole URI or only raw base64).
65+
- `options` (optional) - An [`InstanceSegmentationOptions`](../../06-api-reference/interfaces/InstanceSegmentationOptions.md) object with the following fields:
66+
- `confidenceThreshold` - Minimum confidence score for including instances. Defaults to the model's configured threshold (typically `0.5`).
67+
- `iouThreshold` - IoU threshold for non-maximum suppression. Defaults to `0.5`.
68+
- `maxInstances` - Maximum number of instances to return. Defaults to `100`.
69+
- `classesOfInterest` - Filter results to include only specific classes (e.g. `['PERSON', 'CAR']`).
70+
- `returnMaskAtOriginalResolution` - Whether to resize masks to the original image resolution. Defaults to `true`.
71+
- `inputSize` - Input size for the model (e.g. `384`, `512`, `640`). Must be one of the model's available input sizes. If the model has only one forward method (i.e. no `availableInputSizes` configured), this option is not needed.
72+
73+
`forward` returns a promise resolving to an array of [`SegmentedInstance`](../../06-api-reference/interfaces/SegmentedInstance.md) objects, each containing:
74+
75+
- `bbox` - A [`Bbox`](../../06-api-reference/interfaces/Bbox.md) object with `x1`, `y1` (top-left corner) and `x2`, `y2` (bottom-right corner) coordinates in the original image's pixel space.
76+
- `label` - The class name of the detected instance, typed to the label map of the chosen model.
77+
- `score` - The confidence score of the detection, between 0 and 1.
78+
- `mask` - A `Uint8Array` binary mask (0 or 1) representing the instance's segmentation.
79+
- `maskWidth` - Width of the mask array.
80+
- `maskHeight` - Height of the mask array.
81+
- `instanceId` - Unique identifier for this instance.
82+
83+
## Example
84+
85+
```typescript
86+
import { useInstanceSegmentation } from 'react-native-executorch';
87+
88+
function App() {
89+
const model = useInstanceSegmentation({
90+
model: {
91+
modelName: 'yolo26n-seg',
92+
modelSource: 'https://huggingface.co/.../yolo26n-seg.pte',
93+
},
94+
});
95+
96+
const handleSegment = async () => {
97+
if (!model.isReady) return;
98+
99+
const imageUri = 'file:///Users/.../photo.jpg';
100+
101+
try {
102+
const instances = await model.forward(imageUri, {
103+
confidenceThreshold: 0.5,
104+
inputSize: 640,
105+
});
106+
107+
for (const instance of instances) {
108+
console.log('Label:', instance.label);
109+
console.log('Score:', instance.score);
110+
console.log('Bounding box:', instance.bbox);
111+
console.log('Mask size:', instance.maskWidth, 'x', instance.maskHeight);
112+
}
113+
} catch (error) {
114+
console.error(error);
115+
}
116+
};
117+
118+
// ...
119+
}
120+
```
121+
122+
## Supported models
123+
124+
| Model | Number of classes | Class list | Available input sizes |
125+
| ----------- | ----------------- | -------------------------------------------------------- | --------------------- |
126+
| yolo26n-seg | 80 | [COCO](../../06-api-reference/enumerations/CocoLabel.md) | 384, 512, 640 |
127+
| yolo26s-seg | 80 | [COCO](../../06-api-reference/enumerations/CocoLabel.md) | 384, 512, 640 |
128+
| yolo26m-seg | 80 | [COCO](../../06-api-reference/enumerations/CocoLabel.md) | 384, 512, 640 |
129+
| yolo26l-seg | 80 | [COCO](../../06-api-reference/enumerations/CocoLabel.md) | 384, 512, 640 |
130+
| yolo26x-seg | 80 | [COCO](../../06-api-reference/enumerations/CocoLabel.md) | 384, 512, 640 |

‎docs/docs/04-typescript-api/02-computer-vision/InstanceSegmentationModule.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
---
2+
title: InstanceSegmentationModule
3+
---
4+
5+
TypeScript API implementation of the [useInstanceSegmentation](../../03-hooks/02-computer-vision/useInstanceSegmentation.md) hook.
6+
7+
## API Reference
8+
9+
- For detailed API Reference for `InstanceSegmentationModule` see: [`InstanceSegmentationModule` API Reference](../../06-api-reference/classes/InstanceSegmentationModule.md).
10+
11+
## High Level Overview
12+
13+
```typescript
14+
import { InstanceSegmentationModule } from 'react-native-executorch';
15+
16+
const imageUri = 'path/to/image.png';
17+
18+
// Creating an instance from a built-in model
19+
const segmentation = await InstanceSegmentationModule.fromModelName({
20+
modelName: 'yolo26n-seg',
21+
modelSource: 'https://huggingface.co/.../yolo26n-seg.pte',
22+
});
23+
24+
// Running the model
25+
const instances = await segmentation.forward(imageUri);
26+
```
27+
28+
### Methods
29+
30+
All methods of `InstanceSegmentationModule` are explained in details here: [`InstanceSegmentationModule` API Reference](../../06-api-reference/classes/InstanceSegmentationModule.md)
31+
32+
## Loading the model
33+
34+
There are two ways to create an `InstanceSegmentationModule`:
35+
36+
### From a built-in model
37+
38+
Use [`fromModelName`](../../06-api-reference/classes/InstanceSegmentationModule.md#frommodelname) for pre-configured models. It accepts:
39+
40+
- `config` - An object containing:
41+
- `modelName` - The name of a built-in model (e.g. `'yolo26n-seg'`).
42+
- `modelSource` - Location of the model binary (a URL or a bundled resource).
43+
- `onDownloadProgress` (optional) - Callback to track download progress, receiving a value between 0 and 1.
44+
45+
```typescript
46+
const segmentation = await InstanceSegmentationModule.fromModelName({
47+
modelName: 'yolo26n-seg',
48+
modelSource: 'https://huggingface.co/.../yolo26n-seg.pte',
49+
});
50+
```
51+
52+
### From a custom config
53+
54+
Use [`fromCustomConfig`](../../06-api-reference/classes/InstanceSegmentationModule.md#fromcustomconfig) for custom-exported models with your own label map. It accepts:
55+
56+
- `modelSource` - Location of the model binary.
57+
- `config` - An [`InstanceSegmentationConfig`](../../06-api-reference/interfaces/InstanceSegmentationConfig.md) object with:
58+
- `labelMap` - An enum-like object mapping class names to indices.
59+
- `preprocessorConfig` (optional) - Normalization parameters (`normMean`, `normStd`).
60+
- `postprocessorConfig` (optional) - Postprocessing settings (`applyNMS`).
61+
- `defaultConfidenceThreshold` (optional) - Default confidence threshold.
62+
- `defaultIouThreshold` (optional) - Default IoU threshold.
63+
- `availableInputSizes` and `defaultInputSize` (optional) - Supported input sizes and the default.
64+
- `onDownloadProgress` (optional) - Callback to track download progress.
65+
66+
```typescript
67+
const MyLabels = { GRAPE_GREEN: 0, GRAPE_RED: 1, LEAF: 2 } as const;
68+
69+
const segmentation = await InstanceSegmentationModule.fromCustomConfig(
70+
'https://huggingface.co/.../grape_seg.pte',
71+
{
72+
labelMap: MyLabels,
73+
availableInputSizes: [640],
74+
defaultInputSize: 640,
75+
defaultConfidenceThreshold: 0.4,
76+
postprocessorConfig: { applyNMS: true },
77+
}
78+
);
79+
```
80+
81+
For more information on loading resources, take a look at [loading models](../../01-fundamentals/02-loading-models.md) page.
82+
83+
## Custom model output contract
84+
85+
If you want to use a custom-exported model, it must conform to the following output contract:
86+
87+
The model must produce **3 output tensors**:
88+
89+
| Tensor | Shape | Description |
90+
| ----------- | -------------- | --------------------------------------------------------------- |
91+
| bboxes | `[1, N, 4]` | Bounding boxes as `[x1, y1, x2, y2]` in model input coordinates |
92+
| scores | `[1, N, 2]` | `[max_score, class_id]` — scores must be post-sigmoid |
93+
| mask_logits | `[1, N, H, W]` | Per-detection binary mask logits — pre-sigmoid |
94+
95+
### Method naming convention
96+
97+
- If the model supports **multiple input sizes**, it must expose separate methods named `forward_{inputSize}` (e.g. `forward_384`, `forward_512`, `forward_640`).
98+
- If the model supports **only one input size**, it should expose a single `forward` method.
99+
100+
## Running the model
101+
102+
To run the model, use the [`forward`](../../06-api-reference/classes/InstanceSegmentationModule.md#forward) method. It accepts two arguments:
103+
104+
- `imageSource` (required) - The image to process. Can be a remote URL, a local file URI, or a base64-encoded image (whole URI or only raw base64).
105+
- `options` (optional) - An [`InstanceSegmentationOptions`](../../06-api-reference/interfaces/InstanceSegmentationOptions.md) object for configuring the segmentation (confidence threshold, IoU threshold, input size, classes of interest, etc.).
106+
107+
The method returns a promise resolving to an array of [`SegmentedInstance`](../../06-api-reference/interfaces/SegmentedInstance.md) objects. Each object contains bounding box coordinates, a binary segmentation mask, the label of the detected instance, and the confidence score.
108+
109+
## Managing memory
110+
111+
The module is a regular JavaScript object, and as such its lifespan will be managed by the garbage collector. In most cases this should be enough, and you should not worry about freeing the memory of the module yourself, but in some cases you may want to release the memory occupied by the module before the garbage collector steps in. In this case use the method [`delete`](../../06-api-reference/classes/InstanceSegmentationModule.md#delete) on the module object you will no longer use, and want to remove from the memory. Note that you cannot use [`forward`](../../06-api-reference/classes/InstanceSegmentationModule.md#forward) after [`delete`](../../06-api-reference/classes/InstanceSegmentationModule.md#delete) unless you load the module again.

0 commit comments

Comments
 (0)