Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions apps/computer-vision/app/_layout.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,10 @@ export default function _layout() {
}}
/>
<Drawer.Screen
name="image_segmentation/index"
name="semantic_segmentation/index"
options={{
drawerLabel: 'Image Segmentation',
title: 'Image Segmentation',
drawerLabel: 'Semantic Segmentation',
title: 'Semantic Segmentation',
headerTitleStyle: { color: ColorPalette.primary },
}}
/>
Expand Down
4 changes: 2 additions & 2 deletions apps/computer-vision/app/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ export default function Home() {
</TouchableOpacity>
<TouchableOpacity
style={styles.button}
onPress={() => router.navigate('image_segmentation/')}
onPress={() => router.navigate('semantic_segmentation/')}
>
<Text style={styles.buttonText}>Image Segmentation</Text>
<Text style={styles.buttonText}>Semantic Segmentation</Text>
</TouchableOpacity>
<TouchableOpacity
style={styles.button}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { BottomBar } from '../../components/BottomBar';
import { getImage } from '../../utils';
import {
DEEPLAB_V3_RESNET50,
useImageSegmentation,
useSemanticSegmentation,
} from 'react-native-executorch';
import {
Canvas,
Expand Down Expand Up @@ -42,10 +42,10 @@ const numberToColor: number[][] = [
[162, 51, 255], // 20 Amethyst
];

export default function ImageSegmentationScreen() {
export default function SemanticSegmentationScreen() {
const { setGlobalGenerating } = useContext(GeneratingContext);
const { isReady, isGenerating, downloadProgress, forward } =
useImageSegmentation({
useSemanticSegmentation({
model: DEEPLAB_V3_RESNET50,
});
const [imageUri, setImageUri] = useState('');
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/02-benchmarks/inference-time.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ Benchmark times for text embeddings are highly dependent on the sentence length.
Image embedding benchmark times are measured using 224×224 pixel images, as required by the model. All input images, whether larger or smaller, are resized to 224×224 before processing. Resizing is typically fast for small images but may be noticeably slower for very large images, which can increase total inference time.
:::

## Image Segmentation
## Semantic Segmentation

:::warning
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization.
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/02-benchmarks/memory-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ The reported memory usage values include the memory footprint of the Phonemis pa
| --------------------------- | :--------------------: | :----------------: |
| CLIP_VIT_BASE_PATCH32_IMAGE | 345 | 340 |

## Image Segmentation
## Semantic Segmentation

:::warning
Data presented in the following sections is based on inference with non-resized output. When resize is enabled, expect higher memory usage and inference time with higher resolutions.
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/02-benchmarks/model-size.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ title: Model Size
| --------------------------- | :----------: |
| CLIP_VIT_BASE_PATCH32_IMAGE | 352 |

## Image Segmentation
## Semantic Segmentation

| Model | XNNPACK [MB] |
| ----------------- | ------------ |
Expand Down
Original file line number Diff line number Diff line change
@@ -1,27 +1,27 @@
---
title: useImageSegmentation
title: useSemanticSegmentation
---

Semantic image segmentation, akin to image classification, tries to assign the content of the image to one of the predefined classes. However, in case of segmentation this classification is done on a per-pixel basis, so as the result the model provides an image-sized array of scores for each of the classes. You can then use this information to detect objects on a per-pixel basis. React Native ExecuTorch offers a dedicated hook `useImageSegmentation` for this task.
Semantic semantic segmentation, akin to image classification, tries to assign the content of the image to one of the predefined classes. However, in case of segmentation this classification is done on a per-pixel basis, so as the result the model provides an image-sized array of scores for each of the classes. You can then use this information to detect objects on a per-pixel basis. React Native ExecuTorch offers a dedicated hook `useSemanticSegmentation` for this task.

:::warning
It is recommended to use models provided by us which are available at our [Hugging Face repository](https://huggingface.co/collections/software-mansion/image-segmentation-68d5291bdf4a30bee0220f4f), you can also use [constants](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts) shipped with our library.
It is recommended to use models provided by us which are available at our [Hugging Face repository](https://huggingface.co/collections/software-mansion/semantic-segmentation-68d5291bdf4a30bee0220f4f), you can also use [constants](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts) shipped with our library.
:::

## API Reference

- For detailed API Reference for `useImageSegmentation` see: [`useImageSegmentation` API Reference](../../06-api-reference/functions/useImageSegmentation.md).
- For all image segmentation models available out-of-the-box in React Native ExecuTorch see: [Image Segmentation Models](../../06-api-reference/index.md#models---image-segmentation).
- For detailed API Reference for `useSemanticSegmentation` see: [`useSemanticSegmentation` API Reference](../../06-api-reference/functions/useSemanticSegmentation.md).
- For all semantic segmentation models available out-of-the-box in React Native ExecuTorch see: [Semantic Segmentation Models](../../06-api-reference/index.md#models---semantic-segmentation).

## High Level Overview

```typescript
import {
useImageSegmentation,
useSemanticSegmentation,
DEEPLAB_V3_RESNET50,
} from 'react-native-executorch';

const model = useImageSegmentation({
const model = useSemanticSegmentation({
model: DEEPLAB_V3_RESNET50,
});

Expand All @@ -37,24 +37,24 @@ try {

### Arguments

`useImageSegmentation` takes [`ImageSegmentationProps`](../../06-api-reference/interfaces/ImageSegmentationProps.md) that consists of:
`useSemanticSegmentation` takes [`SemanticSegmentationProps`](../../06-api-reference/interfaces/SemanticSegmentationProps.md) that consists of:

- `model` - An object containing:
- `modelName` - The name of a built-in model. See [`ModelSources`](../../06-api-reference/type-aliases/ModelSources.md) for the list of supported models.
- `modelName` - The name of a built-in model. See [`SemanticSegmentationModelSources`](../../06-api-reference/type-aliases/SemanticSegmentationModelSources.md) for the list of supported models.
- `modelSource` - The location of the model binary (a URL or a bundled resource).
- An optional flag [`preventLoad`](../../06-api-reference/interfaces/ImageSegmentationProps.md#preventload) which prevents auto-loading of the model.
- An optional flag [`preventLoad`](../../06-api-reference/interfaces/SemanticSegmentationProps.md#preventload) which prevents auto-loading of the model.

The hook is generic over the model config — TypeScript automatically infers the correct label type based on the `modelName` you provide. No explicit generic parameter is needed.

You need more details? Check the following resources:

- For detailed information about `useImageSegmentation` arguments check this section: [`useImageSegmentation` arguments](../../06-api-reference/functions/useImageSegmentation.md#parameters).
- For all image segmentation models available out-of-the-box in React Native ExecuTorch see: [Image Segmentation Models](../../06-api-reference/index.md#models---image-segmentation).
- For detailed information about `useSemanticSegmentation` arguments check this section: [`useSemanticSegmentation` arguments](../../06-api-reference/functions/useSemanticSegmentation.md#parameters).
- For all semantic segmentation models available out-of-the-box in React Native ExecuTorch see: [Semantic Segmentation Models](../../06-api-reference/index.md#models---semantic-segmentation).
- For more information on loading resources, take a look at [loading models](../../01-fundamentals/02-loading-models.md) page.

### Returns

`useImageSegmentation` returns an [`ImageSegmentationType`](../../06-api-reference/interfaces/ImageSegmentationType.md) object containing:
`useSemanticSegmentation` returns an [`SemanticSegmentationType`](../../06-api-reference/interfaces/SemanticSegmentationType.md) object containing:

- `isReady` - Whether the model is loaded and ready to process images.
- `isGenerating` - Whether the model is currently processing an image.
Expand All @@ -64,11 +64,11 @@ You need more details? Check the following resources:

## Running the model

To run the model, use the [`forward`](../../06-api-reference/interfaces/ImageSegmentationType.md#forward) method. It accepts three arguments:
To run the model, use the [`forward`](../../06-api-reference/interfaces/SemanticSegmentationType.md#forward) method. It accepts three arguments:

- [`imageSource`](../../06-api-reference/interfaces/ImageSegmentationType.md#forward) (required) - The image to segment. Can be a remote URL, a local file URI, or a base64-encoded image (whole URI or only raw base64).
- [`classesOfInterest`](../../06-api-reference/interfaces/ImageSegmentationType.md#forward) (optional) - An array of label keys indicating which per-class probability masks to include in the output. Defaults to `[]` (no class masks). The `ARGMAX` map is always returned regardless of this parameter.
- [`resizeToInput`](../../06-api-reference/interfaces/ImageSegmentationType.md#forward) (optional) - Whether to resize the output masks to the original input image dimensions. Defaults to `true`. If `false`, returns the raw model output dimensions (e.g. 224x224 for `DEEPLAB_V3_RESNET50`).
- [`imageSource`](../../06-api-reference/interfaces/SemanticSegmentationType.md#forward) (required) - The image to segment. Can be a remote URL, a local file URI, or a base64-encoded image (whole URI or only raw base64).
- [`classesOfInterest`](../../06-api-reference/interfaces/SemanticSegmentationType.md#forward) (optional) - An array of label keys indicating which per-class probability masks to include in the output. Defaults to `[]` (no class masks). The `ARGMAX` map is always returned regardless of this parameter.
- [`resizeToInput`](../../06-api-reference/interfaces/SemanticSegmentationType.md#forward) (optional) - Whether to resize the output masks to the original input image dimensions. Defaults to `true`. If `false`, returns the raw model output dimensions (e.g. 224x224 for `DEEPLAB_V3_RESNET50`).

:::warning
Setting `resizeToInput` to `false` will make `forward` faster.
Expand All @@ -85,13 +85,13 @@ The return type is fully typed — TypeScript narrows it based on the labels you

```typescript
import {
useImageSegmentation,
useSemanticSegmentation,
DEEPLAB_V3_RESNET50,
DeeplabLabel,
} from 'react-native-executorch';

function App() {
const model = useImageSegmentation({
const model = useSemanticSegmentation({
model: DEEPLAB_V3_RESNET50,
});

Expand Down
Original file line number Diff line number Diff line change
@@ -1,26 +1,26 @@
---
title: ImageSegmentationModule
title: SemanticSegmentationModule
---

TypeScript API implementation of the [useImageSegmentation](../../03-hooks/02-computer-vision/useImageSegmentation.md) hook.
TypeScript API implementation of the [useSemanticSegmentation](../../03-hooks/02-computer-vision/useSemanticSegmentation.md) hook.

## API Reference

- For detailed API Reference for `ImageSegmentationModule` see: [`ImageSegmentationModule` API Reference](../../06-api-reference/classes/ImageSegmentationModule.md).
- For all image segmentation models available out-of-the-box in React Native ExecuTorch see: [Image Segmentation Models](../../06-api-reference/index.md#models---image-segmentation).
- For detailed API Reference for `SemanticSegmentationModule` see: [`SemanticSegmentationModule` API Reference](../../06-api-reference/classes/SemanticSegmentationModule.md).
- For all semantic segmentation models available out-of-the-box in React Native ExecuTorch see: [Semantic Segmentation Models](../../06-api-reference/index.md#models---semantic-segmentation).

## High Level Overview

```typescript
import {
ImageSegmentationModule,
SemanticSegmentationModule,
DEEPLAB_V3_RESNET50,
} from 'react-native-executorch';

const imageUri = 'path/to/image.png';

// Creating an instance from a built-in model
const segmentation = await ImageSegmentationModule.fromModelName({
const segmentation = await SemanticSegmentationModule.fromModelName({
modelName: 'deeplab-v3',
modelSource: DEEPLAB_V3_RESNET50,
});
Expand All @@ -32,18 +32,18 @@ const result = await segmentation.forward(imageUri);

### Methods

All methods of `ImageSegmentationModule` are explained in details here: [`ImageSegmentationModule` API Reference](../../06-api-reference/classes/ImageSegmentationModule.md)
All methods of `SemanticSegmentationModule` are explained in details here: [`SemanticSegmentationModule` API Reference](../../06-api-reference/classes/SemanticSegmentationModule.md)

## Loading the model

`ImageSegmentationModule` uses static factory methods instead of `new()` + `load()`. There are two ways to create an instance:
`SemanticSegmentationModule` uses static factory methods instead of `new()` + `load()`. There are two ways to create an instance:

### Built-in models — `fromModelName`

Use [`fromModelName`](../../06-api-reference/classes/ImageSegmentationModule.md#frommodelname) for models that ship with built-in label maps and preprocessing configs:
Use [`fromModelName`](../../06-api-reference/classes/SemanticSegmentationModule.md#frommodelname) for models that ship with built-in label maps and preprocessing configs:

```typescript
const segmentation = await ImageSegmentationModule.fromModelName(
const segmentation = await SemanticSegmentationModule.fromModelName(
DEEPLAB_V3_RESNET50,
(progress) => console.log(`Download: ${Math.round(progress * 100)}%`)
);
Expand All @@ -53,12 +53,12 @@ The `config` parameter is a discriminated union — TypeScript ensures you provi

### Custom models — `fromCustomConfig`

Use [`fromCustomConfig`](../../06-api-reference/classes/ImageSegmentationModule.md#fromcustomconfig) for custom-exported segmentation models with your own label map:
Use [`fromCustomConfig`](../../06-api-reference/classes/SemanticSegmentationModule.md#fromcustomconfig) for custom-exported segmentation models with your own label map:

```typescript
const MyLabels = { BACKGROUND: 0, FOREGROUND: 1 } as const;

const segmentation = await ImageSegmentationModule.fromCustomConfig(
const segmentation = await SemanticSegmentationModule.fromCustomConfig(
'https://example.com/custom_model.pte',
{
labelMap: MyLabels,
Expand All @@ -76,11 +76,11 @@ For more information on loading resources, take a look at [loading models](../..

## Running the model

To run the model, use the [`forward`](../../06-api-reference/classes/ImageSegmentationModule.md#forward) method. It accepts three arguments:
To run the model, use the [`forward`](../../06-api-reference/classes/SemanticSegmentationModule.md#forward) method. It accepts three arguments:

- [`imageSource`](../../06-api-reference/classes/ImageSegmentationModule.md#forward) (required) - The image to segment. Can be a remote URL, a local file URI, or a base64-encoded image (whole URI or only raw base64).
- [`classesOfInterest`](../../06-api-reference/classes/ImageSegmentationModule.md#forward) (optional) - An array of label keys indicating which per-class probability masks to include in the output. Defaults to `[]`. The `ARGMAX` map is always returned regardless.
- [`resizeToInput`](../../06-api-reference/classes/ImageSegmentationModule.md#forward) (optional) - Whether to resize the output masks to the original input image dimensions. Defaults to `true`. If `false`, returns the raw model output dimensions.
- [`imageSource`](../../06-api-reference/classes/SemanticSegmentationModule.md#forward) (required) - The image to segment. Can be a remote URL, a local file URI, or a base64-encoded image (whole URI or only raw base64).
- [`classesOfInterest`](../../06-api-reference/classes/SemanticSegmentationModule.md#forward) (optional) - An array of label keys indicating which per-class probability masks to include in the output. Defaults to `[]`. The `ARGMAX` map is always returned regardless.
- [`resizeToInput`](../../06-api-reference/classes/SemanticSegmentationModule.md#forward) (optional) - Whether to resize the output masks to the original input image dimensions. Defaults to `true`. If `false`, returns the raw model output dimensions.

:::warning
Setting `resizeToInput` to `false` will make `forward` faster.
Expand All @@ -107,4 +107,4 @@ result.DOG; // Float32Array

## Managing memory

The module is a regular JavaScript object, and as such its lifespan will be managed by the garbage collector. In most cases this should be enough, and you should not worry about freeing the memory of the module yourself, but in some cases you may want to release the memory occupied by the module before the garbage collector steps in. In this case use the method [`delete`](../../06-api-reference/classes/ImageSegmentationModule.md#delete) on the module object you will no longer use, and want to remove from the memory. Note that you cannot use [`forward`](../../06-api-reference/classes/ImageSegmentationModule.md#forward) after [`delete`](../../06-api-reference/classes/ImageSegmentationModule.md#delete) unless you create a new instance.
The module is a regular JavaScript object, and as such its lifespan will be managed by the garbage collector. In most cases this should be enough, and you should not worry about freeing the memory of the module yourself, but in some cases you may want to release the memory occupied by the module before the garbage collector steps in. In this case use the method [`delete`](../../06-api-reference/classes/SemanticSegmentationModule.md#delete) on the module object you will no longer use, and want to remove from the memory. Note that you cannot use [`forward`](../../06-api-reference/classes/SemanticSegmentationModule.md#forward) after [`delete`](../../06-api-reference/classes/SemanticSegmentationModule.md#delete) unless you create a new instance.
Loading
Loading