Skip to content

Commit 7ec9f07

Browse files
JakubGoneraJakubGonera
authored
docs: Add documentation for image segmentation (#149)
## Description Add documentation for image segmentation ### Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] Documentation update (improves or adds clarity to existing documentation)
1 parent 9ca8288 commit 7ec9f07

File tree

5 files changed

+181
-3
lines changed

5 files changed

+181
-3
lines changed

‎docs/docs/computer-vision/useClassification.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Usually, the class with the highest probability is the one that is assigned to a
1010
:::
1111

1212
:::caution
13-
It is recommended to use models provided by us, which are available at our [Hugging Face repository](https://huggingface.co/software-mansion/react-native-executorch-efficientnet-v2-s). You can also use [constants](https://github.com/software-mansion/react-native-executorch/tree/main/src/constants/modelUrls.ts) shipped with our library
13+
It is recommended to use models provided by us, which are available at our [Hugging Face repository](https://huggingface.co/software-mansion/react-native-executorch-efficientnet-v2-s). You can also use [constants](https://github.com/software-mansion/react-native-executorch/tree/main/src/constants/modelUrls.ts) shipped with our library.
1414
:::
1515

1616
## Reference

‎docs/docs/computer-vision/useImageSegmentation.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
---
2+
title: useImageSegmentation
3+
sidebar_position: 2
4+
---
5+
6+
Semantic image segmentation, akin to image classification, tries to assign the content of the image to one of the predefined classes. However, in case of segmentation this classification is done on a per-pixel basis, so as the result the model provides an image-sized array of scores for each of the classes. You can then use this information to detect objects on a per-pixel basis. React Native ExecuTorch offers a dedicated hook `useImageSegmentation` for this task.
7+
8+
:::caution
9+
It is recommended to use models provided by us which are available at our [Hugging Face repository](https://huggingface.co/software-mansion/react-native-executorch-style-transfer-candy), you can also use [constants](https://github.com/software-mansion/react-native-executorch/tree/main/src/constants/modelUrls.ts) shipped with our library.
10+
:::
11+
12+
## Reference
13+
14+
```typescript
15+
import {
16+
useImageSegmentation,
17+
DEEPLABV3_RESNET50,
18+
} from 'react-native-executorch';
19+
20+
const model = useImageSegmentation({
21+
modelSource: DEEPLABV3_RESNET50,
22+
});
23+
24+
const imageUri = 'file::///Users/.../cute_cat.png';
25+
26+
try {
27+
const outputDict = await model.forward(imageUri);
28+
} catch (error) {
29+
console.error(error);
30+
}
31+
```
32+
33+
### Arguments
34+
35+
**`modelSource`**
36+
A string that specifies the location of the model binary. For more information, take a look at [loading models](../fundamentals/loading-models.md) page.
37+
38+
### Returns
39+
40+
| Field | Type | Description |
41+
| ------------------ | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
42+
| `forward` | `(input: string, classesOfInterest?: DeeplabLabel[], resize?: boolean) => Promise<{[key in DeeplabLabel]?: number[]}>` | Executes the model's forward pass, where: <br/> \* `input` can be a fetchable resource or a Base64-encoded string. <br/> \* `classesOfInterest` is an optional list of `DeeplabLabel` used to indicate additional arrays of probabilities to output (see section "Running the model"). The default is an empty list. <br/> \* `resize` is an optional boolean to indicate whether the output should be resized to the original image dimensions, or left in the size of the model (see section "Running the model"). The default is `false`. <br/> <br/> The return is a dictionary containing: <br/> \* for the key `DeeplabLabel.ARGMAX` an array of integers corresponding to the most probable class for each pixel <br/> \* an array of floats for each class from `classesOfInterest` corresponding to the probabilities for this class. |
43+
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load. |
44+
| `isGenerating` | `boolean` | Indicates whether the model is currently processing an inference. |
45+
| `isReady` | `boolean` | Indicates whether the model has successfully loaded and is ready for inference. |
46+
| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1. |
47+
48+
## Running the model
49+
50+
To run the model, you can use the `forward` method. It accepts three arguments: a required image, an optional list of classes, and an optional flag whether to resize the output to the original dimensions.
51+
52+
- The image can be a remote URL, a local file URI, or a base64-encoded image.
53+
- The `classesOfInterest` list contains classes for which to output the full results. By default the list is empty, and only the most probable classes are returned (esentially an arg max for each pixel). Look at [`DeeplabLabel`](http://github.com/software-mansion/react-native-executorch/blob/main/src/types/image_segmentation.ts) enum for possible classes.
54+
- The `resize` flag says whether the output will be rescaled back to the size of the image you put in. The default is `false`. The model runs inference on a scaled (probably smaller) version of your image (224x224 for `DEEPLABV3_RESNET50`). If you choose to resize, the output will be `number[]` of size `width * height` of your original image.
55+
56+
:::caution
57+
Setting `resize` to true will make `forward` slower.
58+
:::
59+
60+
`forward` returns a promise which can resolve either to an error or a dictionary containing number arrays with size depending on `resize`:
61+
62+
- For the key `DeeplabLabel.ARGMAX` the array contains for each pixel an integer corresponding to the class with the highest probability.
63+
- For every other key from `DeeplabLabel`, if the label was included in `classesOfInterest` the dictionary will contain an array of floats corresponding to the probability of this class for every pixel.
64+
65+
## Example
66+
67+
```typescript
68+
function App(){
69+
const model = useImageSegmentation(
70+
modelSource: DEEPLABV3_RESNET50,
71+
);
72+
73+
...
74+
const imageUri = 'file::///Users/.../cute_cat.png';
75+
76+
try{
77+
const outputDict = await model.forward(imageUri, [DeeplabLabel.CAT], true);
78+
}catch(error){
79+
console.error(error);
80+
}
81+
...
82+
}
83+
```
84+
85+
## Supported models
86+
87+
| Model | Number of classes | Class list |
88+
| ------------------------------------------------------------------------------------------------------------------------------ | ----------------- | -------------------------------------------------------------------------------------------------------------------- |
89+
| [deeplabv3_resnet50](https://pytorch.org/vision/0.20/models/generated/torchvision.models.segmentation.deeplabv3_resnet50.html) | 21 | [DeeplabLabel](http://github.com/software-mansion/react-native-executorch/blob/main/src/types/image_segmentation.ts) |
90+
91+
## Benchmarks
92+
93+
### Model size
94+
95+
| Model | XNNPACK [MB] |
96+
| ----------------- | ------------ |
97+
| DEELABV3_RESNET50 | 168 |
98+
99+
### Memory usage
100+
101+
:::warning warning
102+
Data presented in the following sections is based on inference with non-resized output. When resize is enabled, expect higher memory usage and inference time with higher resolutions.
103+
:::
104+
105+
| Model | Android (XNNPACK) [MB] | iOS (XNNPACK) [MB] |
106+
| ----------------- | ---------------------- | ------------------ |
107+
| DEELABV3_RESNET50 | 930 | 660 |
108+
109+
### Inference time
110+
111+
:::warning warning
112+
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization.
113+
:::
114+
115+
| Model | iPhone 16 Pro (Core ML) [ms] | iPhone 14 Pro Max (Core ML) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] |
116+
| ----------------- | ---------------------------- | -------------------------------- | --------------------------------- |
117+
| DEELABV3_RESNET50 | 1000 | 670 | 700 |

‎docs/docs/computer-vision/useStyleTransfer.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ sidebar_position: 2
66
Style transfer is a technique used in computer graphics and machine learning where the visual style of one image is applied to the content of another. This is achieved using algorithms that manipulate data from both images, typically with the aid of a neural network. The result is a new image that combines the artistic elements of one picture with the structural details of another, effectively merging art with traditional imagery. React Native ExecuTorch offers a dedicated hook `useStyleTransfer`, for this task. However before you start you'll need to obtain ExecuTorch-compatible model binary.
77

88
:::caution
9-
It is recommended to use models provided by us which are available at our [Hugging Face repository](https://huggingface.co/software-mansion/react-native-executorch-style-transfer-candy), you can also use [constants](https://github.com/software-mansion/react-native-executorch/tree/main/src/constants/modelUrls.ts) shipped with our library
9+
It is recommended to use models provided by us which are available at our [Hugging Face repository](https://huggingface.co/software-mansion/react-native-executorch-style-transfer-candy), you can also use [constants](https://github.com/software-mansion/react-native-executorch/tree/main/src/constants/modelUrls.ts) shipped with our library.
1010
:::
1111

1212
## Reference

0 commit comments

Comments
 (0)