Skip to content

Commit 2b6ea20

Browse files
barhancmsluszniak
barhanc
and
msluszniak
authored
chore: add support for semantic segmentation models DeeplabV3, LRASPP, FCN (#881)
## Description Adds support for semantic segmentation models DeeplabV3, LRASPP, FCN, both in quantized and FP32 versions. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [ ] Bug fix (change which fixes an issue) - [ ] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [x] Other (chores, tests, code style improvements etc.) ### Tested on - [x] iOS - [x] Android ### Testing instructions 1. Run the computer vision semantic segmentation example app with all added models. 2. Check HF pages for added models in the semantic segmentation collection: - https://huggingface.co/software-mansion/react-native-executorch-deeplab-v3 - https://huggingface.co/software-mansion/react-native-executorch-lraspp - https://huggingface.co/software-mansion/react-native-executorch-fcn ### Screenshots <!-- Add screenshots here, if applicable --> ### Related issues <!-- Link related issues here using #issue-number --> Closes #799 #830 ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes <!-- Include any additional information, assumptions, or context that reviewers might need to understand this PR. --> --------- Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>
1 parent 3f4fdb5 commit 2b6ea20

39 files changed

Lines changed: 402 additions & 77 deletions

‎.cspell-wordlist.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,3 +116,5 @@ antonov
116116
rfdetr
117117
basemodule
118118
IMAGENET
119+
lraspp
120+
LRASPP

‎docs/docs/03-hooks/02-computer-vision/useSemanticSegmentation.md‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,12 @@ function App() {
117117

118118
## Supported models
119119

120-
| Model | Number of classes | Class list |
121-
| ------------------------------------------------------------------------------------------------ | ----------------- | ----------------------------------------------------------------------------------------- |
122-
| [deeplabv3_resnet50](https://huggingface.co/software-mansion/react-native-executorch-deeplab-v3) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) |
123-
| selfie-segmentation | 2 | [SelfieSegmentationLabel](../../06-api-reference/enumerations/SelfieSegmentationLabel.md) |
120+
| Model | Number of classes | Class list | Quantized |
121+
| ----------------------------------------------------------------------------------------------------------- | ----------------- | ----------------------------------------------------------------------------------------- | :-------: |
122+
| [deeplab-v3-resnet50](https://huggingface.co/software-mansion/react-native-executorch-deeplab-v3) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) | Yes |
123+
| [deeplab-v3-resnet101](https://huggingface.co/software-mansion/react-native-executorch-deeplab-v3) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) | Yes |
124+
| [deeplab-v3-mobilenet-v3-large](https://huggingface.co/software-mansion/react-native-executorch-deeplab-v3) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) | Yes |
125+
| [lraspp-mobilenet-v3-large](https://huggingface.co/software-mansion/react-native-executorch-lraspp) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) | Yes |
126+
| [fcn-resnet50](https://huggingface.co/software-mansion/react-native-executorch-fcn) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) | Yes |
127+
| [fcn-resnet101](https://huggingface.co/software-mansion/react-native-executorch-fcn) | 21 | [DeeplabLabel](../../06-api-reference/enumerations/DeeplabLabel.md) | Yes |
128+
| [selfie-segmentation](https://huggingface.co/software-mansion/react-native-executorch-selfie-segmentation) | 2 | [SelfieSegmentationLabel](../../06-api-reference/enumerations/SelfieSegmentationLabel.md) | No |

‎docs/docs/04-typescript-api/02-computer-vision/SemanticSegmentationModule.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ const segmentation = await SemanticSegmentationModule.fromModelName(
4949
);
5050
```
5151

52-
The `config` parameter is a discriminated union — TypeScript ensures you provide the correct fields for each model name. Available built-in models: `'deeplab-v3'`, `'selfie-segmentation'`.
52+
The `config` parameter is a discriminated union — TypeScript ensures you provide the correct fields for each model name. Available built-in models: `'deeplab-v3-resnet50'`, `'deeplab-v3-resnet50-quantized'`, `'deeplab-v3-resnet101'`, `'deeplab-v3-resnet101-quantized'`, `'deeplab-v3-mobilenet-v3-large'`, `'deeplab-v3-mobilenet-v3-large-quantized'`, `'lraspp-mobilenet-v3-large'`, `'lraspp-mobilenet-v3-large-quantized'`, `'fcn-resnet50'`, `'fcn-resnet50-quantized'`, `'fcn-resnet101'`, `'fcn-resnet101-quantized'`, and `'selfie-segmentation'`.
5353

5454
### Custom models — `fromCustomConfig`
5555

‎docs/docs/06-api-reference/classes/SemanticSegmentationModule.md‎

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
# Class: SemanticSegmentationModule\<T\>
22

3-
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:60](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L60)
3+
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:81](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L81)
44

55
Generic semantic segmentation module with type-safe label maps.
6-
Use a model name (e.g. `'deeplab-v3'`) as the generic parameter for built-in models,
6+
Use a model name (e.g. `'deeplab-v3-resnet50'`) as the generic parameter for built-in models,
77
or a custom label enum for custom configs.
88

99
## Extends
@@ -16,8 +16,13 @@ or a custom label enum for custom configs.
1616

1717
`T` _extends_ [`SemanticSegmentationModelName`](../type-aliases/SemanticSegmentationModelName.md) \| [`LabelEnum`](../type-aliases/LabelEnum.md)
1818

19-
Either a built-in model name (`'deeplab-v3'`, `'selfie-segmentation'`)
20-
or a custom [LabelEnum](../type-aliases/LabelEnum.md) label map.
19+
Either a built-in model name (`'deeplab-v3-resnet50'`,
20+
`'deeplab-v3-resnet50-quantized'`, `'deeplab-v3-resnet101'`,
21+
`'deeplab-v3-resnet101-quantized'`, `'deeplab-v3-mobilenet-v3-large'`,
22+
`'deeplab-v3-mobilenet-v3-large-quantized'`, `'lraspp-mobilenet-v3-large'`,
23+
`'lraspp-mobilenet-v3-large-quantized'`, `'fcn-resnet50'`,
24+
`'fcn-resnet50-quantized'`, `'fcn-resnet101'`, `'fcn-resnet101-quantized'`,
25+
`'selfie-segmentation'`) or a custom [LabelEnum](../type-aliases/LabelEnum.md) label map.
2126

2227
## Properties
2328

@@ -57,7 +62,7 @@ Unloads the model from memory.
5762

5863
> **forward**\<`K`\>(`imageSource`, `classesOfInterest?`, `resizeToInput?`): `Promise`\<`Record`\<`"ARGMAX"`, `Int32Array`\<`ArrayBufferLike`\>\> & `Record`\<`K`, `Float32Array`\<`ArrayBufferLike`\>\>\>
5964
60-
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:176](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L176)
65+
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:197](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L197)
6166

6267
Executes the model's forward pass to perform semantic segmentation on the provided image.
6368

@@ -166,7 +171,7 @@ The input shape as an array of numbers.
166171

167172
> **load**(): `Promise`\<`void`\>
168173
169-
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:76](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L76)
174+
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:97](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L97)
170175

171176
#### Returns
172177

@@ -182,7 +187,7 @@ Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:76](https://g
182187

183188
> `static` **fromCustomConfig**\<`L`\>(`modelSource`, `config`, `onDownloadProgress?`): `Promise`\<`SemanticSegmentationModule`\<`L`\>\>
184189
185-
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:142](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L142)
190+
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:163](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L163)
186191

187192
Creates a segmentation instance with a user-provided label map and custom config.
188193
Use this when working with a custom-exported segmentation model that is not one of the built-in models.
@@ -235,7 +240,7 @@ const segmentation = await SemanticSegmentationModule.fromCustomConfig(
235240

236241
> `static` **fromModelName**\<`C`\>(`config`, `onDownloadProgress?`): `Promise`\<`SemanticSegmentationModule`\<[`ModelNameOf`](../type-aliases/ModelNameOf.md)\<`C`\>\>\>
237242
238-
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:95](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L95)
243+
Defined in: [modules/computer_vision/SemanticSegmentationModule.ts:116](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/modules/computer_vision/SemanticSegmentationModule.ts#L116)
239244

240245
Creates a segmentation instance for a built-in model.
241246
The config object is discriminated by `modelName` — each model can require different fields.

‎docs/docs/06-api-reference/enumerations/DeeplabLabel.md‎

Lines changed: 22 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Enumeration: DeeplabLabel
22

3-
Defined in: [types/semanticSegmentation.ts:53](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L53)
3+
Defined in: [types/semanticSegmentation.ts:70](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L70)
44

55
Labels used in the DeepLab semantic segmentation model.
66

@@ -10,164 +10,164 @@ Labels used in the DeepLab semantic segmentation model.
1010

1111
> **AEROPLANE**: `1`
1212
13-
Defined in: [types/semanticSegmentation.ts:55](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L55)
13+
Defined in: [types/semanticSegmentation.ts:72](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L72)
1414

1515
---
1616

1717
### BACKGROUND
1818

1919
> **BACKGROUND**: `0`
2020
21-
Defined in: [types/semanticSegmentation.ts:54](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L54)
21+
Defined in: [types/semanticSegmentation.ts:71](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L71)
2222

2323
---
2424

2525
### BICYCLE
2626

2727
> **BICYCLE**: `2`
2828
29-
Defined in: [types/semanticSegmentation.ts:56](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L56)
29+
Defined in: [types/semanticSegmentation.ts:73](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L73)
3030

3131
---
3232

3333
### BIRD
3434

3535
> **BIRD**: `3`
3636
37-
Defined in: [types/semanticSegmentation.ts:57](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L57)
37+
Defined in: [types/semanticSegmentation.ts:74](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L74)
3838

3939
---
4040

4141
### BOAT
4242

4343
> **BOAT**: `4`
4444
45-
Defined in: [types/semanticSegmentation.ts:58](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L58)
45+
Defined in: [types/semanticSegmentation.ts:75](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L75)
4646

4747
---
4848

4949
### BOTTLE
5050

5151
> **BOTTLE**: `5`
5252
53-
Defined in: [types/semanticSegmentation.ts:59](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L59)
53+
Defined in: [types/semanticSegmentation.ts:76](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L76)
5454

5555
---
5656

5757
### BUS
5858

5959
> **BUS**: `6`
6060
61-
Defined in: [types/semanticSegmentation.ts:60](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L60)
61+
Defined in: [types/semanticSegmentation.ts:77](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L77)
6262

6363
---
6464

6565
### CAR
6666

6767
> **CAR**: `7`
6868
69-
Defined in: [types/semanticSegmentation.ts:61](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L61)
69+
Defined in: [types/semanticSegmentation.ts:78](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L78)
7070

7171
---
7272

7373
### CAT
7474

7575
> **CAT**: `8`
7676
77-
Defined in: [types/semanticSegmentation.ts:62](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L62)
77+
Defined in: [types/semanticSegmentation.ts:79](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L79)
7878

7979
---
8080

8181
### CHAIR
8282

8383
> **CHAIR**: `9`
8484
85-
Defined in: [types/semanticSegmentation.ts:63](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L63)
85+
Defined in: [types/semanticSegmentation.ts:80](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L80)
8686

8787
---
8888

8989
### COW
9090

9191
> **COW**: `10`
9292
93-
Defined in: [types/semanticSegmentation.ts:64](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L64)
93+
Defined in: [types/semanticSegmentation.ts:81](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L81)
9494

9595
---
9696

9797
### DININGTABLE
9898

9999
> **DININGTABLE**: `11`
100100
101-
Defined in: [types/semanticSegmentation.ts:65](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L65)
101+
Defined in: [types/semanticSegmentation.ts:82](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L82)
102102

103103
---
104104

105105
### DOG
106106

107107
> **DOG**: `12`
108108
109-
Defined in: [types/semanticSegmentation.ts:66](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L66)
109+
Defined in: [types/semanticSegmentation.ts:83](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L83)
110110

111111
---
112112

113113
### HORSE
114114

115115
> **HORSE**: `13`
116116
117-
Defined in: [types/semanticSegmentation.ts:67](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L67)
117+
Defined in: [types/semanticSegmentation.ts:84](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L84)
118118

119119
---
120120

121121
### MOTORBIKE
122122

123123
> **MOTORBIKE**: `14`
124124
125-
Defined in: [types/semanticSegmentation.ts:68](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L68)
125+
Defined in: [types/semanticSegmentation.ts:85](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L85)
126126

127127
---
128128

129129
### PERSON
130130

131131
> **PERSON**: `15`
132132
133-
Defined in: [types/semanticSegmentation.ts:69](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L69)
133+
Defined in: [types/semanticSegmentation.ts:86](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L86)
134134

135135
---
136136

137137
### POTTEDPLANT
138138

139139
> **POTTEDPLANT**: `16`
140140
141-
Defined in: [types/semanticSegmentation.ts:70](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L70)
141+
Defined in: [types/semanticSegmentation.ts:87](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L87)
142142

143143
---
144144

145145
### SHEEP
146146

147147
> **SHEEP**: `17`
148148
149-
Defined in: [types/semanticSegmentation.ts:71](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L71)
149+
Defined in: [types/semanticSegmentation.ts:88](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L88)
150150

151151
---
152152

153153
### SOFA
154154

155155
> **SOFA**: `18`
156156
157-
Defined in: [types/semanticSegmentation.ts:72](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L72)
157+
Defined in: [types/semanticSegmentation.ts:89](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L89)
158158

159159
---
160160

161161
### TRAIN
162162

163163
> **TRAIN**: `19`
164164
165-
Defined in: [types/semanticSegmentation.ts:73](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L73)
165+
Defined in: [types/semanticSegmentation.ts:90](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L90)
166166

167167
---
168168

169169
### TVMONITOR
170170

171171
> **TVMONITOR**: `20`
172172
173-
Defined in: [types/semanticSegmentation.ts:74](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L74)
173+
Defined in: [types/semanticSegmentation.ts:91](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L91)

‎docs/docs/06-api-reference/enumerations/SelfieSegmentationLabel.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Enumeration: SelfieSegmentationLabel
22

3-
Defined in: [types/semanticSegmentation.ts:82](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L82)
3+
Defined in: [types/semanticSegmentation.ts:99](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L99)
44

55
Labels used in the selfie semantic segmentation model.
66

@@ -10,12 +10,12 @@ Labels used in the selfie semantic segmentation model.
1010

1111
> **BACKGROUND**: `1`
1212
13-
Defined in: [types/semanticSegmentation.ts:84](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L84)
13+
Defined in: [types/semanticSegmentation.ts:101](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L101)
1414

1515
---
1616

1717
### SELFIE
1818

1919
> **SELFIE**: `0`
2020
21-
Defined in: [types/semanticSegmentation.ts:83](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L83)
21+
Defined in: [types/semanticSegmentation.ts:100](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/semanticSegmentation.ts#L100)

0 commit comments

Comments
 (0)