Skip to content

Commit 6556b74

Browse files
barhancbarhanc
committed
docs: update inference time and model size
1 parent 94789f1 commit 6556b74

File tree

2 files changed

+77
-39
lines changed

2 files changed

+77
-39
lines changed

‎docs/docs/02-benchmarks/inference-time.md‎

Lines changed: 54 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -8,24 +8,41 @@ Times presented in the tables are measured as consecutive runs of the model. Ini
88

99
## Classification
1010

11-
| Model | iPhone 17 Pro (Core ML) [ms] | iPhone 16 Pro (Core ML) [ms] | iPhone SE 3 (Core ML) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
12-
| ----------------- | :--------------------------: | :--------------------------: | :------------------------: | :-------------------------------: | :-----------------------: |
13-
| EFFICIENTNET_V2_S | 64 | 68 | 217 | 205 | 198 |
11+
:::info
12+
Times presented below are _model inference times only_ and do not include time taken for pre-processing (e.g. image resizing, normalization) or post-processing (e.g. image resizing) which are dependent on input size.
13+
:::
14+
15+
| Model | iPhone 17 Pro (Core ML) [ms] | Google Pixel 10 (XNNPACK) [ms] |
16+
| --------------------------- | :--------------------------: | :----------------------------: |
17+
| EFFICIENTNET_V2_S | 12 | 100 |
18+
| EFFICIENTNET_V2_S_QUANTIZED | 5 | 38 |
1419

1520
## Object Detection
1621

17-
| Model | iPhone 17 Pro (XNNPACK) [ms] | iPhone 16 Pro (XNNPACK) [ms] | iPhone SE 3 (XNNPACK) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
18-
| ------------------------------ | :--------------------------: | :--------------------------: | :------------------------: | :-------------------------------: | :-----------------------: |
19-
| SSDLITE_320_MOBILENET_V3_LARGE | 71 | 74 | 257 | 115 | 109 |
22+
:::info
23+
Times presented below are _model inference times only_ and do not include time taken for pre-processing (e.g. image resizing, normalization) or post-processing (e.g. image resizing) which are dependent on input size.
24+
:::
25+
26+
| Model | iPhone 17 Pro (Core ML) [ms] | Google Pixel 10 (XNNPACK) [ms] |
27+
| ------------------------------ | :--------------------------: | :----------------------------: |
28+
| SSDLITE_320_MOBILENET_V3_LARGE | 8 | 18 |
2029

2130
## Style Transfer
2231

23-
| Model | iPhone 17 Pro (Core ML) [ms] | iPhone 16 Pro (Core ML) [ms] | iPhone SE 3 (Core ML) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
24-
| ---------------------------- | :--------------------------: | :--------------------------: | :------------------------: | :-------------------------------: | :-----------------------: |
25-
| STYLE_TRANSFER_CANDY | 1400 | 1485 | 4255 | 2510 | 2355 |
26-
| STYLE_TRANSFER_MOSAIC | 1400 | 1485 | 4255 | 2510 | 2355 |
27-
| STYLE_TRANSFER_UDNIE | 1400 | 1485 | 4255 | 2510 | 2355 |
28-
| STYLE_TRANSFER_RAIN_PRINCESS | 1400 | 1485 | 4255 | 2510 | 2355 |
32+
:::info
33+
Times presented below are _model inference times only_ and do not include time taken for pre-processing (e.g. image resizing, normalization) or post-processing (e.g. image resizing) which are dependent on input size.
34+
:::
35+
36+
| Model | iPhone 17 Pro (Core ML) [ms] | Google Pixel 10 (XNNPACK) [ms] |
37+
| -------------------------------------- | :--------------------------: | :----------------------------: |
38+
| STYLE_TRANSFER_CANDY | 100 | 1025 |
39+
| STYLE_TRANSFER_MOSAIC | 100 | 1025 |
40+
| STYLE_TRANSFER_UDNIE | 100 | 1025 |
41+
| STYLE_TRANSFER_RAIN_PRINCESS | 100 | 1025 |
42+
| STYLE_TRANSFER_CANDY_QUANTIZED | 150 | 430 |
43+
| STYLE_TRANSFER_MOSAIC_QUANTIZED | 150 | 430 |
44+
| STYLE_TRANSFER_UDNIE_QUANTIZED | 150 | 430 |
45+
| STYLE_TRANSFER_RAIN_PRINCESS_QUANTIZED | 150 | 430 |
2946

3047
## OCR
3148

@@ -109,23 +126,39 @@ Benchmark times for text embeddings are highly dependent on the sentence length.
109126

110127
## Image Embeddings
111128

112-
| Model | iPhone 17 Pro (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
113-
| --------------------------- | :--------------------------: | :-----------------------: |
114-
| CLIP_VIT_BASE_PATCH32_IMAGE | 18 | 55 |
129+
:::info
130+
Times presented below are _model inference times only_ and do not include time taken for pre-processing (e.g. image resizing, normalization) or post-processing (e.g. image resizing) which are dependent on input size.
131+
:::
115132

116133
:::info
117-
Image embedding benchmark times are measured using 224×224 pixel images, as required by the model. All input images, whether larger or smaller, are resized to 224×224 before processing. Resizing is typically fast for small images but may be noticeably slower for very large images, which can increase total inference time.
134+
Image embedding benchmark times are measured using 224×224 pixel images, as required by the model. All input images, whether larger or smaller, are resized to 224×224 before processing. Resizing is typically fast for small images but may be noticeably slower for very large images, which can increase total time.
118135
:::
119136

137+
| Model | iPhone 17 Pro (XNNPACK) [ms] | Google Pixel 10 (XNNPACK) [ms] |
138+
| ------------------------------------- | :--------------------------: | :----------------------------: |
139+
| CLIP_VIT_BASE_PATCH32_IMAGE | 14 | 68 |
140+
| CLIP_VIT_BASE_PATCH32_IMAGE_QUANTIZED | 11 | 31 |
141+
120142
## Semantic Segmentation
121143

122-
:::warning
123-
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization.
144+
:::info
145+
Times presented below are _model inference times only_ and do not include time taken for pre-processing (e.g. image resizing, normalization) or post-processing (e.g. image resizing) which are dependent on input size.
124146
:::
125147

126-
| Model | iPhone 16 Pro (Core ML) [ms] | iPhone 14 Pro Max (Core ML) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] |
127-
| ----------------- | ---------------------------- | -------------------------------- | --------------------------------- |
128-
| DEELABV3_RESNET50 | 1000 | 670 | 700 |
148+
| Model | iPhone 17 Pro (XNNPACK) [ms] | Google Pixel 10 (XNNPACK) [ms] |
149+
| --------------------------------------- | :--------------------------: | :----------------------------: |
150+
| DEEPLAB_V3_RESNET50 | 2000 | 2200 |
151+
| DEEPLAB_V3_RESNET50_QUANTIZED | 118 | 380 |
152+
| DEEPLAB_V3_RESNET101 | 2900 | 3300 |
153+
| DEEPLAB_V3_RESNET101_QUANTIZED | 174 | 660 |
154+
| DEEPLAB_V3_MOBILENET_V3_LARGE | 131 | 153 |
155+
| DEEPLAB_V3_MOBILENET_V3_LARGE_QUANTIZED | 17 | 40 |
156+
| LRASPP_MOBILENET_V3_LARGE | 13 | 36 |
157+
| LRASPP_MOBILENET_V3_LARGE_QUANTIZED | 12 | 20 |
158+
| FCN_RESNET50 | 1800 | 2160 |
159+
| FCN_RESNET50_QUANTIZED | 100 | 320 |
160+
| FCN_RESNET101 | 2600 | 3160 |
161+
| FCN_RESNET101_QUANTIZED | 160 | 620 |
129162

130163
## Text to image
131164

‎docs/docs/02-benchmarks/model-size.md‎

Lines changed: 23 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -4,24 +4,24 @@ title: Model Size
44

55
## Classification
66

7-
| Model | XNNPACK [MB] | Core ML [MB] |
8-
| ----------------- | :----------: | :----------: |
9-
| EFFICIENTNET_V2_S | 85.6 | 43.9 |
7+
| Model | XNNPACK FP32 [MB] | XNNPACK INT8 [MB] | Core ML FP32 [MB] | Core ML FP16 [MB] |
8+
| :---------------- | :---------------: | :---------------: | :---------------: | :---------------: |
9+
| EFFICIENTNET_V2_S | 85.7 | 22.9 | 86.5 | 43.9 |
1010

1111
## Object Detection
1212

13-
| Model | XNNPACK [MB] |
14-
| ------------------------------ | :----------: |
15-
| SSDLITE_320_MOBILENET_V3_LARGE | 13.9 |
13+
| Model | XNNPACK FP32 [MB] | Core ML FP32 [MB] | Core ML FP16 [MB] |
14+
| ------------------------------ | :---------------: | :---------------: | :---------------: |
15+
| SSDLITE_320_MOBILENET_V3_LARGE | 13.9 | 15.6 | 8.46 |
1616

1717
## Style Transfer
1818

19-
| Model | XNNPACK [MB] | Core ML [MB] |
20-
| ---------------------------- | :----------: | :----------: |
21-
| STYLE_TRANSFER_CANDY | 6.78 | 5.22 |
22-
| STYLE_TRANSFER_MOSAIC | 6.78 | 5.22 |
23-
| STYLE_TRANSFER_UDNIE | 6.78 | 5.22 |
24-
| STYLE_TRANSFER_RAIN_PRINCESS | 6.78 | 5.22 |
19+
| Model | XNNPACK FP32 [MB] | XNNPACK INT8 [MB] | Core ML FP32 [MB] | Core ML FP16 [MB] |
20+
| ---------------------------- | :---------------: | :---------------: | :---------------: | :---------------: |
21+
| STYLE_TRANSFER_CANDY | 6.82 | 1.84 | 7.12 | 3.79 |
22+
| STYLE_TRANSFER_MOSAIC | 6.82 | 1.84 | 7.12 | 3.79 |
23+
| STYLE_TRANSFER_UDNIE | 6.82 | 1.84 | 7.12 | 3.79 |
24+
| STYLE_TRANSFER_RAIN_PRINCESS | 6.82 | 1.84 | 7.12 | 3.79 |
2525

2626
## OCR
2727

@@ -82,15 +82,20 @@ title: Model Size
8282

8383
## Image Embeddings
8484

85-
| Model | XNNPACK [MB] |
86-
| --------------------------- | :----------: |
87-
| CLIP_VIT_BASE_PATCH32_IMAGE | 352 |
85+
| Model | XNNPACK FP32 [MB] | XNNPACK INT8 [MB] |
86+
| --------------------------- | :---------------: | :---------------: |
87+
| CLIP_VIT_BASE_PATCH32_IMAGE | 352 | 96.4 |
8888

8989
## Semantic Segmentation
9090

91-
| Model | XNNPACK [MB] |
92-
| ----------------- | ------------ |
93-
| DEELABV3_RESNET50 | 168 |
91+
| Model | XNNPACK FP32 [MB] | XNNPACK INT8 [MB] |
92+
| ----------------------------- | :---------------: | :---------------: |
93+
| DEEPLAB_V3_RESNET50 | 168 | 42.4 |
94+
| DEEPLAB_V3_RESNET101 | 244 | 61.7 |
95+
| DEEPLAB_V3_MOBILENET_V3_LARGE | 44.1 | 11.4 |
96+
| LRASPP_MOBILENET_V3_LARGE | 12.9 | 3.53 |
97+
| FCN_RESNET50 | 141 | 35.7 |
98+
| FCN_RESNET101 | 217 | 55 |
9499

95100
## Text to image
96101

0 commit comments

Comments
 (0)