Commit 814ed10
Add ImageProcessor library to ExecuTorch (#19967)
Summary:
Introduces an ImageProcessor library to ExecuTorch: a small, well-tested
core for turning camera/decoded image buffers into normalized model-input
tensors. This initial implementation is fully portable; performant
platform-specific backends (e.g. Apple Accelerate/Core Image) are added in
follow-up diffs behind the same API.
Given source pixels (BGRA/RGBA bytes or semi-planar YUV planes), the
processor crops/resizes to the configured target and emits a
[1, 3, target_height, target_width] float tensor, normalized per the
configured scheme.
Configuration (`ImageProcessorConfig`):
- target_width / target_height: output spatial size.
- resize_mode: how the source is fit to the target.
- letterbox_anchor: placement of content when letterboxing.
- pad_value: fill value for letterbox padding.
- normalization: how uint8 [0,255] pixels map to floats.
- gpu_min_input_pixels: minimum source pixel count (width * height) at which
the GPU path is used; smaller inputs run on CPU. Sentinels: 0 = always GPU,
INT64_MAX = always CPU (defaults to ~1080p, i.e. 1920*1080 + 1). Platform
backends consult this to pick CPU vs GPU. (Note GPU is always platform optional; but GPU will never be routed when below gpu_min_input_pixels.)
Enum options:
Enum options:
- **ColorFormat** (`BGRA`, `RGBA`): interleaved 8-bit input layout.
- **YUVFormat** (`NV12`, `NV21`): semi-planar YUV input layout.
- **ResizeMode** (`STRETCH`, `LETTERBOX`): `STRETCH` fills the target, ignoring aspect ratio; `LETTERBOX` preserves aspect ratio and pads.
- **LetterboxAnchor** (`CENTER`, `TOP_LEFT`): where padded content is anchored.
- **Orientation** (`UP`): source orientation (upright).
- **Normalization** (`zeroToOne`, `imagenet`): `zeroToOne` scales to [0, 1]; `imagenet` additionally applies per-channel mean/std.
Example:
```
#include <executorch/extension/image/image_processor.h>
using namespace executorch::extension::image;
// Resize to 224x224, letterboxed, with ImageNet normalization.
ImageProcessorConfig config;
config.target_width = 224;
config.target_height = 224;
config.resize_mode = ResizeMode::LETTERBOX;
config.normalization = Normalization::imagenet();
ImageProcessor processor(config);
// RGBA/BGRA input -> normalized [1, 3, 224, 224] float tensor.
auto result = processor.process(
pixels, width, height, /*stride_bytes=*/width * 4, ColorFormat::RGBA);
if (result.ok()) {
TensorPtr tensor = std::move(result.get());
// feed `tensor` to the model...
}
// Semi-planar YUV (e.g. NV12 camera frame):
auto yuv = processor.process_yuv(
y_plane, y_stride, uv_plane, uv_stride, width, height, YUVFormat::NV12);
```
Reviewed By: psiddh
Differential Revision: D1068984211 parent fb1e212 commit 814ed10
16 files changed
Lines changed: 2268 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
864 | 864 | | |
865 | 865 | | |
866 | 866 | | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
867 | 872 | | |
868 | 873 | | |
869 | 874 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
0 commit comments