Skip to content

Commit b7adfba

Browse files
metascroyfacebook-github-bot
authored andcommitted
Add ImageProcessor library to ExecuTorch (pytorch#19967)
Summary: Introduces an ImageProcessor library to ExecuTorch: a small, well-tested core for turning camera/decoded image buffers into normalized model-input tensors. This initial implementation is fully portable; performant platform-specific backends (e.g. Apple Accelerate/Core Image) are added in follow-up diffs behind the same API. Given source pixels (BGRA/RGBA bytes or semi-planar YUV planes), the processor crops/resizes to the configured target and emits a [1, 3, target_height, target_width] float tensor, normalized per the configured scheme. Configuration (`ImageProcessorConfig`): - target_width / target_height: output spatial size. - resize_mode: how the source is fit to the target. - letterbox_anchor: placement of content when letterboxing. - pad_value: fill value for letterbox padding. - normalization: how uint8 [0,255] pixels map to floats. - gpu_min_input_pixels: minimum source pixel count (width * height) at which the GPU path is used; smaller inputs run on CPU. Sentinels: 0 = always GPU, INT64_MAX = always CPU (defaults to ~1080p, i.e. 1920*1080 + 1). Platform backends consult this to pick CPU vs GPU. (Note GPU is always platform optional; but GPU will never be routed when below gpu_min_input_pixels.) Enum options: Enum options: - **ColorFormat** (`BGRA`, `RGBA`): interleaved 8-bit input layout. - **YUVFormat** (`NV12`, `NV21`): semi-planar YUV input layout. - **ResizeMode** (`STRETCH`, `LETTERBOX`): `STRETCH` fills the target, ignoring aspect ratio; `LETTERBOX` preserves aspect ratio and pads. - **LetterboxAnchor** (`CENTER`, `TOP_LEFT`): where padded content is anchored. - **Orientation** (`UP`): source orientation (upright). - **Normalization** (`zeroToOne`, `imagenet`): `zeroToOne` scales to [0, 1]; `imagenet` additionally applies per-channel mean/std. Example: ``` #include <executorch/extension/image/image_processor.h> using namespace executorch::extension::image; // Resize to 224x224, letterboxed, with ImageNet normalization. ImageProcessorConfig config; config.target_width = 224; config.target_height = 224; config.resize_mode = ResizeMode::LETTERBOX; config.normalization = Normalization::imagenet(); ImageProcessor processor(config); // RGBA/BGRA input -> normalized [1, 3, 224, 224] float tensor. auto result = processor.process( pixels, width, height, /*stride_bytes=*/width * 4, ColorFormat::RGBA); if (result.ok()) { TensorPtr tensor = std::move(result.get()); // feed `tensor` to the model... } // Semi-planar YUV (e.g. NV12 camera frame): auto yuv = processor.process_yuv( y_plane, y_stride, uv_plane, uv_stride, width, height, YUVFormat::NV12); ``` Differential Revision: D106898421
1 parent 1925a86 commit b7adfba

16 files changed

Lines changed: 2268 additions & 0 deletions

CMakeLists.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -864,6 +864,11 @@ if(EXECUTORCH_BUILD_EXTENSION_TENSOR)
864864
list(APPEND _executorch_extensions extension_tensor)
865865
endif()
866866

867+
if(EXECUTORCH_BUILD_EXTENSION_IMAGE)
868+
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/extension/image)
869+
list(APPEND _executorch_extensions extension_image)
870+
endif()
871+
867872
if(EXECUTORCH_BUILD_PTHREADPOOL AND EXECUTORCH_BUILD_CPUINFO)
868873
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/extension/threadpool)
869874
endif()

extension/image/BUCK

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
load(":targets.bzl", "define_common_targets")
2+
3+
oncall("executorch")
4+
5+
define_common_targets()

extension/image/CMakeLists.txt

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
cmake_minimum_required(VERSION 3.19)
8+
9+
# stb_image_resize: lightweight header-only library used by the resize step in
10+
# image_processor.cpp.
11+
include(FetchContent)
12+
FetchContent_Declare(
13+
stb
14+
GIT_REPOSITORY https://github.com/nothings/stb.git
15+
GIT_TAG f0569113c93ad095470c54bf34a17b36646bbbb5
16+
)
17+
FetchContent_MakeAvailable(stb)
18+
19+
add_library(extension_image image_processor_common.cpp image_processor.cpp)
20+
21+
target_include_directories(
22+
extension_image PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/../..
23+
)
24+
25+
# stb_image_resize.h lives under deprecated/ in current stb. Private: only the
26+
# .cpp uses it, not the installed public headers.
27+
target_include_directories(
28+
extension_image PRIVATE ${stb_SOURCE_DIR} ${stb_SOURCE_DIR}/deprecated
29+
)
30+
31+
target_link_libraries(extension_image PUBLIC executorch_core extension_tensor)
32+
33+
install(
34+
TARGETS extension_image
35+
EXPORT ExecuTorchTargets
36+
DESTINATION lib
37+
)
38+
39+
install(FILES image_processor.h image_processor_config.h
40+
DESTINATION include/executorch/extension/image
41+
)
42+
43+
if(BUILD_TESTING)
44+
add_subdirectory(test)
45+
endif()

extension/image/TARGETS

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
load(":targets.bzl", "define_common_targets")
2+
3+
oncall("executorch")
4+
5+
define_common_targets()

0 commit comments

Comments
 (0)