Skip to content

Commit 4279ad4

Browse files
NorbertKlockiewiczclaudechmjkb
authored
feat!: integrate all vision models with vision camera (#880)
## Description - Added `PixelData` input support to all vision model hooks and modules — `forward()` now accepts both string URIs and raw pixel buffers - Added `runOnFrame` worklet to all vision model hooks for real-time VisionCamera v5 frame processing (`useClassification`, `useImageEmbeddings`, `useOCR`, `useVerticalOCR`, `useObjectDetection`, `useSemanticSegmentation`, `useStyleTransfer`) - Refactored `StyleTransfer` C++ to unified `generateFromString` / `generateFromPixels` with a `saveToFile` flag - Added new VisionCamera integration docs page with setup guide, full example, Module API pattern, and common issues section - Updated all vision model hook and module docs to document `PixelData` input and `runOnFrame` - Added integration tests: `VisionModelTest` (unit tests for `extractFromPixels` and `preprocess`), `FrameProcessorTest` (unit tests for `pixelsToMat`), `StyleTransferTest` (full coverage of both output modes), and `generateFromPixels` smoke tests across all vision models ### Breaking changes #### `StyleTransferType.forward` return type changed Before: ```typescript forward(imageSource: string): Promise<string> ``` After: ```typescript forward(input: string | PixelData, output?: 'pixelData' | 'url'): Promise<PixelData | string> ``` ### Known Issues The orientation handling is static right now, it will be fixed in the next PR. ### Introduces a breaking change? - [x] Yes - [ ] No ### Type of change - [ ] Bug fix (change which fixes an issue) - [x] New feature (change which adds functionality) - [x] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [x] iOS - [x] Android ### Testing instructions - [x] Run computer vision app -> vision camera screen -> test all models - [ ] Also test rest of the models in this app. - [ ] Run test suite ### Screenshots <!-- Add screenshots here, if applicable --> ### Related issues <!-- Link related issues here using #issue-number --> ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [ ] My changes generate no new warnings ### Additional notes <!-- Include any additional information, assumptions, or context that reviewers might need to understand this PR. --> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Jakub Chmura <92989966+chmjkb@users.noreply.github.com>
1 parent 3397b78 commit 4279ad4

88 files changed

Lines changed: 3074 additions & 940 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

apps/computer-vision/app/_layout.tsx

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,15 @@ export default function _layout() {
5959
headerTitleStyle: { color: ColorPalette.primary },
6060
}}
6161
>
62+
<Drawer.Screen
63+
name="vision_camera/index"
64+
options={{
65+
drawerLabel: 'Vision Camera',
66+
title: 'Vision Camera',
67+
headerShown: false,
68+
headerTitleStyle: { color: ColorPalette.primary },
69+
}}
70+
/>
6271
<Drawer.Screen
6372
name="classification/index"
6473
options={{
@@ -83,14 +92,6 @@ export default function _layout() {
8392
headerTitleStyle: { color: ColorPalette.primary },
8493
}}
8594
/>
86-
<Drawer.Screen
87-
name="object_detection_live/index"
88-
options={{
89-
drawerLabel: 'Object Detection (Live)',
90-
title: 'Object Detection (Live)',
91-
headerTitleStyle: { color: ColorPalette.primary },
92-
}}
93-
/>
9495
<Drawer.Screen
9596
name="ocr/index"
9697
options={{

apps/computer-vision/app/index.tsx

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,12 @@ export default function Home() {
1111
<ExecutorchLogo width={64} height={64} />
1212
<Text style={styles.headerText}>Select a demo model</Text>
1313
<View style={styles.buttonContainer}>
14+
<TouchableOpacity
15+
style={styles.button}
16+
onPress={() => router.navigate('vision_camera/')}
17+
>
18+
<Text style={styles.buttonText}>Vision Camera</Text>
19+
</TouchableOpacity>
1420
<TouchableOpacity
1521
style={styles.button}
1622
onPress={() => router.navigate('classification/')}
@@ -29,12 +35,6 @@ export default function Home() {
2935
>
3036
<Text style={styles.buttonText}>Object Detection</Text>
3137
</TouchableOpacity>
32-
<TouchableOpacity
33-
style={styles.button}
34-
onPress={() => router.navigate('object_detection_live/')}
35-
>
36-
<Text style={styles.buttonText}>Object Detection Live</Text>
37-
</TouchableOpacity>
3838
<TouchableOpacity
3939
style={styles.button}
4040
onPress={() => router.navigate('ocr/')}

apps/computer-vision/app/object_detection_live/index.tsx

Lines changed: 0 additions & 222 deletions
This file was deleted.

apps/computer-vision/app/style_transfer/index.tsx

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,20 +16,24 @@ export default function StyleTransferScreen() {
1616
useEffect(() => {
1717
setGlobalGenerating(model.isGenerating);
1818
}, [model.isGenerating, setGlobalGenerating]);
19+
1920
const [imageUri, setImageUri] = useState('');
21+
const [styledUri, setStyledUri] = useState('');
22+
2023
const handleCameraPress = async (isCamera: boolean) => {
2124
const image = await getImage(isCamera);
2225
const uri = image?.uri;
2326
if (typeof uri === 'string') {
24-
setImageUri(uri as string);
27+
setImageUri(uri);
28+
setStyledUri('');
2529
}
2630
};
2731

2832
const runForward = async () => {
2933
if (imageUri) {
3034
try {
31-
const output = await model.forward(imageUri);
32-
setImageUri(output);
35+
const uri = await model.forward(imageUri, 'url');
36+
setStyledUri(uri);
3337
} catch (e) {
3438
console.error(e);
3539
}
@@ -52,9 +56,11 @@ export default function StyleTransferScreen() {
5256
style={styles.image}
5357
resizeMode="contain"
5458
source={
55-
imageUri
56-
? { uri: imageUri }
57-
: require('../../assets/icons/executorch_logo.png')
59+
styledUri
60+
? { uri: styledUri }
61+
: imageUri
62+
? { uri: imageUri }
63+
: require('../../assets/icons/executorch_logo.png')
5864
}
5965
/>
6066
</View>

0 commit comments

Comments
 (0)