Commit 4279ad4
feat!: integrate all vision models with vision camera (#880)
## Description
- Added `PixelData` input support to all vision model hooks and modules
— `forward()` now accepts both string URIs and raw pixel buffers
- Added `runOnFrame` worklet to all vision model hooks for real-time
VisionCamera v5 frame processing (`useClassification`,
`useImageEmbeddings`, `useOCR`, `useVerticalOCR`, `useObjectDetection`,
`useSemanticSegmentation`, `useStyleTransfer`)
- Refactored `StyleTransfer` C++ to unified `generateFromString` /
`generateFromPixels` with a `saveToFile` flag
- Added new VisionCamera integration docs page with setup guide, full
example, Module API pattern, and common issues section
- Updated all vision model hook and module docs to document `PixelData`
input and `runOnFrame`
- Added integration tests: `VisionModelTest` (unit tests for
`extractFromPixels` and `preprocess`), `FrameProcessorTest` (unit tests
for `pixelsToMat`), `StyleTransferTest` (full coverage of both output
modes), and `generateFromPixels` smoke tests across all vision models
### Breaking changes
#### `StyleTransferType.forward` return type changed
Before:
```typescript
forward(imageSource: string): Promise<string>
```
After:
```typescript
forward(input: string | PixelData, output?: 'pixelData' | 'url'): Promise<PixelData | string>
```
### Known Issues
The orientation handling is static right now, it will be fixed in the
next PR.
### Introduces a breaking change?
- [x] Yes
- [ ] No
### Type of change
- [ ] Bug fix (change which fixes an issue)
- [x] New feature (change which adds functionality)
- [x] Documentation update (improves or adds clarity to existing
documentation)
- [ ] Other (chores, tests, code style improvements etc.)
### Tested on
- [x] iOS
- [x] Android
### Testing instructions
- [x] Run computer vision app -> vision camera screen -> test all models
- [ ] Also test rest of the models in this app.
- [ ] Run test suite
### Screenshots
<!-- Add screenshots here, if applicable -->
### Related issues
<!-- Link related issues here using #issue-number -->
### Checklist
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have updated the documentation accordingly
- [ ] My changes generate no new warnings
### Additional notes
<!-- Include any additional information, assumptions, or context that
reviewers might need to understand this PR. -->
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Jakub Chmura <92989966+chmjkb@users.noreply.github.com>1 parent 3397b78 commit 4279ad4
88 files changed
Lines changed: 3074 additions & 940 deletions
File tree
- apps/computer-vision
- app
- object_detection_live
- style_transfer
- vision_camera
- components/vision_camera
- tasks
- utils
- docs/docs
- 03-hooks/02-computer-vision
- 04-typescript-api/02-computer-vision
- packages/react-native-executorch
- common/rnexecutorch
- data_processing
- host_objects
- models
- classification
- embeddings/image
- object_detection
- ocr
- semantic_segmentation
- style_transfer
- vertical_ocr
- tests
- integration
- unit
- utils
- src
- controllers
- hooks
- computer_vision
- modules
- computer_vision
- types
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
62 | 71 | | |
63 | 72 | | |
64 | 73 | | |
| |||
83 | 92 | | |
84 | 93 | | |
85 | 94 | | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | 95 | | |
95 | 96 | | |
96 | 97 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
14 | 20 | | |
15 | 21 | | |
16 | 22 | | |
| |||
29 | 35 | | |
30 | 36 | | |
31 | 37 | | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| |||
Lines changed: 0 additions & 222 deletions
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
| 21 | + | |
| 22 | + | |
20 | 23 | | |
21 | 24 | | |
22 | 25 | | |
23 | 26 | | |
24 | | - | |
| 27 | + | |
| 28 | + | |
25 | 29 | | |
26 | 30 | | |
27 | 31 | | |
28 | 32 | | |
29 | 33 | | |
30 | 34 | | |
31 | | - | |
32 | | - | |
| 35 | + | |
| 36 | + | |
33 | 37 | | |
34 | 38 | | |
35 | 39 | | |
| |||
52 | 56 | | |
53 | 57 | | |
54 | 58 | | |
55 | | - | |
56 | | - | |
57 | | - | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
58 | 64 | | |
59 | 65 | | |
60 | 66 | | |
| |||
0 commit comments