Skip to content

feat: handle all orientations when using vision camera#980

Merged
NorbertKlockiewicz merged 20 commits intomainfrom
@nk/vision-camera-orientation
Mar 20, 2026
Merged

feat: handle all orientations when using vision camera#980
NorbertKlockiewicz merged 20 commits intomainfrom
@nk/vision-camera-orientation

Conversation

@NorbertKlockiewicz
Copy link
Copy Markdown
Contributor

@NorbertKlockiewicz NorbertKlockiewicz commented Mar 18, 2026

Description

  • Add FrameTransform utilities (rotateFrameForModel, inverseRotateBbox, inverseRotateMat, inverseRotatePoints) that pre-rotate camera frames so vision models always see upright images, then map results back to screen space
  • Support all device orientations (portrait, landscape-left/right, upside-down), front/back cameras, and iOS vs Android platform differences
  • Pass isFrontCamera via Synchronizable (worklet-compatible) instead of React props, with iOS front-camera 180° fix (#if __APPLE__ guarded)
  • Add 342 lines of unit tests covering all orientations for rotate/inverse functions
  • Add example app tasks for OCR and style transfer, update existing tasks

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

Run computer vision example app, test all models (back/front camera), rotate the phone

Screenshots

Related issues

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

@NorbertKlockiewicz NorbertKlockiewicz self-assigned this Mar 18, 2026
@NorbertKlockiewicz NorbertKlockiewicz added 3rd party package Issue related to 3rd party packages, but not ExecuTorch, e.g. Expo improvement PRs or issues focused on improvements in the current codebase labels Mar 18, 2026
@NorbertKlockiewicz NorbertKlockiewicz linked an issue Mar 18, 2026 that may be closed by this pull request
@NorbertKlockiewicz NorbertKlockiewicz changed the title @nk/vision camera orientation feat: handle all orientations when using vision camera Mar 19, 2026
@NorbertKlockiewicz NorbertKlockiewicz force-pushed the @nk/vision-camera-orientation branch from e022520 to 639cd9c Compare March 19, 2026 15:12
@NorbertKlockiewicz NorbertKlockiewicz marked this pull request as ready for review March 19, 2026 15:21
@NorbertKlockiewicz NorbertKlockiewicz force-pushed the @nk/vision-camera-orientation branch from f5be4f9 to 7bea029 Compare March 20, 2026 08:46
@msluszniak

This comment was marked as off-topic.

Copy link
Copy Markdown
Member

@msluszniak msluszniak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had only physical android device, so someone needs to test it on iOS, but everything worked as expected.

Add comment explaining isMirrored property handling in VisionModule.
…isionModule.ts

Co-authored-by: Mateusz Sluszniak <56299341+msluszniak@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

@benITo47 benITo47 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on iOS - works seamlessly!
No more concerns from my side.
LGTM!

@NorbertKlockiewicz NorbertKlockiewicz merged commit 2d535a4 into main Mar 20, 2026
5 checks passed
@NorbertKlockiewicz NorbertKlockiewicz deleted the @nk/vision-camera-orientation branch March 20, 2026 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3rd party package Issue related to 3rd party packages, but not ExecuTorch, e.g. Expo improvement PRs or issues focused on improvements in the current codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vision camera integration - figure out orientation handling

3 participants