feat: add integration with vision camera v5#810
Conversation
NorbertKlockiewicz
changed the title
NorbertKlockiewicz
force-pushed
the
@nk/vision-camera
branch
2 times, most recently
from
96f2c14 to
f3e17e2
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
NorbertKlockiewicz
changed the title
NorbertKlockiewicz
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
NorbertKlockiewicz
force-pushed
the
@nk/vision-camera
branch
3 times, most recently
from
c1ff490 to
983242e
Compare
msluszniak
left a comment
msluszniak
left a comment
There was a problem hiding this comment.
LGTM, but I would wait for a review from someone that knows TS more than me ;)
| /** | ||
| * Thrown when a feature or platform is not supported in the current environment. | ||
| */ | ||
| PlatformNotSupported = 119, |
There was a problem hiding this comment.
Make sure to add this to error handling docs
| model, | ||
| preventLoad: preventLoad, | ||
| }); | ||
| }) as ObjectDetectionType; |
There was a problem hiding this comment.
do we need the type assertion here?
NorbertKlockiewicz
force-pushed
the
@nk/vision-camera
branch
from
74be0c6 to
28a7bad
Compare
|
I think you need to rebase because your changes to auto-generated api is as for the older code (before my commit with packages bump). |
msluszniak
force-pushed
the
@nk/vision-camera
branch
from
02bb7aa to
6ee9f4d
Compare
|
The memory leak we've catched is caused by vision camera and hopefully will be fixed soon. The PR should be now ready to merge, just need an approve from @chmjkb |
chmjkb
left a comment
chmjkb
left a comment
There was a problem hiding this comment.
lgtm, good job 👏🏻
make sure to add the issue we talked about today regarding weird orientation behavior cc @NorbertKlockiewicz
| cv::Mat extractFrame(jsi::Runtime &runtime, const jsi::Object &frameData) { | ||
| if (!hasNativeBuffer(runtime, frameData)) { | ||
| throw RnExecutorchError(RnExecutorchErrorCode::InvalidUserInput, | ||
| "FrameProcessor: No nativeBuffer found in frame"); | ||
| } | ||
|
|
||
| auto nativeBufferValue = frameData.getProperty(runtime, "nativeBuffer"); | ||
| uint64_t bufferPtr = static_cast<uint64_t>( | ||
| nativeBufferValue.asBigInt(runtime).asUint64(runtime)); | ||
|
|
||
| return extractFromNativeBuffer(bufferPtr); | ||
| } |
There was a problem hiding this comment.
Are you releasing the NativeBuffer again via .release()?
I think it might make sense to either do this in JS via a helper worklet function (simpler), or in native C++ (where you'd have to carry the jsi::Function that does the release() along yourself).
There was a problem hiding this comment.
Maybe this is what you observed as a memory leak.
NativeBuffer is a +1 retained buffer, you need to call release() on it. I do this in <SkiaCamera /> too.
4 participants
Description
This PR adds real-time camera frame processing capabilities to React Native ExecuTorch by integrating with VisionCamera v5. It introduces a new VisionModule base class and extends ObjectDetectionModule to support three input methods: image URLs/paths, raw pixel data (PixelData), and live camera frames.
This PR includes:
Introduces a breaking change?
Type of change
Tested on
Testing instructions
Run the computer vision example app and test object detection and object detection (live)
Screenshots
Related issues
Checklist
Additional notes