feat: port style transfer implementation to C++ by JakubGonera · Pull Request #229 · software-mansion/react-native-executorch

JakubGonera · 2025-04-29T18:29:13Z

Port native implementation of style transfer to C++ #227. This is a part of a larger effort to merge native code from Kotlin and Objective C to a single implementation in C++.

Changes by sections

Typescript

Old style transfer modules have been replaced by new, stateful implementation. For this purpose non-static versions of useModule and BaseModule have been introduced.

Objective C & Kotlin

Old implementations of style transfer have been removed. Added methods for handling fetching data via https that are passed on as function objects to C++.

C++

Ported image processing capabilities equivalent to the ObjC/Kotlin implementations.
ModelHostObject serves as an automatic interface between model implementations and JSI. This is done by template metaprogramming; defining methods in JsiConversions.h for types used by the model is necessary for it to work.
A factory method for loading a style transfer model is registered in RnExecutorchInstaller.

Code attribution

common/rnexecutorch/jsi originates from react-native-audio-api
base64.* is used for converting from base64 due to https://renenyffenegger.ch/notes/development/Base64/Encoding-and-decoding-base-64-with-cpp/
ada is a header-only library for parsing urls due to https://github.com/ada-url/ada

Dependencies introduced/updated

headers for Executorch 0.6 have been updated
OpenCV 4.11.0 dependency is introduced for Android/C++. For iOS the version is bumped via Cocoapods to 4.11.0.
C10 headers are the dependency of OpenCV.

Known issues

Exceptions in native C++ do not result in promise getting rejected when forwarding yet.
Garbage collection can malfunction for host objects. C++ host objects need to notify JS runtime about their size via external memory pressure for the garbage collector to know to free them. This is not yet done.

chmjkb · 2025-05-09T07:51:27Z

+  std::filesystem::path filePath = tempDir / filename;
+
+  if (!cv::imwrite(filePath.string(), image)) {
+    throw std::runtime_error("Failed to save the image: " + filePath.string());


Btw, how are those exceptions resolved on the JS side?

All of these have to be caught and currently they are then used to reject a promise -- if they are not caught we will get a runtime error that might crash the app. If the code throwing an exception is not called via a promise, but we still want the error to be pushed to JS, a JSError needs to be created and registered with the JS runtime. I could add a helper function to throw JSError with a C++ exception, the question is whether to do this in this PR. Thoughts?

sounds like a useful helper function to me

It turns out that it's sufficient to throw a JSError with the runtime attached to make the exception readable in JS. The only place where we call the native code without a promise is RnExecutorchInstaller, so right now we catch there any exceptions and rethrow them as JSErorrs.

chmjkb · 2025-05-13T10:23:33Z

      modelSource
    );
-    return global.loadStyleTransfer(loadedPaths[0] || '');
+    this.nativeModule = global.loadStyleTransfer(paths[0] || '');


Make sure this actually works when loading fails

isn't global.loadStyleTransfer possibly null?

Indeed it could be. The question is whether we could do anything about this. Right now the user will have an exception thrown at startup with a message indicating this. If this is null then this means that all the native code went out the window.

mkopcins · 2025-05-14T09:25:05Z

+  std::filesystem::path filePath = tempDir / filename;
+
+  if (!cv::imwrite(filePath.string(), image)) {
+    throw std::runtime_error("Failed to save the image: " + filePath.string());


sounds like a useful helper function to me

mkopcins · 2025-05-14T09:35:59Z

+    while (std::getline(uriStream, stringData, ',')) {
+      if (segmentIndex == 1)
+        break;
+      ++segmentIndex;
+    }
+    if (segmentIndex != 1) {


looking at this, am I right that if uriStream is malformed so that it contains multiple comas this breaks? Shouldn't we parse whole uriStream?

right now segmentIndex is either 0 or 1

std::getline will fail if there are two or more consecutive separators.

So that's a good point, that we need to somehow check for some malformed uris.

Right now if the user supplies a string that has two commas next to each other we will pass an empty string to OpenCV which will return an empty image which in turn will result in an exception "Read image error: invalid argument" being thrown. The same holds for other cases where the payload does not represent a valid image. The exception will then result in a rejected promise.

I don't think that we should do any more work here on parsing. As far as I understand URI specification everything after the comma should be a base64 payload, and if the user is not conforming to this standard it shouldn't make any difference whether we fail while parsing a part of the intended payload or while parsing the whole.

mkopcins · 2025-05-14T10:49:12Z

      modelSource
    );
-    return global.loadStyleTransfer(loadedPaths[0] || '');
+    this.nativeModule = global.loadStyleTransfer(paths[0] || '');


isn't global.loadStyleTransfer possibly null?

Port native implementation of style transfer to C++ #227. This is a part of a larger effort to merge native code from Kotlin and Objective C to a single implementation in C++. Old style transfer modules have been replaced by new, stateful implementation. For this purpose non-static versions of useModule and BaseModule have been introduced. Old implementations of style transfer have been removed. Added methods for handling fetching data via https that are passed on as function objects to C++. - Ported image processing capabilities equivalent to the ObjC/Kotlin implementations. - `ModelHostObject` serves as an automatic interface between model implementations and JSI. This is done by template metaprogramming; defining methods in `JsiConversions.h` for types used by the model is necessary for it to work. - A factory method for loading a style transfer model is registered in `RnExecutorchInstaller`. - `common/rnexecutorch/jsi` originates from react-native-audio-api - `base64.*` is used for converting from base64 due to https://renenyffenegger.ch/notes/development/Base64/Encoding-and-decoding-base-64-with-cpp/ - `ada` is a header-only library for parsing urls due to https://github.com/ada-url/ada - headers for Executorch 0.6 have been updated - OpenCV 4.11.0 dependency is introduced for Android/C++. For iOS the version is bumped via Cocoapods to 4.11.0. - C10 headers are the dependency of OpenCV. - Exceptions in native C++ do not result in promise getting rejected when forwarding yet. - Garbage collection can malfunction for host objects. C++ host objects need to notify JS runtime about their size via external memory pressure for the garbage collector to know to free them. This is not yet done.

JakubGonera force-pushed the @jakubgonera/cpp branch from 304f24d to 08b71a8 Compare April 30, 2025 06:08

JakubGonera marked this pull request as draft April 30, 2025 08:25

chmjkb linked an issue Apr 30, 2025 that may be closed by this pull request

Port style transfer model to C++ #227

Closed

JakubGonera force-pushed the @jakubgonera/style-transfer-cpp branch from b593b51 to cfe8795 Compare April 30, 2025 09:09

JakubGonera marked this pull request as ready for review April 30, 2025 09:21

msluszniak reviewed Apr 30, 2025

View reviewed changes

Comment thread common/rnexecutorch/data_processing/ImageProcessing.cpp Outdated

Comment thread common/rnexecutorch/host_objects/ModelHostObject.h

Comment thread common/rnexecutorch/models/BaseModel.cpp Outdated

JakubGonera requested review from NorbertKlockiewicz and mkopcins May 5, 2025 11:02

chmjkb reviewed May 9, 2025

View reviewed changes

NorbertKlockiewicz requested changes May 9, 2025

View reviewed changes

Comment thread common/rnexecutorch/data_processing/ImageProcessing.cpp

Comment thread common/rnexecutorch/models/BaseModel.cpp Outdated

Comment thread common/rnexecutorch/models/StyleTransfer.cpp Outdated

Comment thread common/rnexecutorch/models/StyleTransfer.cpp Outdated

JakubGonera requested review from NorbertKlockiewicz and chmjkb May 13, 2025 09:26

chmjkb reviewed May 13, 2025

View reviewed changes

chmjkb approved these changes May 14, 2025

View reviewed changes

mkopcins requested changes May 14, 2025

View reviewed changes

JakubGonera requested a review from mkopcins May 15, 2025 09:05

JakubGonera added 10 commits May 19, 2025 09:26

feat: Add ET library for direct use in C++

1409cdc

fix: remove ETInstaller module wrapper

4fce9f4

chore: remove additional executorch archive

9175886

feat: port style transfer to C++

d609e8b

fix: address C++ comments

3820e6f

enhancement: add additional module wrapping on the native module

072a624

fix: propagate exceptions with full info to JS

a4c5872

fix: address some of the review comments

beb88dc

fix: address the rest of the comments

6567e87

fix: resolve duplicate .so files

a991245

JakubGonera force-pushed the @jakubgonera/style-transfer-cpp branch from c8cbe4a to a991245 Compare May 19, 2025 07:57

JakubGonera added 3 commits May 20, 2025 10:27

fix: revert excluding executorch .so

29b9bd6

fix: conflicting enum id on iOS

9d5e53b

fix: check for multiple commas in a base64 URI

68d2715

mkopcins approved these changes May 20, 2025

View reviewed changes

JakubGonera merged commit 1d3224c into @jakubgonera/cpp May 20, 2025
2 checks passed

JakubGonera deleted the @jakubgonera/style-transfer-cpp branch May 20, 2025 13:13

Conversation

JakubGonera commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes by sections

Typescript

Objective C & Kotlin

C++

Code attribution

Dependencies introduced/updated

Known issues

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JakubGonera commented Apr 29, 2025 •

edited

Loading