✨ Migrate to new-arch (react-native 0.74, Fabric/TurboModules/CodeGen/bridgeless) about react-native-vision-camera

[idrisssakhi]

A small patch for those having a build issue. Not sure if the frame processor will continue working after this patch, waiting for your inputs. But the camera will work, and the build issues will be resolved.

diff --git a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/frameprocessors/VisionCameraProxy.kt b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/frameprocessors/VisionCameraProxy.kt
index d697befe..8de418b0 100644
--- a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/frameprocessors/VisionCameraProxy.kt
+++ b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/frameprocessors/VisionCameraProxy.kt
@@ -7,12 +7,14 @@ import com.facebook.jni.HybridData
 import com.facebook.proguard.annotations.DoNotStrip
 import com.facebook.react.bridge.ReactApplicationContext
 import com.facebook.react.bridge.UiThreadUtil
+import com.facebook.react.common.annotations.FrameworkAPI
 import com.facebook.react.turbomodule.core.CallInvokerHolderImpl
 import com.facebook.react.uimanager.UIManagerHelper
 import com.mrousavy.camera.core.ViewNotFoundError
 import com.mrousavy.camera.react.CameraView
 import java.lang.ref.WeakReference
 
+@OptIn(FrameworkAPI::class)
 @Suppress("KotlinJniMissingFunction") // we use fbjni.
 class VisionCameraProxy(private val reactContext: ReactApplicationContext) {
   companion object {

[cipolleschi]

Hi @mrousavy, thanks for this issue, it is super helpful!

1. ❌ I have both a View (CameraView) and two Modules (CameraModule + DevicesModule) in the codebase. Currently, the create-react-native-library template does not have a template for multiple CodeGen specs (views + modules).

This should not be a problem. In your codegenConfig field of the vision-camera package, you can specify all as type, and CodeGen shuld generate the native code for both modules and classes.
In this case, the jsSrcs field should be a parent folder that contains all the specs. For example:

react-native-camera-module
'-> js
    '-> Module
          '-> NativeCameraModule.js
     '-> NativeComponent
         '-> CameraViewComponent.js 

and you pass just js as folder in the jsSrcs.

2. ❌ For the Frame Processor runtime I need access to the jsi::Value/jsi::Function that the user passes to the view directly, instead of converting it to a callback within the TurboModules/Native Modules system because the Frame Processor function is a Worklet. This is how this works currently:
   JS: Camera.tsx (uses findNodeHandle(..) + a global setFrameProcessor(..) func)
   iOS: VisionCameraProxy.mm (uses UIManager::viewForReactTag to find the view)
   Android: VisionCameraProxy.kt (uses UIManagerHelper.getUIManager to find the View)
3. ❌ For the Frame Processor runtime I need access to the jsi::Runtime as early as possible (on demand). Currently I get the jsi::Runtime from the RCTCxxBridge/CatalystInstance, which is a kinda private and unsafe API.
   iOS: VisionCameraInstaller::install
   Android: VisionCameraProxy.kt.

I don't have a strong guidance here, but a coleague is working on 3. and perhaps with is work you'll be able to make 2. works as well? Not sure about this.

1. I want to pass multiple callbacks to a native function. This was a limitation with the Bridge, but should now work with TurboModules I think? In my case startRecording() takes both onRecordingFinished and onRecordingError callbacks, and also returns a Promise which resolves once the recording has actually been started.

I read internally a proposal to make this happen, but I don't have an update on the work here.

2. I couldn't find a clear script to trigger CodeGen (something like npx react-native codegen) to generate both iOS and Android specs.

In 0.74, we have npx react-native codegen, but you need to call it twice, one for iOS and one for Android.
It should be easy to add a

"script" : {
  "run-codegen": "scripts/execute-codegen.sh",
  }

to react-native-vision-camera where you run:

npx react-native codegen <ios params>
npx react-native codegen <android params>

and then you can call

yarn run-codegen

3. New arch doesn't have first class Swift support as far as I know?

No, and it will not have in the short term. To make it work properly, we need to radically change the ReactCommon file structure and it is a long work which we can't prioritize at the moment.

Anyway, it is possible to work around the issue: you should be able to create a thin Objective-C++ layer that:

1. implements the module/components specs.
2. holds a Swift object.
3. forward all the invocation to the held Swift object.

It would be actually very interesting to try this out in a library that is as complex as yours: we might be able to generalize the wrapper somehow... 🤔

4. New arch doesn't support custom "hybrid" objects (so e.g. I could return an in-memory UIImage/Image instance in takePhoto(), instead of writing it to a file and returning a string as that's the only supported type)

I think expo manage to achieve something like this for some of their packages. There are ways to do it probably, but I feel that they might depends heavily on your use case.

Currently only the renderer interop layer can be used, but there are some issues:
#2613

We did a lot of work on interop layers for 0.74. The issue author is not mentioning which version of React Native they are using. It is likely that we fix them already in 0.74 (or they will be fixed soon).

[mrousavy]

It's only me doing the migration - and no, only V4 will be migrated. If you are on V3, I'd recommend to switch over to V4 because there are tons of improvements

[joacub]

vision camera currently can be working with react native 0.74.1 um using with that and no issues, the camera is even working super fast, ios buils with no issues and andoird you just need to include the error that android studio shows up and fix including the symbols that android studio suggest to you.

[coder-xiaomo]

Thanks for the author's open source project, which has helped me a lot. I would like to know, is it a big workload to migrate v4 to new architecture, and does it take a lot of time and energy?

I am preparing to migrate to react native 0.74.1, but this project does not support the new architecture at present, I would like to know how long will it be appropriate to migrate, such as weeks, months, or even longer?

Thanks again for the author's open source project, which has helped me a lot!

[chawkip]

A small patch for those having a build issue. Not sure if the frame processor will continue working after this patch, waiting for your inputs. But the camera will work, and the build issues will be resolved.

diff --git a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/frameprocessors/VisionCameraProxy.kt b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/frameprocessors/VisionCameraProxy.kt
index d697befe..8de418b0 100644
--- a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/frameprocessors/VisionCameraProxy.kt
+++ b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/frameprocessors/VisionCameraProxy.kt
@@ -7,12 +7,14 @@ import com.facebook.jni.HybridData
 import com.facebook.proguard.annotations.DoNotStrip
 import com.facebook.react.bridge.ReactApplicationContext
 import com.facebook.react.bridge.UiThreadUtil
+import com.facebook.react.common.annotations.FrameworkAPI
 import com.facebook.react.turbomodule.core.CallInvokerHolderImpl
 import com.facebook.react.uimanager.UIManagerHelper
 import com.mrousavy.camera.core.ViewNotFoundError
 import com.mrousavy.camera.react.CameraView
 import java.lang.ref.WeakReference
 
+@OptIn(FrameworkAPI::class)
 @Suppress("KotlinJniMissingFunction") // we use fbjni.
 class VisionCameraProxy(private val reactContext: ReactApplicationContext) {
   companion object {

Works like a charm! Thanks 🙏🏻

[mrousavy]

I don't know. It's pretty losely coupled, but the only part that cannot be losely coupled is the install() function, which uses JSI code that doesn't work in bridgeless.
So either way it's probably the best to just migrate everything to new arch.

[fabOnReact]

For the Frame Processor runtime I need access to the jsi::Value/jsi::Function that the user passes to the view directly, instead of converting it to a callback within the TurboModules/Native Modules system because the Frame Processor function is a Worklet.

https://react-native-vision-camera.com/docs/guides/frame-processors

If I try to solve the issue without using worklet (we can try to add it later).

function App() {
  const userDefinedFunction = (frame) => {
    const objects = detectObjects(frame)
    console.log(`Detected ${objects.length} objects.`)
  }
  const frameProcessor = useFrameProcessor(userDefinedFunction)

  return (
    <Camera
      {...cameraProps}
      frameProcessor={frameProcessor}
    />
  )
}

1. Register userDefinedFunction using createFromHostFunction (codegen command already does this for you)
2. Java/iOS invoke the callback with the frame as parameter https://github.com/facebook/react-native/blob/57ed0fb30931979742634a1faa9a4d3b5261e50d/packages/react-native/ReactAndroid/src/main/java/com/facebook/react/fabric/FabricUIManager.java#L1048-L1064
3. To implement it as worklet, I would look into other libs like reanimated

https://docs.swmansion.com/react-native-reanimated/docs/2.x/fundamentals/worklets/

https://github.com/software-mansion/react-native-reanimated/blob/6806d3d89983ffacd3b1bebbf4fecf86ba56addb/Common/cpp/ReanimatedRuntime/WorkletRuntimeCollector.h#L12-L15

  // When worklet runtime is created, we inject an instance of this class as a
  // `jsi::HostObject` into the global object. When worklet runtime is
  // terminated, the object is garbage-collected, which runs the C++ destructor.
  // In the destructor, we unregister the worklet runtime from the registry.

Sorry. I'm trying to learn more about this so maybe my comment not clear.

[mrousavy]

hiiiii thanks so much for this post; reading this and the code i'm not totally understanding what the core issue is - is it that the UIManager APIs themselves are not supported / behaving as expected in the new architecture?

Hey @philIip thanks for tuning in! 👋 so the idea is that VisionCamera's View uses the RN architecture for almost everything:

• device
• format
• isActive
• resizeMode
• ...etc

But there is one prop which the user passes to the Camera that I cannot use the RN architecture for, and that is the frameProcessor. The reason is that I do some special transformations to that function, and need to get it as the pure jsi::Function on the native side to convert it to a callable Worklet on my native side.
If I don't manually get it as a jsi::Function and instead use the bridge for this, it will convert it to a RCTCallback or something, which is not what I want.

This then runs on the Camera Thread and is synchronously invoked with a jsi::HostObject as a parameter.

But this is only one specific prop for the Camera (frameProcessor) the rest should use the RN architecture as normal.

The RN architecture simply doesn't have a mechanism for Worklets, or custom jsi::HostObjects currently, so I need to write that manually in JSI/C++.

another question, i'm assuming that the problem you're implying with the worklet is that it needs to be run on a specific thread, but the callback conversion forces it to always be run on JS thread? the piece that i'm missing here is where the native module block conversion is happening? it seems like everything in your code pointers is pure JSI at the moment and doesn't involve the native module infra.

Exactly, that and that I cannot pass a custom jsi::HostObject ("Frame") to the callback.

This is where I eventually convert the jsi::Function to a normal Objective-C/Swift method:

• react-native-vision-camera/package/ios/Frame Processor/VisionCameraProxy.mm

  Line 77 in bca9472

  view.frameProcessor = [[FrameProcessor alloc] initWithWorklet:worklet context:_workletContext];

• react-native-vision-camera/package/ios/Frame Processor/FrameProcessor.mm

  Lines 33 to 43 in bca9472

  	- (void)callWithFrameHostObject:(std::shared_ptr<FrameHostObject>)frameHostObject {
  	// Call the Frame Processor on the Worklet Runtime
  	jsi::Runtime& runtime = _workletContext->getWorkletRuntime();
  	// Wrap HostObject as JSI Value
  	auto argument = jsi::Object::createFromHostObject(runtime, frameHostObject);
  	jsi::Value jsValue(std::move(argument));
  	// Call the Worklet with the Frame JS Host Object as an argument
  	_workletInvoker->call(runtime, jsi::Value::undefined(), &jsValue, 1);
  	}

[mrousavy]

No worries! Yes this is something that does not work in old or new arch, and I don't think it even should be part of core.
It's very custom stuff.

The reason I thought it was a blocker for me is because the workaround in old arch was using the RCTCxxBridge/CatalystInstance to get the Runtime, and this now no longer works in new arch - so I want to use a Cxx TurboModule to get access to the runtime directly, but I have my doubts that codegen will work in this "hybrid" mode (generate everything for C++, but also bridge 90% of the props over to Java/Swift, while the others remain in C++ JSI)

[necmettindev]

hi guys, which RN version should i to use to use this lib? while is working on the migration :)

0.73.6

[mrousavy]

if you have any design docs or info you can share on how the global memory address is used and set but is a reference to a C++ variable

Thanks - not sure if I understand your question, but this is a thing I'd probably more rather tackle alone as it is quite a big refactor on the native side. Normally I'm happy about every contribution, but in this case I'd more rather build this myself as many things just require deep knowledge about the codebase of this.

[joacub]

Im using a expo so dont know in bare, but if you have a android folder just open that in android studio and build, thats it, I guess in bare rn will be the same

[tremblerz]

Thanks @idrisssakhi that worked for me!
Also had to add implementation 'com.google.mlkit:barcode-scanning:17.2.0' in build.gradle inside the src folder for my barcode use-case.

[mrousavy]

Thanks so much for your detailed answer @cipolleschi! This is really helpful :)

[rayronvictor]

We did a lot of work on interop layers for 0.74. The issue author is not mentioning which version of React Native they are using. It is likely that we fix them already in 0.74 (or they will be fixed soon).

I'm using RN 0.73.2

[fabOnReact]

2. For the Frame Processor runtime I need access to the jsi::Value/jsi::Function that the user passes to the view directly, instead of converting it to a callback within the TurboModules/Native Modules system because the Frame Processor function is a Worklet. This is how this works currently:
   JS: Camera.tsx (uses findNodeHandle(..) + a global setFrameProcessor(..) func)
   iOS: VisionCameraProxy.mm (uses UIManager::viewForReactTag to find the view)
   Android: VisionCameraProxy.kt (uses UIManagerHelper.getUIManager to find the View)

I believe there is an example of implementation in TextInput.js which uses the codegen command setTextAndSelection instead of using UIManager + findNodeHandle.

Codegen Commands replaces setNativeProp, which was used with functionalities like react-native-camera zoom prop. Re-rendering the entire component every time there is a change in zoom, would cause the react-native app to slow down, for this reason we would use setNativeProp({zoom: newZoomValue}).

It is sometimes necessary to make changes directly to a component without using state/props to trigger a re-render of the entire subtree. When using React in the browser for example, you sometimes need to directly modify a DOM node, and the same is true for views in mobile apps. setNativeProps is the React Native equivalent to setting properties directly on a DOM node.

More info here facebook/react-native#42701 (comment)

The codegen command calls FabricRenderer dispatchCommand which uses createFromHostFunction to register a function on the native Android/iOS API. JavaScript can call the function in the native iOS/Android API.

1. Function::createFromHostFunction() registers a new function with methodNam is dispatchCommand

/// A function which has this type can be registered as a function
/// callable from JavaScript using Function::createFromHostFunction().
/// When the function is called, args will point to the arguments, and
/// count will indicate how many arguments are passed.  The function
/// can return a Value to the caller, or throw an exception.  If a C++
/// exception is thrown, a JS Error will be created and thrown into
/// JS; if the C++ exception extends std::exception, the Error's
/// message will be whatever what() returns. Note that it is undefined whether
/// HostFunctions may or may not be called in strict mode; that is `thisVal`
/// can be any value - it will not necessarily be coerced to an object or
/// or set to the global object.

2. The HostFunction for the methodName dispatchCommand is passed as the last argument of createFromHostFunction

https://github.com/facebook/react-native/blob/8317325fb2bf6563a9314431c42c90ff68fb35fb/packages/react-native/ReactCommon/jsi/jsi/jsi.h#L100-L111

  /// Create a function which, when invoked, calls C++ code. If the
  /// function throws an exception, a JS Error will be created and
  /// thrown.
  /// \param name the name property for the function.
  /// \param paramCount the length property for the function, which
  /// may not be the number of arguments the function is passed.

https://github.com/facebook/react-native/blob/8ff05b5a18db85ab699323d1745a5f17cdc8bf6c/packages/react-native/ReactCommon/react/renderer/uimanager/UIManagerBinding.cpp#L603-L618

[uiManager, methodName, paramCount](
    jsi::Runtime& runtime,
    const jsi::Value& /*thisValue*/,
    const jsi::Value* arguments,
    size_t count) -> jsi::Value {
  validateArgumentCount(runtime, methodName, paramCount, count);


  auto shadowNode = shadowNodeFromValue(runtime, arguments[0]);
  if (shadowNode) {
    uiManager->dispatchCommand(
        shadowNode,
        stringFromValue(runtime, arguments[1]),
        commandArgsFromValue(runtime, arguments[2]));
  }
  return jsi::Value::undefined();
});

3. The dispatchCommand HostFunction calls FabricMountingManager::dispatchCommand:

[philIip]

❌ For the Frame Processor runtime I need access to the jsi::Value/jsi::Function that the user passes to the view directly, instead of converting it to a callback within the TurboModules/Native Modules system because the Frame Processor function is a Worklet. This is how this works currently:

hiiiii thanks so much for this post; reading this and the code i'm not totally understanding what the core issue is - is it that the UIManager APIs themselves are not supported / behaving as expected in the new architecture?

another question, i'm assuming that the problem you're implying with the worklet is that it needs to be run on a specific thread, but the callback conversion forces it to always be run on JS thread? the piece that i'm missing here is where the native module block conversion is happening? it seems like everything in your code pointers is pure JSI at the moment and doesn't involve the native module infra.

❌ For the Frame Processor runtime I need access to the jsi::Runtime as early as possible (on demand). Currently I get the jsi::Runtime from the RCTCxxBridge/CatalystInstance, which is a kinda private and unsafe API.

right now we made this backward compatible, but i will be writing some docs on what the future-looking APIs are for this!

[philIip]

ok i see, so bear with me just want to make sure i'm getting it right! this worklet scenario is not properly supported in the old or new architecture, but you've figured out a way to make it work doing all this custom JSI stuff.

is it a blocker because this is something you feel strongly should be baked into the framework, or is there actually a gap in the old / new architecture here? i think that's the last part i'm missing

[joacub]

hi @mrousavy thanks for this great project, im looking for this as we are using the scanner in a ver small application for cuba and the phones in there are very old ones, and i think this will be a really good for the performance of the overall scanner, do you ahve any time on when this will be migrated to the new arch ?, the react-native-workles is a blocker for us to use the new arch

[mrousavy]

Not yet, I asked the Meta guys a question here: reactwg/react-native-new-architecture#167 (reply in thread) and I am waiting for a reply - that'll decide how easy it's gonna be to migrate over to new arch.

[flexingCode]

hi guys, which RN version should i to use to use this lib? while is working on the migration :)

[chawkip]

hi guys, which RN version should i to use to use this lib? while is working on the migration :)

0.73.7

[404-html]

Do you guys plan migrate both, V3 and V4, to the new-arch?

[MinskLeo]

Thanks for author's effort. Would be really good to have react-native-vision-camera on 0.74+.

[angelo-hub]

@mrousavy this sounds like kinda an exciting low level problem to solve, I don't have full context on the C++ code, but i'd love to take a crack at it, if you have any design docs or info you can share on how the global memory address is used and set but is a reference to a C++ variable

[mrousavy]

Would be really good to have react-native-vision-camera on 0.74+

Yea, you can fund the development of this by sponsoring to the Polar pool linked here in the issue. Then we all get it faster.

I would like to know, is it a big workload to migrate v4 to new architecture, and does it take a lot of time and energy?

Yes, because the Frame Processor runtime works a lot with the jsi::Runtime, and all of that needs to be migrated to a CxxTurboModule as the Bridge no longer exists (which is currently used for a lot of APIs). react-native-worklets-core also needs to be migrated, as that's a dependency of VisionCamera.

So I'm guessing if I really dedicate a full week to this, I could hack this out. Once the sponsor pool (Polar, linked here) is full, I can take a stab at this.

[Alper-Demir-Eng]

Just a question, for apps that use VC with the Frame Processor disabled, is it possible to somehow compile successfully under RN 0.74?

[Alper-Demir-Eng]

@joacub Could you please create a tutorial, I've never built on Android Studio and not sure how to go about it.

[joacub]

@joacub Could you please create a tutorial, I've never built on Android Studio and not sure how to go about it.

Are you using a bare react native or expo managed ?

[Alper-Demir-Eng]

@joacub Bare RN.

[nikitapilgrim]

Im using a expo so dont know in bare, but if you have a android folder just open that in android studio and build, thats it, I guess in bare rn will be the same

can you make patch pls?

[joacub]

Im using a expo so dont know in bare, but if you have a android folder just open that in android studio and build, thats it, I guess in bare rn will be the same

can you make patch pls?

this is the patch:

#2614 (comment)

but frameprocessor and codeScanner together does not work

[cjadhav]

Hi Guys,

What's the tentative plan to release this fix.?
4.0.5 is still missing this fix.

Thanks

[mrousavy]

Hi Guys,

What's the tentative plan to release this fix.? 4.0.5 is still missing this fix.

Thanks

This is not a fix, it's quite a big migration. As for status; I haven't worked on this yet.

Thanks for all sponsors so far!