orientationOutput isn't applied in IOS with skia about react-native-vision-camera

[tomerh2001]

When I replace useSkiaFrameProcessor with useFrameProcessor it shows up normally.

This is blocking a production build, any chance for some help? Even if it's just the file and I'll patch it until an official fix is released

[mrousavy]

1. Without reproduceable code I cannot reproduce the issue
2. Without RELEVANT log output I cannot determine what went wrong on the native side when applying orientations
3. Without knowing the Camera device's sensorOrientation I don't know what the natural orientation is (counter-rotate)

[tomerh2001]

1. Without reproduceable code I cannot reproduce the issue
2. Without RELEVANT log output I cannot determine what went wrong on the native side when applying orientations
3. Without knowing the Camera device's sensorOrientation I don't know what the natural orientation is (counter-rotate)

Have you tried using output orientation with skia on iOS? We don't do anything special, just the very basics and the orientation doesn't work

[tomerh2001]

1. Without reproduceable code I cannot reproduce the issue
2. Without RELEVANT log output I cannot determine what went wrong on the native side when applying orientations
3. Without knowing the Camera device's sensorOrientation I don't know what the natural orientation is (counter-rotate)

Have you tried using output orientation with skia on iOS? We don't do anything special, just the very basics and the orientation doesn't work

??

[mrousavy]

Please stop spamming.

[mrousavy]

Update the issue with logs then I can help you.

[tomerh2001]

Update the issue with logs then I can help you.

I can't, my coworker has a MacBook and he's out for surgery until next week, but I'm being pressured to fix this issue until tomorrow.

So I can't provide logs now, but as I said, it seemed like a generic problem:

• W. Normal frame processors it showed up normally

• W. Skia frame processors it always happens

Has anyone tested IOS + outputRotation + skia?

We only needed the skia rotation fix, but it was released together with the ouputRotation feature which is a breaking change, so we're forced to use it - hence why I'm here

If you could release 4.1.1 with only the skia fix, or just give me a patch file or something for the skia rotation bug that'll be enough for me...

[tomerh2001]

Can you open this issue again? This issue is not resolved, and you are likely to be ignoring a major bug in your library.

For others thinking about donating here, note this behavior before donating. I also donated and that's what I get.

[mrousavy]

Hey @tomerh2001 I'm sorry but a "major" bug is a bit exaggerated. You can always use a fixed orientation of portrait to keep it locked, that should work fine.

Orientation took a ton of time and effort to build, and even though the sponsor pool wasn't nearly full I decided to implement it in my free time anyways. Instead of only complaining about something not working and it being a "major" bug you could also thank me for building orientation as far as it is right now. Everything's working apart from orientation in Skia.
I will fix this bug soon, but I don't have time now.

Also, don't get me wrong I appreciate your sponsorship, but this is a complex issue and just demanding that I fix it immediately without even providing logs or a clear reproduction is just unfair.

I need to keep the issues clean, and if you refuse to add logs or use a proper title to at least help me organize and debug this better, then it's staying closed.

Please update with logs.

[tomerh2001]

Hey @tomerh2001 I'm sorry but a "major" bug is a bit exaggerated. You can always use a fixed orientation of portrait to keep it locked, that should work fine.

Orientation took a ton of time and effort to build, and even though the sponsor pool wasn't nearly full I decided to implement it in my free time anyways. Instead of only complaining about something not working and it being a "major" bug you could also thank me for building orientation as far as it is right now. Everything's working apart from orientation in Skia.
I will fix this bug soon, but I don't have time now.

Also, don't get me wrong I appreciate your sponsorship, but this is a complex issue and just demanding that I fix it immediately without even providing logs or a clear reproduction is just unfair.

I need to keep the issues clean, and if you refuse to add logs or use a proper title to at least help me organize and debug this better, then it's staying closed.

Please update with logs.

There was a bug with the transformation matrix when using skia, a fix was released which is what I needed - but it was released together with the orientation which isn't fully working (this issue). If you would've released the bug fix first, then the orientation feature in another release, I could've just used the working bug fix version and could've spared this issue.

While I genuinely appreciate this library and its features, choosing to use it seems to just come back and continuously bite me when it comes to time sensitive builds.

[mrousavy]

While I genuinely appreciate this library and its features, choosing to use it seems to just come back and continuously bite me when it comes to time sensitive builds.

With all due respect - if you think this library is just biting you then don't use it.

I work on it in my free time, and if you need something fixed asap, contact me through my consulting agency to get full enterprise level support and almost instant fixes - billed with an hourly rate for short term fixes.

You can also always add features yourself and submit PRs/contributions here, this is how open-source works.
I am not responsible for your app's/client's deadline. If you/they need to release something sooner, maybe paying me to get it done is quicker than writing it yourself or switching to a different library (I don't think there are any other libraries that support Skia Frame Processors).

With that said; I will maybe focus on this soon in my free time, but I have a lot of other priorities right now.

[tomerh2001]

While I genuinely appreciate this library and its features, choosing to use it seems to just come back and continuously bite me when it comes to time sensitive builds.

With all due respect - if you think this library is just biting you then don't use it.

I work on it in my free time, and if you need something fixed asap, contact me through my consulting agency to get full enterprise level support and almost instant fixes - billed with an hourly rate for short term fixes.

You can also always add features yourself and submit PRs/contributions here, this is how open-source works. I am not responsible for your app's/client's deadline. If you/they need to release something sooner, maybe paying me to get it done is quicker than writing it yourself or switching to a different library (I don't think there are any other libraries that support Skia Frame Processors).

With that said; I will maybe focus on this soon in my free time, but I have a lot of other priorities right now.

Fair enough. I would appreciate if this can be looked at, or at least point me to the possible culprit file/function and i'll try to fix it

[mrousavy]

See withRotatedFrame:

react-native-vision-camera/package/src/skia/useSkiaFrameProcessor.ts

Lines 53 to 90 in 1723408

	function withRotatedFrame(frame: Frame, canvas: SkCanvas, func: () => void): void {
	'worklet'
	// 1. save current translation matrix
	canvas.save()
	try {
	// 2. properly rotate canvas so Frame is rendered up-right.
	switch (frame.orientation) {
	case 'portrait':
	// do nothing
	break
	case 'landscape-left':
	// rotate two flips on (0,0) origin and move X + Y into view again
	canvas.translate(frame.height, frame.width)
	canvas.translate(270, 0)
	break
	case 'portrait-upside-down':
	// rotate three flips on (0,0) origin and move Y into view again
	canvas.translate(frame.width, frame.height)
	canvas.rotate(180, 0, 0)
	break
	case 'landscape-right':
	// rotate one flip on (0,0) origin and move X into view again
	canvas.translate(frame.height, 0)
	canvas.rotate(90, 0, 0)
	break
	default:
	throw new Error(`Invalid frame.orientation: ${frame.orientation}!`)
	}
	// 3. call actual processing code
	func()
	} finally {
	// 4. restore matrix again to original base
	canvas.restore()
	}
	}

[tomerh2001]

This is my outputOrientation:

I noticed my frame.orientation is always portrait. Maybe this obvious info.

But I found that I get my expected behavior if frame.orientation = Camera.outputOrientation, which I wasn't able to force (couldn't find where), so I just copied the rotation from landscape-right (the value I chose for outputOrientation) to portrait (the constant value of my frame.orientation, I.e.:

and also did the same in getSurfaceSize:

This gives me the correct preview on IOS w. Skia.
This is definitely not a solution, but I guess it's at least a direction?
As a temp fix for now i might apply this only to IOS.

Also, can you please reopen this issue now so that others could help as well?

Edit: Managed to force the orientation value of frame.orientation, that's the patch-file, if anyone needs it:

diff --git a/node_modules/react-native-vision-camera/src/skia/useSkiaFrameProcessor.ts b/node_modules/react-native-vision-camera/src/skia/useSkiaFrameProcessor.ts
index 2983a07..e1a36a4 100644
--- a/node_modules/react-native-vision-camera/src/skia/useSkiaFrameProcessor.ts
+++ b/node_modules/react-native-vision-camera/src/skia/useSkiaFrameProcessor.ts
@@ -8,6 +8,7 @@ import { WorkletsProxy } from '../dependencies/WorkletsProxy'
 import { SkiaProxy } from '../dependencies/SkiaProxy'
 import { withFrameRefCounting } from '../frame-processors/withFrameRefCounting'
 import { VisionCameraProxy } from '../frame-processors/VisionCameraProxy'
+import { Platform } from 'react-native'
 
 /**
  * Represents a Camera Frame that can be directly drawn to using Skia.
@@ -223,6 +223,13 @@ export function createSkiaFrameProcessor(
     frameProcessor: withFrameRefCounting((frame) => {
       'worklet'
 
+        const previousOrientation = frame.orientation;
+        Object.defineProperty(frame, 'orientation', {
+          get() {
+            return Platform.OS === 'ios' ? 'landscape-right' : previousOrientation;
+          },
+        })
+
+
       // 1. Set up Skia Surface with size of Frame
       const surface = getSkiaSurface(frame)

[mrousavy]

frame.orientation is the Frame buffer's physical orientation relative to the output orientation.

Buffers need to be rotated with transforms/Matrixes, instead of physically rotating the pixels.

So I think orientation is properly computed. I need to double check this, but it is not the same as outputOrientation, by design. That's just how Cameras work. That's why the orientation feature isn't as easy as some people think it is.