AugmentedFacesExample getting EXC_BAD_ACCESS when run for a moment about arcore-ios-sdk

[Bersaelor]

Thank you for believing me!
(since the whole reproduction of the problem always takes a few minutes to crashand other people I worked with would have just said "Oh, it's working for me")

Using the delegate-call works flawlessly, here's the method in case someone else wants to copy-paste:

// MARK: - Face Session delegate

extension FacesViewController : GARAugmentedFaceSessionDelegate {
    
    public func didUpdate(_ frame: GARAugmentedFaceFrame) {
        if let face = frame.face {
            faceTextureNode.geometry = faceMeshConverter.geometryFromFace(face)
            faceTextureNode.geometry?.firstMaterial = faceTextureMaterial
            faceOccluderNode.geometry = faceTextureNode.geometry?.copy() as? SCNGeometry
            faceOccluderNode.geometry?.firstMaterial = faceOccluderMaterial
            
            faceNode.simdWorldTransform = face.centerTransform
            updateTransform(face.transform(for: .nose), for: noseTipNode)
            updateTransform(face.transform(for: .foreheadLeft), for: foreheadLeftNode)
            updateTransform(face.transform(for: .foreheadRight), for: foreheadRightNode)
        }
        
        // Set the scene camera's transform to the projection matrix for this frame.
        sceneCamera.projectionTransform = SCNMatrix4.init(
            frame.projectionMatrix(
                forViewportSize: sceneView.bounds.size,
                presentationOrientation: .portrait,
                mirrored: false,
                zNear: 0.05,
                zFar: 100)
        )
        
        // Update the camera image layer's transform to the display transform for this frame.
        cameraImageLayer.contents = frame.capturedImage as CVPixelBuffer
        cameraImageLayer.setAffineTransform(
            frame.displayTransform(
                forViewportSize: cameraImageLayer.bounds.size,
                presentationOrientation: .portrait,
                mirrored: true
            )
        )
        
        // Only show AR content when a face is detected.
        sceneView.scene?.rootNode.isHidden = frame.face == nil
    }
    
}

[rsfuller]

Ahh, okay. You appear to have found a deeper issue in the SDK. I will keep this open for tracking.

In the meantime, if you'd like something more stable, I would suggest switching to the push model (delegate callback). You can look at previous releases to see how that's done in the sample app.

Thanks for finding this and reporting it so thoroughly!

[andy-at]

Would be great to have some priority or acknowledgement of this issue. Our app also crashes in the same conditions detailed by @kapsyst.

[JamieHeatherZozoNZ]

Any updates on this? I'm getting the same problem :(

[Bersaelor]

After some more testing it seems the problem is inside the ARCore framework, potentially introduced in Version 1.16.0.

I rearranged the example code to the following:

    let projectionSIMD = frame.projectionMatrix(
        forViewportSize: sceneBounds,
        presentationOrientation: .portrait,
        mirrored: false,
        zNear: 0.05,
        zFar: 100
    )
    let projectionM = SCNMatrix4.init(projectionSIMD)
    // Set the scene camera's transform to the projection matrix for this frame.
    DispatchQueue.main.sync {
      sceneCamera.projectionTransform = projectionM
    }

The crash always happens during the frame.projectionMatrix() call.

As far as I understand, EXC_BAD_ACCESS in objc_msgSend means that the frame object was already released when the frame.projectionMatrix call is sent. Thats why I tried saving the result of the call and only using the result in the Dispatch.sync call. Unfortunately it's still crashing, so its maybe an internal ARCore issue.

[rsfuller]

Thanks for this bug report!

The fix will likely be switching the sceneView.bounds.size call (UI API) to cameraImageLayer.bounds.size (which is safe to call on a bg thread), and not dispatching the block to the main queue.

[Bersaelor]

@rsfuller thanks for answering, I thought about that, thats why in the above example I saved the bounds under:

    private var sceneBounds: CGSize = .zero

in

    public override func viewDidLayoutSubviews() {
        super.viewDidLayoutSubviews()
        
        sceneBounds = sceneView.bounds.size
    }

and then read the instance var when calling projectionMatrix. Unfortunately that didn't make a difference it still crashed (as you can see in the screenshot of my previous comment).

The next thing I did was moving the code before the face-block, like so:

  public func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
    guard let frame = faceSession?.currentFrame else { return }

    let projectionSIMD = frame.projectionMatrix(
        forViewportSize: sceneBounds,
        presentationOrientation: .portrait,
        mirrored: false,
        zNear: 0.05,
        zFar: 100
    )

    if let face = frame.face {

and at least for the last 13mins it's running crash free. But that isn't really a fix, since it shouldn't matter in which line of your method we call a method, I guess?

[rsfuller]

Yes, I would expect your alteration to not crash either, because you're not dispatching to the main queue. [It is not clear to me why the frame is sporadically not retained in the closure. I would have to dig more into that.]

The Augmented Faces example app previously had the scene camera's transform updated on a callback from the ARCore API, but we switched to a pull model (calling .currentFrame) in the example app for simplicity in 1.16, and now everything is updated in sync.

However, on reading this bug I realized there is no good reason for using the sceneView bounds as it is tied to the view bounds (and hence the cameraImageLayer bounds (see line #87). So we may prefer to simply switch to that and not dispatch to the main queue.

[Bersaelor]

Agreed (not not using the sceneView bounds).

I just let it run for another 30mins, with my changes (had a magazine with a person on the cover sitting in front of the ipad while I had dinner). And it crashed again with a released frame.

It seems that the currentFrame is released before the renderer( method is finished, if I move the projectionMatrix earlier in the method, I still get the crash, just this time on frame.capturedImage:

[yinglieng]

Bersaelor, can you still reproduce the crash with the latest version v1.19.0?

I just tested the version v1.19.0

Test device: iPhone 6s Plus iOS 13.3.1

Run XCode to install the app on the device. Let the app run with the debugger attached. The face is in the frame.
The app ran for 12 min, without crash. Then I stopped the experiment.

It would be appreciated with your new test. Thanks!

[Bersaelor]

Unfortunately it looks like I got the same result:

(I let the demo running while it looks at a picture of a person, so it gets a face at every frame)

(iPhone X, iOS 13.7 Xcode 11.7)

[kapsyst]

Is there any update on this? I'm experiencing a very similar issue.
While I'm working on a more complex application, the general usage pattern is identical to the example app. I get an EXC_BAD_ACCESS error on the render thread, especially for debug builds, and/or after running for some time.

The crash seems to occur when performing more work on the render thread (hence debug builds). As @Bersaelor mentioned, this could indicate AR Frame memory is being released too soon.

Any help appreciated.

Dev environment:
iPhone X (iOS 14.2)
Xcode Version 12.4 (12D4e)
AR Augmented Faces 1.23.0 (occurs in earlier versions too)

[kapsyst]

Also just to confirm, the source code for AR Augmented Faces is not available, correct?

All I can find is:
https://dl.google.com/dl/cpdc/f3dd4e97ce621828/ARCore-1.23.0.tar.gz

Which contains precompiled frameworks.

[kapsyst]

Hi @rsfuller, @Bersaelor I just wanted to give you a quick update.

Since my last post I've tried the following:

• Remove GAR completely from our application, replacing with pre saved meshes and video. The crash no longer occurred.
• As suggested, switched to updates via GARAugmentedFaceSessionDelegate didUpdate function. The crash still occurred (however in a different location on the render thread stack).
• Soak tested the Google example app (https://github.com/google-ar/arcore-ios-sdk/tree/master/Examples/AugmentedFacesExample), without modifications. It crashed in a very similar fashion to our own app, and to the way @Bersaelor has reported.

Here is a screen shot, and the crash log:

AugmentedFacesExample 2021-04-20 14-32.txt

Dev environment:
iPhone 8 (iOS 14.2)
Xcode Version 12.4 (12D4e)
AR Augmented Faces 1.23.0

It's worth noting that it occurs more frequently in debug builds, and on lower spec devices (iPhone 8 in particular). The location of the EXC_BAD_ACCESS changes when the stack composition is changed too.

At this stage it's fairly safe to say that GAR is the cause of the crash, as far as I can see. Since releasing our application, we've had about 12000 occurrences, so any help or updates would be very much appreciated.

[jrullman]

A fix for this will be in the 1.25 release of ARCore, anticipated in mid-June.

[jrullman]

Until then, a workaround would be to use the delegate to get frames, instead of using currentFrame.

[kapsyst]

Thanks for the update @jrullman -- very much looking forward to trying out the fix.

Until then, a workaround would be to use the delegate to get frames, instead of using currentFrame.

FYI we did try using the didUpdate function in GARAugmentedFaceSessionDelegate.

We would convert GAR meshes to SceneKit meshes (copying), and save them with video frames and camera matrices in a sync'd ring buffer. They would then get read from the render thread as they became available. In other words, we weren't directly accessing GARFrames from the render thread. However, this still crashed in the same manner as before.

We also tried saving the results of this ring buffer on disk, and "playing" them back without any calls GAR, which worked without crashing.

[jrullman]

My suggestion is simpler: have your own currentFrame property, which is protected by a lock. Set this property in the delegate method and get it where the code is currently using session.currentFrame. Locking is necessary to prevent the issue.

[kapsyst]

Thanks @jrullman, although I'm not quite sure I understand what you mean - could you provide an example?

[kapsyst]

Hi @jrullman, I just saw there was an update for 1.25 -- have yet to test with our application but looking forward to trying it out.

FWIW, here is the relevant code for a fix we attempted using a semaphore for locking. I'm not sure if this is what you had in mind. If not, any guidance would be appreciated.

We tested this version and were still able to recreate the crash in question.

class ViewController: UIViewController {

    fileprivate var updateGARSemaphore = DispatchSemaphore(value: 0)
    fileprivate var lastGARFrame: GARAugmentedFaceFrame?
}

extension ViewController: SCNSceneRendererDelegate {

    public func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {

        guard let frame = self.lastGARFrame else {
            self.updateGARSemaphore.signal()
            return
        }

        // Use the frame here.

        self.updateGARSemaphore.signal()
    }
}

extension ViewController: GARAugmentedFaceSessionDelegate {

    public func didUpdate(_ frame: GARAugmentedFaceFrame) {

        // Don't wait more than a frame, in case the render update takes too long.
        let result = self.updateGARSemaphore.wait(timeout: (DispatchTime.now() + Double(1.0/60.0)))
        if (result == .success) {
            self.lastGARFrame = frame
        }
    }
}

[cpsauer]

FWIW, I'm not seeing this crash running the sample app for 1.25 (1.26 broken per #47).

How about you all? Are you still seeing the issue, or no?