current code is slow, will you release a faster python code? about ar-depth

[mpottinger]

@zjd1988 Just fyi I was able to get this code working very fast with minor modifications using numba. It still took me an all-nighter to do it, and I don't currently update or use it. In fact I lost the code. It was about as fast as you can expect in Python. Still not quite 30fps real time though. More like 5 fps.

I opted to use real depth sensors instead though because that is much less computationally expensive and more accurate.

[zjd1988]

@mpottinger thank you very much , I will try numba to acclerate performance

[mpottinger]

@zjd1988 No problem.

If numba supported acceleration for all Python code then it would be super easy, but unfortunately Numba is limited to accelerating pure python code without custom classes.

For example it can't accelerate a function that uses calls to an outside library like OpenCV.

The only library/module calls it supports are numpy functions.

So you need to isolate the parts that can be accelerated into numba decorated functions by putting @njit, parallel, etc on top of the def.

It is not that hard, just tedious and time consuming.

[mpottinger]

@zjd1988 Just so you know someone did a C++ implementation of this algorithm.

I was excited to see it, it is close to real time but unfortunately, still not a high enough framerate for AR, unless it were possible to do some kind of latency correction.

https://github.com/muskie82/AR-Depth-cpp

[zjd1988]

@mpottinger thank you for your tips, I have try this, i had tried to port AR-depth to iphone, but it was not good enough for our scene. Now i was assigned another task, thanks again.

[holynski]

Hey -- sweet! I'm happy to see someone managed to implement this.

There are a bunch of ways to speed up the implemented system (which are also valid for the linked C++ implementation). Some of these were used for the original paper implementation, and some were chalked up to "future work", but you should be able to get a more-than-realtime implementation on a phone with some combination of these. From a previous post:

Things we did:

• Solver: For the results presented in the paper, we used hierarchical preconditioning to reduce the number of overall solver iterations required (see the end of section 4.4 of the paper, or the original paper on the solver here). Adding this component is likely what will make the biggest difference.

^ This will make a massive difference in the number of iterations you'll have to run the solver for. The system will basically converge in under 10 iterations, as opposed to the 400 being used in the C++ implementation.

• Flow: To get even faster results, you can change the settings of the flow estimation to the faster preset. More specifically, change the "2" in the initialization line to "1" or "0".

Other speedups from the future work section:

• Implementing a real-time solver similar to the one described here.
• Reducing the resolution at which the flow/reliability/gradient is computed. These are blurred after being computed anyways, so it likely doesn't matter if they're computed at a lower resolution and upsampled.
• Reducing the number of solver iterations at non-keyframes (places without new depth points), since we're not getting strong information about changes in 3D structure, and are mostly just moving object boundaries slightly.

[mpottinger]

@holynski Wow thanks for the comphensive overview of how this algorithm could be made to run faster.

Yeah I am amazed at how he translated it to C++. Code still looks very clear and readable.

My attempt with Numba was just as fast or a bit faster, but because of the limitations of Numba the code was spaghettified and I gave up.

I wish I were skilled enough to implement the optimizations you recommended, but its a bit out of my league I think! Especially the cg solver.

How about if you already have more complete depth information?

I do not know much about cg solvers, or 'preconditioning' a solver. However if I have more dense depth information from say a real depth sensor, would that help?

For example I have a Huawei P30 Pro and I am working on an AR application that takes advantage of the TOF sensor. Occlusion works, but the data from the sensor is sometimes noisy or incomplete.

Could this algorithm be faster at refining dense but still imperfect depth, or does it not work that way? If so I may try it.

Edit:

Also how about GPU acceleration? I do not think writing my own cg solver is within my capabilities, but there are GPU accelerated alternatives to Eigen for linear algebra such as ViennaCL

[holynski]

How about if you already have more complete depth information?

I do not know much about cg solvers, or 'preconditioning' a solver. However if I have more dense depth information from say a real depth sensor, would that help?

Absolutely -- having more (accurate) depth information would certainly allow the solver to converge more quickly.

If you have a mostly complete depth map, but just want to make the edges a bit sharper, you could definitely use a technique like this to smooth things out and make the edges sharp. In practice, you could also limit the number of iterations of the solver, or only operate on pixels within some distance of a detected occlusion boundary.

If you depth maps are already close to perfect, however, you may consider using just an image filter, like a bilateral filter/solver. These will usually snap your depth maps to image edges without the need for fancier inference. The technique I'm using here is really most useful when you have sparse points. It'll work for all depth maps, but if they're already very good, it might just be less useful. It all depends on what your inputs are.

Also how about GPU acceleration? I do not think writing my own cg solver is within my capabilities, but there are GPU accelerated alternatives to Eigen for linear algebra such as ViennaCL

GPU acceleration is certainly an option. There are already some implemented versions (that run in real-time) in the links in my previous post, and only small parts of the formulation would have to be changed in order to solve this problem.

[mpottinger]

@holynski Ok thanks for the feedback. That will encourage me to give it a try when I have time.

I am using a tof depth sensor with ARCore which works great but has limited resolution and range.

Mostly I am using it to build point cloud maps, more interested in mapping/tracking than AR, but it would be fun to try to feed a dense pointcloud map built using the sensor and see what the results are.

So far my first attempt failed, i had a blank depth map.

[holynski]

I'm working on a refactor of the code now, and will be uploading fixes of some of the other open issues (#4, #5), so stay tuned.

[mpottinger]

@holynski Well, looks like the game has changed. Google will be releasing motion stereo depth in ARCore, sometime this year.

It can already be previewed in the houzz app. From what I have seen the occlusion is imperfect, edges are not as sharp as they could be, range is limited, etc. So your algorithm could still fill in that gap but havr access to dense depth on most ARCore phones instead of sparse depth points.

[holynski]

Neat! I'm looking forward to trying it as well!

[holynski]

Closing this issue for now, since there's no ongoing issue with the code not addressed elsewhere. Feel free to open another one or message me directly if anything comes up.

[Tord-Zhang]

@holynski Hi, thanks for your wonderful work! Have you updated the code? It is really slow......