facebookresearch / ar-depth

Fast Depth Densification for Occlusion-Aware Augmented Reality

Please see the Python notebook for the full README.

License

AR-Depth is MIT licensed, as found in the LICENSE file.

ar-depth's Issues

Missing depth reprojection

The solver initialization, the sparse points, and the TemporalMedian function all need to reproject the depth points from another viewpoint. This component seems to be missing.

How to get the 3d points and image points

Hello,

I tried to use your data with dso, but I can not repeat your result. As for me, I save 2d points and depth data of keyframe and project back to get 3d points, which is more than 1 million points. And I check the points you provide, it is only around 60 thousand points. How did you save the 3d points?

Wrong results when value is negative using maximum in GetImageGradient function

If rgb gradients all are positive, maximum get max gradient. But If rgb gradients all are negative, maximum get min gradient.
It seems that the code is not correct.

Get Sparse Depth map

Hello, there is something about the GetSparseDepthMap function that I don't understand. The following line appears unclear to me: depth_map[int(coord[1]), int(coord[0])] = depth with depth_map = np.zeros((camera.height, camera.width), dtype=np.float32)

This means that every 2D points that have an index superior to the camera image size (here 1080 and 1920) cause an error? Or they are actually not considered by the algorithm?

By the way, in my data, I have like 31000 points, so every points that have index below 31000 cause an error?

Please explain me how the algorithm read the images.txt file, and espacially the line with the 2D points.

run the sample data, has a keyerror

I download the whole rep and the sample data. Following the instructions, I run all the cells. However I met a keyerror. PS: I found that the first line in "images.txt" is 000009.png, not 000000.png.
As below:

Number of points: 62786
Number of frames: 691
Processing frame 000009.png

KeyError Traceback (most recent call last)
in ()
5 for frame in recon.ViewIds():
6 print "Processing frame " + recon.views[frame].name
----> 7 kfs = recon.GetReferenceFrames(frame)
8 if (len(kfs) == 0):
9 continue

in GetReferenceFrames(self, view_id)
24
25 def GetReferenceFrames(self, view_id):
---> 26 kf = self.GetNeighboringKeyframes(view_id)
27 dist = np.linalg.norm(self.views[kf[1]].Position() - self.views[kf[0]].Position()) / 2
28 pos = self.views[view_id].Position()

in GetNeighboringKeyframes(self, view_id)
12 next_keyframe = -1
13 for idx in range(view_id - 1, 0, -1):
---> 14 if self.views[idx].IsKeyframe():
15 previous_keyframe = idx
16 break

KeyError: 7

Error (Keyframe, refframe, wrong translation info ...)

1. When I run the provided code, I met the error "key frame 7".
   I guess this is caused by the omitted? or deleted data of images. (images.txt file starts from 000009.png file instead of 000001.png)

2. So, I slightly fixed my code and run again the code.
   -> Solved the keyframe error.
   However, the code couldn't compute the reference frame in this time. There is no difference between kf[0].Position() and kf[1].Position(). So I checked the translation info in the dictionary of views and found that they were all the same. [ 0.17589158 -0.09372958 0.2583495 ]

It seems that the code is not working with the current version.

Depth outputs all blank

I am running the code with the sample data that ships with the repo. Speed issues aside, all of the depth outputs I have seen so far (~25 frames) are all the same empty image. Each pixel in the output PNG is [68, 1, 84, 255]. Furthermore, I have to see the solver converge. Each frame runs 500 iterations and quits.

Am I doing something wrong? The intermediate images all look fine. The sparse depth images are a bit weak though. All I've done is run the code as is.

Here are the input, output and intermediate results for frame 39:

Input

Output

Image gradient

Flow

Flow gradient

Reliability

Soft edges

Solver initialization

Solver smoothness

Sparse points

What if the camera intrinsics is unknown

Hi, thanks for your wonderful work!

It requires the camera intrinsics to get the sparse points using SLAM systems. However, in many cases it is very difficult to satisfy this premise. Is there any possible solution if the came intrinsics is unknown?

Thanks.

current code is slow, will you release a faster python code?