Comments (15)
I think performance on Megadepth can increase a lot in terms of warp accuracy (matching correctly), but due to the "ground truth" being noisy, I think the geometry estimation accuracy may be starting to saturate. Probably a "perfect" method would reach something like 65 AUC@5 with the current ransac estimator.
I'll close the issue :)
from dkm.
At the same time, the points sampled by the model have a chance to produce strange slashes as depicted in the picture
from dkm.
Can tou provide the exact image pair? I have not noticed that bug before.
from dkm.
Can tou provide the exact image pair? I have not noticed that bug before.
Sure! Please check the following pairs from megadepth loftr test set:
/megadepth/Undistorted_SfM/0022/images/3216263284_1c2f358e5a_o.jpg
/megadepth/Undistorted_SfM/0022/images/624570534_077746f40c_o.jpg
/megadepth/Undistorted_SfM/0022/images/3715689873_8b49e3676d_o.jpg
/megadepth/Undistorted_SfM/0022/images/1390794615_4e2efcc84d_o.jpg
/megadepth/Undistorted_SfM/0022/images/2676532132_7c55a4b43a_o.jpg
/megadepth/Undistorted_SfM/0022/images/2899024525_d3c7b3b33c_o.jpg
/megadepth/Undistorted_SfM/0022/images/1469603948_0052cdbe5d_o.jpg
/megadepth/Undistorted_SfM/0022/images/3722427422_de83a590d7_o.jpg
/megadepth/Undistorted_SfM/0022/images/2142521640_9a08bee026_o.jpg
/megadepth/Undistorted_SfM/0022/images/2377284574_dcf50dc5c7_o.jpg
/megadepth/Undistorted_SfM/0022/images/453068332_a802fc9422_o.jpg
/megadepth/Undistorted_SfM/0022/images/325288797_537a42bb6f_o.jpg
/megadepth/Undistorted_SfM/0015/images/2087061093_e736a18497_o.jpg
/megadepth/Undistorted_SfM/0015/images/3305213780_e460e68085_o.jpg
/megadepth/Undistorted_SfM/0015/images/143983789_f030e8f660_o.jpg
/megadepth/Undistorted_SfM/0015/images/2960579238_7ba551628b_o.jpg
/megadepth/Undistorted_SfM/0015/images/3602433917_f463ce87ae_o.jpg
/megadepth/Undistorted_SfM/0015/images/3159096518_744c87899e_o.jpg
from dkm.
from dkm.
the green dots are pairs pruduced by dkm_model.sample() with symmertic=false
The blue point in the right picture is the correct correspondence of the green point in the left picture, and this correspondence is calculated by applying depth and internal parameters
from dkm.
Thanks, I'll check it out.
from dkm.
The dense warp seems to have no issue, might be something wrong with the sampling since I changed recently from np.random.choice -> torch.multinomial. Checking.
from dkm.
I can not reproduce. Could it be that what you are plotting are inliers after RANSAC? It might be the case that the estimation converges to a degenerate case with the camera being mirrored? (saying this because it seems like the translation error is low).
It seems like the points are approximately lying on a plane. If you estimated a fundamental matrix, there might be degenerate cases.
from dkm.
Could you share with me what estimator you used? Was it the opencv estimator used for mega-1500?
from dkm.
Could you share with me what estimator you used? Was it the opencv estimator used for mega-1500?
I first estimated the fundamental matrix, then the Essential matrix, and calculated R and t.
here is the function I used to calculate F and E:
"""
def calculate_F(dkm_model, pair_data):
image1_path = pair_data["image1_path"]
image2_path = pair_data["image2_path"]
W_A, H_A = Image.open(image1_path).size
W_B, H_B = Image.open(image2_path).size
warp_est = pair_data["warp_est"]
warp_certainty_est = pair_data["warp_certainty_est"]
# Sample matches for estimation
sample_matches, sample_certainty = dkm_model.sample(warp_est, warp_certainty_est, num=5000)
kpts1, kpts2 = dkm_model.to_pixel_coordinates(sample_matches, H_A, W_A, H_B, W_B)
# calculate fundamental matrix
F_est, mask = cv2.findFundamentalMat(
kpts1.cpu().numpy(),
kpts2.cpu().numpy(),
ransacReprojThreshold=0.2,
method=cv2.USAC_MAGSAC,
confidence=0.999999,
maxIters=10000
)
# print("percentage of inliers: ", 100 * mask.mean())
# save kpts1, kpts2, mask, certainty, F_est to pair_data
pair_data["F_est"] = F_est
pair_data["kpts1"] = kpts1.cpu().numpy()
pair_data["kpts2"] = kpts2.cpu().numpy()
pair_data["kpts_mask"] = mask
pair_data["sample_certainty"] = sample_certainty.cpu().numpy()
def calculate_E(pair_data):
F_est = pair_data["F_est"]
K1 = pair_data["K1"]
K2 = pair_data["K2"]
kpts1 = pair_data["kpts1"]
kpts2 = pair_data["kpts2"]
mask = pair_data["kpts_mask"]
# calculate E_est
E_est = K2.T @ F_est @ K1
_, R_est, t_est, _ = cv2.recoverPose(E_est, kpts1, kpts2, mask=mask)
pair_data["R_est"] = R_est
pair_data["t_est"] = t_est
"""
from dkm.
Could you share with me what estimator you used? Was it the opencv estimator used for mega-1500?
I found that the estimation error could be very different for different sample (using same sampling method) , the difference could reach 5°, is this normal? Is there any way to make the camera pose estimation more stable?
from dkm.
I can not reproduce. Could it be that what you are plotting are inliers after RANSAC? It might be the case that the estimation converges to a degenerate case with the camera being mirrored? (saying this because it seems like the translation error is low).
It seems like the points are approximately lying on a plane. If you estimated a fundamental matrix, there might be degenerate cases.
That's right, points plotted are only inliers
from dkm.
I can not deeply investigate (lack of time), but I'm guessing that it might be the case that the very low inlier threshold for these images might create issues. Megadepth has issues with incorrect intrinsics, meaning that some pairs will inherently not match perfectly. When you enforce a very strict threshold these types of degenerate solutions may appear. One easy check would be to see if R is in the wrong direction (e.g. points behind the camera etc.). If you have K you could estimate E directly as well.
from dkm.
that the very low inlier threshold
Well It already help a lot! Great thanks
Just one more question, Do you think the performance of the network on the megadepth dataset already being saturated?
from dkm.
Related Issues (20)
- 3d point projection, best way to fetch matches HOT 6
- `numpy.random.choice` to pytorch? HOT 6
- run on batch data HOT 5
- the structure and information of 'warp' HOT 5
- what does low_res_certainty do in dkm.py? HOT 3
- About the pretrained model HOT 3
- Questions about global matcher HOT 3
- About the pretrained model with resnet18 HOT 1
- When using multi-GPU training, there is additional memory occupancy on GPU 0 HOT 1
- About testing results HOT 2
- DKMv3 and DKMv2 HOT 4
- Pretrained weights licensing HOT 2
- Questions about the key points on Megadepth test images HOT 2
- About previous sota pdc-net+ HOT 5
- torch.linalg.inv HOT 6
- synthetic dataset
- How to use GC-RANSAC for pose estimation? HOT 1
- question about loading the training images HOT 2
- if it's possible to use DKMv3 for training when there is no depth information in my own dataset? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dkm.