parskatt / dkm Goto Github PK
View Code? Open in Web Editor NEW[CVPR 2023] DKM: Dense Kernelized Feature Matching for Geometry Estimation
Home Page: https://parskatt.github.io/DKM/
License: Other
[CVPR 2023] DKM: Dense Kernelized Feature Matching for Geometry Estimation
Home Page: https://parskatt.github.io/DKM/
License: Other
Thanks for your excellent work.
I noticed a confusing detail. When you load images from Megadepth dataset, you resize the image to a fixed ht
and wt
, that will change the original aspect ratio(some images may have been taken vertically). I'm wondering why you do not maintain the aspect ratio and pad the image to the specified size, which is a common practice in other computer vision tasks. Is it because the padded areas significantly interfere with the estimation of the warp? If so, would masking out the warp generated by the padded areas be a good solution?
Thanks again!
Hi,
Thank you for sharing your work. It's been helpful for my research.
However, I'm quite new to the dense matching method and have some simple questions regarding the scannet data test code.
I noticed that the error computing process is iterated five times. Is it a normal metric in other dense matching tasks like loftr or PDC-Net+?
https://github.com/Parskatt/DKM/blob/main/dkm/benchmarks/scannet_benchmark.py#L98
for _ in range(5):
shuffling = np.random.permutation(np.arange(len(kpts1)))
...
Additionally, I saw another line that adds errors after the loop. This might include the same error twice.
https://github.com/Parskatt/DKM/blob/main/dkm/benchmarks/scannet_benchmark.py#L120
for _ in range(5):
...
tot_e_t.append(e_t)
tot_e_R.append(e_R)
tot_e_pose.append(e_pose)
tot_e_t.append(e_t)
tot_e_R.append(e_R)
tot_e_pose.append(e_pose)
Thanks in advance.
Congratulations on the great work!
Do you have plans for releasing the training code ? I'm thinking on fine-tuning your weights for more specific tasks.
Great job! I do inference on 3090, it takes about 0.7 seconds to calculate the time of two images, and what other operations can be used for inference acceleration, in addition to reducing the image resolution.
Hello, may I ask if it's possible to use DKMv3 for training when there is no depth information in my own dataset? I only need to obtain the horizontal and vertical offsets for perspective transformation.
It seems that dkm_model.match doesn't accept batch input now?
Hi Johan,
When I train about 160k steps, I got the checkpoint for testing, and then obtained different testing results. For example, 1st testing result: 58.345 (auc5), 2nd testing result: 59.449, 3th testing result: 59.853.
I think this is caused by the randomness of sampling. Will this randomness be alleviated as the number of iterations increases?
Thank you so much for your help!
Why does PDC-net perform reasonably well in pck but much worse in two-view geometry estimation?
pdc-net pck:
Is it because the confidence it predicts is learned self-supervised? Also is it fair to compare DKM with pdc-net without retraining it fully supervised?
If I understand it wrong, please point it out, Thank you! : )
Hi, your great work is really amazing ! Can't wait for full release demo of DKMv3 . When do you intend to do that ?
Hi! In Table 8 of the original paper, do you keep the test image resolution the same as the training image resolution? For example, when training on 384x512 image pairs, do you also resize all the test image pairs to 384x512 for testing? Actually I am following roma. But I found this dense pipeline is a bit sensitive to the resolution setting. So I want to find a way to make the method generalize well to different resolutions. :)
In DKM's sample method, do you intend to set any certainty greater than 0.05 to 1, instead of, perhaps, setting any certainty less than 0.05 to 0?
Line 574 in fb85d63
I ask, because you then go on to sample from the matches based on that certainty which can result in points with certainty less than 0.05 being sampled. In fact, if all the matches have certainty below 0.05, then expansion_factor*num
matches would still be selected.
Hi!
While trying to rerpduce the results using Mega + Synthetic Dataset using "train_mega_synthetic.py", I notice that in the training code, model is set to be DKM (version 1), in Line 31-33. Does this indicate the training scripts "train_mega_synthetic.py" is designed for DKM (v1). Or the scripts are suitable for both versions?
Thanks!
Thank you for your outstanding work!!!
I'm still working on my final project, where I'm trying to estimate the rotation angle between two identical objects. Initially, I tried using the SIFT algorithm, but the results were terrible. Therefore, I'm exploring deep learning methods to calculate the angles.
However, I'm encountering an issue that I hope you can help me with. I'm experiencing an issue where I get a match point regardless of the background color I set. The original image on the left has no background, but it seems like the algorithm is automatically adding a background. I'd like to know if there's a parameter to avoid this behavior.
Steps to Reproduce:
Could you teach me how to run demo_match.py without GPUs ?
Hi, i have this problem:
image1.shape, image2.shape : 3, 256, 256
dkm_model = DKMv3_outdoor(device=device)
dkm_model.w_resized = 256
dkm_model.h_resized = 256
dkm_model.upsample_preds = False
warp, certainty = dkm_model.match(image1, image2, batched=True, device=device)
then all values in 'certainty' tensor is zeros (torch.sum(certainty) is 0.0), so this line code "matches, certainty = dkm_model.sample(warp, certainty)" will cause error:
603 expansion_factor = 4 if "balanced" in self.sample_mode else 1
--> 604 good_samples = torch.multinomial(certainty,
605 num_samples = min(expansion_factor*num, len(certainty)),
606 replacement=False)
607 good_matches, good_certainty = matches[good_samples], certainty[good_samples]
608 if "balanced" not in self.sample_mode:
RuntimeError: invalid multinomial distribution (sum of probabilities <= 0)
Like SuperGlue and LoFTR did
What is the difference between DKMv3 and DKMv2?
I noticed that you use GC-RANSAC for pose estimation. I tried to look into the GC-RANSAC repository but couldn't get it to work. I was hoping to take a look at your code and inquire about which version of GC-RANSAC you are using?
I am performing an image matching test on demo_match.py. What is the structure of the warp obtained in this code ‘warp, certainty = dkm_model.match(im1_path, im2_path, device=device)’, and what is the information stored in it?I want to get the pixel matching relationship of two pictures through ‘warp’, how should I do it?Thank you very much!
Thanks for releasing such a great package.
When I import dkm at my custom script, just importing makes additional memory occupancy on gpu 0.
Moving the following line to tensor_to_pil() seems to fix this.
imagenet_mean = torch.tensor([0.485, 0.456, 0.406]).to(device)
imagenet_std = torch.tensor([0.229, 0.224, 0.225]).to(device)
(https://github.com/Parskatt/DKM/blob/main/dkm/utils/utils.py#L294)
Could you check?
Hi Johan, thanks for your excellent work!
I wonder why np.random.choice is used for selecting good_samples instead of ordering the confidences. Is it for selecting more sparse keypoints?
Hello @Parskatt ,
Congratulations on your paper acceptance! :D
I want to know the details about how many GPUs you use, the capacity memory of each, and how much GPU memory is consumed during your training.
Thank you!!!
Hi,
I have known camera extrinsics/intrinsics and I like to 3d project the pixels of 2 images matched by DKM. What would be a good point to get the matches between the images?
Thanks!
Daniel
Hi Johan,
Thanks for your great contribution,
I noticed that you used Gaussian processes to encode feature maps in the global matcher. We find this approach very novel and completely different from the global 4D-correlation volume used in previous methods.
We wondered what motivates you to use Gaussian processes to model this, and why is the Gaussian process suitable for solving this warp mapping prediction problem?
Best wishes,
Weiguang Zhang
Congratulations on this amazing result. Where can I read your latest papers?
LoFTR uses only 15,300=153*100 image training pairs from MegaDepth.
While DKM iteratively sample 150,000 pairs each iteration from 10,661,614 total traning pairs in 53 iterations.
Great work!
I wonder what is the effect of
"
low_res_certainty = factorlow_res_certainty(low_res_certainty < cert_clamp)
...
dense_certainty = dense_certainty - low_res_certainty
" in dkm/models/dkm.py?
Also, shouldn't this
"
query_coords = torch.meshgrid(
(
torch.linspace(-1 + 1 / hs, 1 - 1 / hs, hs, device=device),
torch.linspace(-1 + 1 / ws, 1 - 1 / ws, ws, device=device),
)
)
" be
"
query_coords = torch.meshgrid(
(
torch.linspace(-1 + 1 /( 2hs), 1 - 1 /(2 hs), hs, device=device),
torch.linspace(-1 + 1 /( 2* w)s, 1 - 1 /( 2* ws), ws, device=device),
)
)
"?
Thanks!
Hi Johan,
First of all, I want to express my gratitude for your recent contributions to the field, particularly your papers DKM and RoMa. They have been a great source of inspiration and motivation for my own research endeavors.
However, while reviewing the accompanying code implementation, I noticed some discrepancies regarding the Coordinate Embedding with random Fourier features because of the default behaviour of pytorch.
In the paper, it's stated that the entries for Fourier basis frequencies are sampled from a normal distribution. It is implemented using a Conv2d layer in pytorch. However, I found that the default initialization for Convolutional weight is Kaiming uniform.
In addition, I found that the Fourier basis frequencies are part of trainable parameters instead of fixed sampling in a previous work
I've visualized the histogram of the pretrained pos_conv
weights
with the following code:
import torch
from torch import nn
from dkm import DKMv3_outdoor
from roma import roma_outdoor
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('darkgrid')
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
dkm_weight=DKMv3_outdoor(device=device).decoder.gps['16'].pos_conv.weight.detach().cpu().numpy()
roma_weight=roma_outdoor(device=device, coarse_res=560, upsample_res=(864, 1152)).decoder.gps['16'].pos_conv.weight.detach().cpu().numpy()
random_weight=nn.Conv2d(2, 256, kernel_size=(1, 1), stride=(1, 1)).weight.detach().cpu().numpy()
plt.figure(figsize=(8,5))
plt.hist(random_weight.flatten(), bins=25, alpha=0.5,density=True, label='Init',range=(-1.5,1.5))
plt.hist(roma_weight.flatten(), bins=25, alpha=0.5,density=True, label='ROMA',range=(-1.5,1.5))
plt.hist(dkm_weight.flatten(), bins=25, alpha=0.5,density=True, label='DKM',range=(-1.5,1.5))
plt.legend()
plt.title('Histogram of the weights pos_conv')
The resulting distribution appears more Gaussian, possibly due to the weight decay in the AdamW optimizer, which assumes a normal prior for parameters.
Based on insights from a previous work, I suspect that the initialization distribution might not matter greatly, with only the standard deviation being crucial. However, I'm curious about whether training could lead to better Fourier basis frequencies. I would greatly appreciate it if you could further investigate these aspects.
Thank you once again for your exceptional contributions to the field.
Thanks for your amazing work!
While trying to utilize this amazing work, I can not understand on implementation in the code.
In funciton "upsample_preds" (used only in function match), the final flow is further refined using "self.conv_refiner", however, the estimated residual flow is not added over the original final flow. Instead, a re-sampling is executed.
query_coords = torch.meshgrid((
torch.linspace(-1 + 1 / h, 1 - 1 / h, h, device="cuda:0"),
torch.linspace(-1 + 1 / w, 1 - 1 / w, w, device="cuda:0"),
))
query_coords = torch.stack((query_coords[1], query_coords[0]))
query_coords = query_coords[None].expand(b, 2, h, w)
delta_certainty, delta_flow = self.conv_refiner["1"](query, support, flow)
displaced_query_coords = torch.stack((
query_coords[:, 0] + delta_flow[:, 0] / (4 * w),
query_coords[:, 1] + delta_flow[:, 1] / (4 * h),
),dim=1,)
flow = F.grid_sample(flow, displaced_query_coords.permute(0, 2, 3, 1))
It's a great work!!! But I met some problem, when I ran your training code, the weights named "train_DKMv3_outdoor_latest.pth" was 852MB. However, the model you provided with the training code is only 258MB. I can successfully run the provided model, but I encounter errors when using my own trained model. Where is the problem and how can I obtain a trained model of the same size that is able to run properly like the one you provided?
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for RegressionMatcher:
Missing key(s) in state_dict: "encoder.net.conv1.weight", "encoder.net.bn1.weight", "encoder.net.bn1.bias", "encoder.net.bn1.running_mean", "encoder.net.bn1.running_var", "encoder.net.layer1.0.conv1.weight",
encoder.net.layer1.0.bn1.weight", "encoder.net.layer1.0.bn1.bias", "encoder.net.layer1.0.bn1.running_mean", "encoder.net.layer1.0.bn1.running_var", "encoder.net.layer1.0.conv2.weight", "encoder.net.layer1.0.bn2.weight",
......
.conv_refiner.1.disp_emb.weight", "decoder.conv_refiner.1.disp_emb.bias".
Unexpected key(s) in state_dict: "model", "n", "optimizer", "lr_scheduler".
Hi,
I have quickly checked the possibility to strip out training code and integrate the bare DKM into kornia.
It seems that the main blocker is numpy.random.choice
Line 585 in b3311e2
And the Pytorch doesn't have it
pytorch/pytorch#16897
Is there a way we can use existing pytorch functionality to get a similar thing?
Hi Johan,
Thank you for your wonderful work.
I am attempting to fine-tune the outdoor weight using the KITTI dataset. I followed the code from train_DKMv3_outdoor.py and made modifications to the MegadepthBuilder and MegadepthDenseBenchmark for the KITTI odometry dataset. After training for only 1,000 steps, I observed that the AUC5 result dropped from 0.7 to 0.3. Could there be an issue with my modifications, or are there specific training details I should pay attention to (such as resolution, depth, etc.)?
Thank you for your assistance!
The work is very exciting! I ask about the scaling process in the following code. Why divide by 4 here? Isn't ins * diaplacement[] supposed to restore to the original image, and /w is to normalize the result?
dense_flow = torch.stack(
(
dense_flow[:, 0] + ins * displacement[:, 0] / (4 * w),
dense_flow[:, 1] + ins * displacement[:, 1] / (4 * h),
),
dim=1,
)
When using
dense_matches, dense_certainty = model.match(img1PIL, img2PIL, check_cycle_consistency=False, do_pred_in_og_res=True)
I got:
738 if do_pred_in_og_res: # Will assume that there is no batching going on.
739 og_query, og_support = self.og_transforms((im1, im2))
--> 740 query_to_support, dense_certainty = self.matcher.upsample_preds(
741 query_to_support,
742 dense_certainty,
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
1129 return modules[name]
1130 raise AttributeError("'{}' object has no attribute '{}'".format(
-> 1131 type(self).__name__, name))
1132
1133 def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:
AttributeError: 'RegressionMatcher' object has no attribute 'matcher'
I belive you need to rename the variable self.matcher
to self.decoder
in line 740 of dkm.py (click here to see)
Hi Johan, I really appreciate your great work.you have provided the pretrained model with resnet50.I noticed from the code that you also have a model for resnet18,Is it possible to provide the pretrained model with resnet18? Thank you very much!
Hey!
First of all great work, I was wondering if we can get a match score of 2 images from this model?
I noticed that DKM resizes the image to a fixed size and aspect ratio (horizontal). I think preserving the original aspect ratio of the images could improve performance, especially for extreme image pairs like this one:
There is any tweak (code modification) to allow this into the current DKM weights?
Hello author, thank you for your outstanding work! My input image resolution is (3648,5742), and getting the homography matrix is not ideal, I tried to change the resolution in the "int.py " file, then setting it to "return DKMv3(weights, 3648, 5472, upsample_preds = False, device= device)", which still fails to approach the true geometric transformation relationship. How can I improve it?
Can you post that part of the code that works on the synthetic dataset? I want to try it on cross-spectrum
Hi Johan, I really appreciate your great work. Is it possible to provide the pretrained model here? Thank you very much!
Hello. Not completely sure if this is an issue, but I am a bit confused about what the conf_thresh
parameter should be used for. I'm referring to this line in dkm.py. Shouldn't this line come after the filtering with relative_confidence_threshold
? Is it correct to say that conf_thresh
filters out outliers and the inliers are given the same probability?
Thanks for your great work and impressive results!
Since your results are so impressive, it is straightforward to consider combing your work as part of the downstream task, like visual odometry. So I am curious about the inference time for DKM compared to other methods, like LoftR or optical flow method RAFT. Have you done any experiments like this? I haven't seen any report about inference time in paper, so it will be so nice for you to help.
Dear @Parskatt, dear authors, thank you for this repo! Great work!
Could you please clarify on the license of the provided DKMv3 model's weights for indoor and outdoor? Were they trained on datasets that imply open license?
Thank you in advance!
hi,parskatt, how to export a onnx model?
Hello author, I wonder how you painted the Figure 4. Coordinate embeddings preserve multimodality in the main paper. Cause even use the coordinate as the GP output, the GP posterior couldn't give out a two-modal result as in the figure4.
Thanks for your excellent work! May I ask for one possible solution for the problem shown as below? Thank you so much!
Traceback (most recent call last):
File "experiments/dkm/train_DKMv3_outdoor.py", line 259, in
train(args)
File "experiments/dkm/train_DKMv3_outdoor.py", line 250, in train
wandb.log(megadense_benchmark.benchmark(model))
File "/mnt/data-disk-1/home/cpii.local/wtwang/IM/codes/DKM/dkm/benchmarks/megadepth_dense_benchmark.py", line 72, in benchmark
matches, certainty = model.match(im1, im2, batched=True)
File "/mnt/data-disk-1/home/cpii.local/wtwang/IM/codes/DKM/dkm/models/dkm.py", line 695, in match
dense_corresps = self.forward(batch, batched = True)
File "/mnt/data-disk-1/home/cpii.local/wtwang/IM/codes/DKM/dkm/models/dkm.py", line 631, in forward
dense_corresps = self.decoder(f_q_pyramid, f_s_pyramid)
File "/mnt/data-disk-1/home/cpii.local/wtwang/miniconda3/envs/im/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/mnt/data-disk-1/home/cpii.local/wtwang/IM/codes/DKM/dkm/models/dkm.py", line 494, in forward
new_stuff = self.gps[new_scale](f1_s, f2_s, dense_flow=dense_flow)
File "/mnt/data-disk-1/home/cpii.local/wtwang/miniconda3/envs/im/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/mnt/data-disk-1/home/cpii.local/wtwang/IM/codes/DKM/dkm/models/dkm.py", line 360, in forward
K_yy_inv = torch.linalg.inv(K_yy + sigma_noise)
torch._C._LinAlgError: linalg.inv: (Batch element 0): The diagonal element 512 is zero, the inversion could not be completed because the input matrix is singular.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.