Comments (10)
Hello,
thank you for your interest in this repo.
Actually, the function inverse_warp does implement a 6DoF pose to extrinsics function :
SfmLearner-Pytorch/inverse_warp.py
Line 136 in 4e6b7e8
So you would have to change the code a little to use matrices instead of pose vectors, and thgen this line will have to go :
SfmLearner-Pytorch/inverse_warp.py
Line 177 in 4e6b7e8
One quick warning with using matrices instead of pose vectors is the output of thge network. It's much more stable to use pose vectors than extrinsics matrices as output to train if every ceofficient of the outputted matrix is a learnable parameter. So the best strategy for you is to have the euler2mat function directly embedded in your network that you will call at each forward.
That way, your network will output directly matrices instead of pose vectors, allowing you to do the inverse wapr with matrices, and will still be stable as if it was using pose vectors.
Hope it helped,
Clément
from sfmlearner-pytorch.
Thanks for your reply!
Yes I am aware that pose can be converted to a matrix by using pose_vec2mat
function.
What I am asking is whether it is possible to find the equivalence of pose
in the form of extrinsics and intrinsics
from sfmlearner-pytorch.
The code already uses an intrinsics matrix separately from pose, so the pose is totally independent from intrinsics.
when we convert pose to a 4x3 matrix, we actually get the extrinsics matrix, so there they are equivalent.
from sfmlearner-pytorch.
So which extrinsics is pose
equivalent to? The target image extrinsics or the source image extrinsics? Or is it
extrinsics_src @ extrinsics_tgt.inverse()
Judging from the code, I believe pose
represents the transformation from target pixel coord to source pixel coord, so a single extrinsics might not suffice.
from sfmlearner-pytorch.
In this setup, the extresincs of target image is always the identity matrix (or the null pose vector)
There is no way for the network to know the pose of both target and reference image, it can only estimate the difference between the two. In other words, it can estimate the extrinsincs of reference image in a coordinate system centered around target image.
If you want the pose of both target and reference images, you need an anchor somewhere. If the anchor is e.g. the first image of the whole sequence, and you need the pose of the Nth image relative to the first frame, you will need to accumulate the pose differences and thus compute the compisition of several extrinsics because they are not written in the same coordinate system.
from sfmlearner-pytorch.
Sorry for the unclear quesrtion. I am currently trying to accommodate your inverse_warp
to my own dataset without using the pose prediction network, because my dataset provides extrinsics and intrinsics.
I technically could just use the network for pose
and call it a day but I just want to explore more possibilities considering the extrinsics and intrinsics are easy to obtain with specialized hardwares so a network is not needed.
from sfmlearner-pytorch.
Oh, now I understand, sorry !
If you have extrinsics for both view, then yes the inverse warp does use the difference of extrinsics inplicitly, which results in your proposed formula in your first pose.
Now the tricky part is to make sure the order is right. Is it intrinsics @ extrinsics_src @ extrinsics_tgt.inverse() @ intrinsics_inverse() @ target_depth
or is it intrinsics @ extrinsics_tgt.inverse() @ extrinsics_src @ intrinsics_inverse() @ target_depth
? Honestly, the best way to be sure is trial and error by visualizing the result.
Hope it helped !
from sfmlearner-pytorch.
I did some quick visualization and here are the results:
intrinsics @ extrinsics_src @ extrinsics_tgt.inverse() @ intrinsics_inverse() @ target_depth
:
intrinsics @ extrinsics_tgt.inverse() @ extrinsics_src @ intrinsics_inverse() @ target_depth
:
it seems that the first one is more correct to me. The second one contains some pixels that was unseen in target image.
Appreciate your input here!
from sfmlearner-pytorch.
Hard to tell without the depth (are you using groundtruth from the sensor ?) but I'd agree with you that first one looks better, especially for the chair onn the bottom.
Note that the duplication of pixels in the first is normal for view that are occluded on ref image but not on tgt image. It's impossible to reconstruct them since they are not visible and thus the algorithm take the color of foreground. It is visible for the table in bottom as well for example.
from sfmlearner-pytorch.
This is the GT depth of the target image. It is actually taken from a synthetic dataset not real life scene.
Also I have a small question, if the source and target images do not have any overlap, does this warp still work?
from sfmlearner-pytorch.
Related Issues (20)
- Large Errors on Pose Prediction Network HOT 3
- why the gpu memory cost of tensorflow version is larger than pytorch version HOT 2
- Weird results from pretrained model on KITTI images HOT 4
- Question about using oxts data HOT 1
- Cannot run `train.py` with nohup HOT 2
- imread during inference load the image as uint8 HOT 4
- How About the Flops, fps and parameter of this model? HOT 1
- regarding the predicted depth map during inverse warp HOT 2
- How to visualize the warped image (ref_img_wapred) HOT 2
- Is the image input of depth network fixed? HOT 2
- Question about diff
- How to load training dataset
- Regarding the depth used for generating target image HOT 5
- Question about the poses predicted by the posenet HOT 2
- about the pose scale HOT 2
- difference of the predicted translation and ground truth vectors HOT 7
- samples in test_files_eigen dont exist in the KITTI
- Seq 10 is not in the test_secens.txt HOT 2
- raw_data_downloading error HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sfmlearner-pytorch.