Comments (9)
This is an extra feature that is not included in the original project, that is why it is not very well documented right now.
The way it is implemented right now is that P = K*H, where K is the intrinsic camera matrix and H is the projection matrix from world to camera coordinates. I have also some options for lens distortion coefficients that will default to 0.
I haven't tested all possible scenarios, so if you think there is a bug in the implementation, can you come up with a simple reproducible example where we can easily compute the ground truth (i.e. render a plane) to help me fix it?
from neural_renderer.
I think there is a bug, because I also tried multiplying P with the intrinsic camera matrix, and the results are wrong. I'll try doing that in the following days.
from neural_renderer.
I've performed a simple test with a cube and it seems to be ok, but flipped (didn't check the actual depth values, just the projection).
However, for my *.obj it's not working.
import torch
import numpy as np
import neural_renderer as nr
import matplotlib.pyplot as plt
cuda0 = torch.device('cuda:0')
v = torch.tensor([[-1., -1., -1.],
[-1., -1., 1.],
[-1., 1., -1.],
[-1., 1., 1.],
[ 1., -1., -1.],
[ 1., -1., 1.],
[ 1., 1., -1.],
[ 1., 1., 1.]], device=cuda0)
f = torch.tensor([[ 0, 6, 4],
[ 0, 2, 6],
[ 0, 3, 2],
[ 0, 1, 3],
[ 2, 7, 6],
[ 2, 3, 7],
[ 4, 6, 7],
[ 4, 7, 5],
[ 0, 4, 5],
[ 0, 5, 1],
[ 1, 5, 7],
[ 1, 7, 3]], dtype=torch.int32, device=cuda0)
P = np.array([[ 6.237720e+02, -2.220966e+03, -9.922800e+01, 2.653734e+03],
[ 1.730178e+03, 3.091560e+02, -1.270062e+03, 1.904076e+03],
[ 7.797000e-01, -2.391000e-01, 5.787000e-01, 4.259900e+00]])
v = v*.5 # just because the original is way too large for this P
# project the vertices to compare
vs = np.pad(v.data.cpu().numpy(), ((0,0), (0,1)), mode='constant', constant_values=1)
vs_px = np.matmul(vs, P.transpose())
# neural renderer
t_P = torch.FloatTensor(np.expand_dims(P, 0)).cuda()
renderer = nr.Renderer(camera_mode='projection', P=t_P, image_size=1000)
im = renderer.render_depth(v[None, :, :], f[None, :, :])
#
plt.imshow(im.data.cpu().numpy()[0])
plt.scatter(vs_px[:, 0]/vs_px[:, 2], vs_px[:, 1]/vs_px[:, 2], s=200)
plt.show()
a = im.data.cpu().numpy()[0]
plt.imshow(np.flip(a, axis=0))
plt.scatter(vs_px[:, 0]/vs_px[:, 2], vs_px[:, 1]/vs_px[:, 2], s=200)
plt.show()
from neural_renderer.
Is it possible to provide the code with your .obj file also? I am quite busy these few days but I can test everything later this week.
from neural_renderer.
@nkolot the formula used in projection.py is obviously wrong, if K is the camera intrinsics, then P = K * H actually maps 3D points into pixel coordinates (i.e., in the range [0, 640] for example). You cannot apply distortion coefficients on this.
from neural_renderer.
@nkolot I am trying to fix this and will try to do a pull request once it is done .
from neural_renderer.
Cool, thanks. I haven't had much time lately to look at the issues.
from neural_renderer.
@liupeidong88 Just letting you know that I fixed this issue today. I need to take care of some additional things and I will merge the changes.
from neural_renderer.
I merged a fix. @pmcrodrigues can you check if you are getting the expected results now?
from neural_renderer.
Related Issues (20)
- Obtaining silhouettes of multiple objects
- i get an error when i try to save obj with textures HOT 3
- ValueError: not enough values to unpack (expected 3, got 1) HOT 3
- cuda10.2 torch1.7.1 install fail
- problem about 'python setup.py install' HOT 1
- it haves some error in cuda11.4 HOT 3
- The variable of camera_up is not defined
- cannot run renderer on any other GPU other than GPU 0 HOT 1
- ImportError: /data/user/anconda3/envs/neural_test/lib/python3.7/site-packages/neural_renderer/cuda/load_textures.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceIdEEPKNS_6detail12TypeMetaDataEv HOT 3
- Error in forward_face_index_map_1: invalid device function Error in forward_face_index_map_2: invalid device function Error in forward_texture_sampling: invalid device function HOT 2
- Render Silhouette For A Trimesh Object
- Render 2D Silhouette For an Input Image
- Support for macOS Apple chip HOT 1
- fails to build on Google Colab with CUDA 10.2
- RuntimeError: Error compiling objects for extension even after changing the ``AT_CHECK`` HOT 5
- The problem of python setup.py install HOT 3
- For those who are new to the installation errors HOT 4
- unknown setup problem, RTX 3090 + cuda 11.1, already convert AT_CHECK to TROCH_CHECK HOT 2
- ValueError: mode should be one of None, 'silhouettes' or 'depth' HOT 1
- pip install neural_renderer_pytorch ERROR: Could not build wheels for neural_renderer_pytorch, which is required to install pyproject.toml-based projects HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from neural_renderer.