Comments (4)
Hi,
Thanks for you interest in our work.
One easy way is to directly use the input images instead of trying to re-render them using PyTorch3D.
If you still want to use PyTorch3D, I will be happy to help with setting it up and share some code. I think the following solution can help. Let me know how it goes and if you face any issue.
Essentially, what one needs is the camera intristics and extrinsics.
Attached is a replay_sample which is saved from here.
replay_sample.pkl.zip
Here is some boiler plate code to load the sample and preprocess it. The preprocessing is same as done here. Note that we cannot apply 3D augmentations in this case.
import torch
import pickle as pkl
import numpy as np
from PIL import Image
from pytorch3d.renderer import FoVPerspectiveCameras
import rvt.utils.peract_utils as peract_utils
import rvt.utils.rvt_utils as rvt_utils
from mvt.renderer import BoxRenderer
with open("replay_sample.pkl", "rb") as file:
replay_sample = pkl.load(file)
cameras = ["front", "left_shoulder", "right_shoulder", "wrist"]
obs, pcd = peract_utils._preprocess_inputs(replay_sample, cameras)
pc, img_feat = rvt_utils.get_pc_img_feat(obs, pcd)
For camera intrisics, you can use something like this.
fov_cam = FoVPerspectiveCameras()
cam_intr = {}
zfar = {
"front": 4.5,
"left_shoulder": 3.2,
"right_shoulder": 3.2,
"wrist": 3.5,
}
fov = {
"front": 40,
"left_shoulder": 40,
"right_shoulder": 40,
"wrist": 60,
}
for cam in cameras:
cam_intr[cam] = fov_cam.compute_projection_matrix(
znear=0.01, zfar=zfar[cam], fov=fov[cam], aspect_ratio=1.0, degrees=True)
Then you need to calculate the camera extrinsics and convert the values to PyTorch3D format. Here is an example:
R = []; T = []; K = []; scale = []
for cam in cameras:
# extrinsics for the zeroth sample
extr = replay_sample[f'{cam}_camera_extrinsics'][0]
assert extr.shape == (1, 4, 4), f"extr.shape={extr.shape}"
_R = extr[0:1, 0:3, 0:3]
_T = extr[0:1, 0:3, 3]
_T = (-_T[0] @ _R[0]).unsqueeze(0)
_scale = torch.ones(_T.shape)
_K = cam_intr[cam]
R.append(_R); T.append(_T); scale.append(_scale); K.append(_K)
R = torch.cat(R, 0)
T = torch.cat(T, 0)
K = torch.cat(K, 0)
scale = torch.cat(scale, 0)
dyn_cam_info = (R, T, scale, K)
Then we can render with PyTorch 3D as following:
renderer = BoxRenderer(img_size=(128, 128), device="cuda:0")
out = renderer(pc[0], img_feat[0], fix_cam=False, dyn_cam_info=[dyn_cam_info,])
img = (255 * out.cpu().numpy()).astype(np.uint8)
Image.fromarray(img.transpose(1, 0, 2, 3).reshape(128, 128*4, 3))
Hope this example is self-sufficient. Let me know if you have any other questions.
from rvt.
thanks you so much. this helps tremendously! just want to double check why we cannot do 3d augmentation in this case? from these real views, if we rerender using PyTorch3D after 3D augmentation, wouldn't that work?
from rvt.
We cannot do 3D augmentation as then the pose of the camera with respect to the point cloud would change. Then the images rendered won't be from the real camera poses.
from rvt.
Thank you!
from rvt.
Related Issues (20)
- Does the line 150 give a Cartesian coordinate of a pixel? HOT 6
- Why is time an essential part of low_dim_state? HOT 2
- Can I run eval.py with multiple processes? HOT 2
- Questions about inference speed HOT 2
- How to train and test with an image_size resolution of 256*256? HOT 4
- About models training HOT 2
- About model training in the real-world HOT 2
- add_final and repeat in replay HOT 4
- [Training] How to evaluate when training? HOT 2
- Questions about the details HOT 5
- About the tp1 in replay HOT 3
- Does the RVT model control Franka robot in Isaac-sim? HOT 4
- Data collection issues HOT 4
- question about camera calibration in real world HOT 8
- AttributeError: 'CustomMultiTaskRLBenchEnv2' object has no attribute 'get_ground_truth_action' HOT 1
- Rendered image with color noise in the background HOT 4
- Question about the up direction of "front" and "back" camera in renderer.py HOT 2
- Details about the real-world experiments HOT 4
- Questions about the scene bounds and img aug in real-robot experiments HOT 4
- Issues while running the training script
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rvt.