javieryu / nerf_bridge Goto Github PK

Is it possible to export the mesh or point cloud after reconstruction.

Hello!

While trying to use the viewer to load trained models, it gives an error that the directory does not exist. Even after specifying the checkpoint file directly, it is not able to load. Perhaps it could be a basic error with the directory path. But the path seems to be correct.
Would appreciate any insight on this !

So far I have only uploaded the nerf_bridge code to the server and installed Ubuntu20.04 and ROS Noetic on the Raspberry PI in the machine car. What should I do next? How to configure orb_slam3_ros? How do you connect nerf_brudge to nerfstudio?
Please help me,please!

git submodule add https://github.com/chaconinc/DbConnector

results

(nerfstudio) rover@rover-Precision-3571:~$ git submodule add https://github.com/chaconinc/DbConnector
fatal: not a git repository (or any of the parent directories): .git

RuntimeError: Error(s) in loading state_dict for NerfactoModel:
size mismatch for field.embedding_appearance.embedding.weight: copying a param with shape torch.Size([500, 32]) from checkpoint, the shape in current model is torch.Size([0, 32]).

please can you please explain to me what you I should save. A code snippet would be highly appreciated.

Hi,
While config.yml files are created in the correct locations, checkpoints are not being saved. Training is running fine but cannot view previous runs.

The nerfstudio_models directory is not being created.

Any help would be appreciated!
thank you!

i'm trying to test the package with gazebo sim before moving on to actual bot and im facing the following issue
this is my JSON file for sim

{
    "camera_model": "PINHOLE",
    "fx": 1696.802685832259,
    "fy": 1696.802685832259,
    "cx": 960.5,
    "cy": 540.5,
    "W": 1920,
    "H": 1080,
    "image_topic": "/camera/image_raw",
    "pose_topic": "/odom"
}

it loads only the first image and a blank white frame in the middle.

Things i tried:

• syncing image and the odom topics (did not work)
• tuning update_freq, topic_sync(to approx) and topic_slop
• custom rosbag data, it loads two images as shown below,

{
    "camera_model": "PINHOLE",
    "fx": 532.9841918945312,
    "fy": 532.9841918945312,
    "cx": 550.3889770507812,
    "cy": 303.86602783203125,
    "W": 1104,
    "H": 621,
    "image_topic": "/zed_node/rgb/image_rect_color",
    "pose_topic": "/zed_node/odom"
}

heres the bag i used.

I try to install the nerfstudio in your repo, but the requirements for torch and cuda might be old, and I cannot successfully install tiny-cuda-nn for it. So the older version cannot run.
So I install the latest nerfstudio, and it can run as well, except the viewer's camera updating logics, because the code ViewerState in nerfstudio has changed so much. I haven't found a proper way to update the cameras till now.

When trying to install the ROS bag file to test the build of this repo on my machine the python file provided ends up creating a .db3 file is there a simple way around this or is there any other place to download a ROS bag.

Thanks

Hello, I have started the SLAM algorithm and played back the rosbag with images. Then, I entered the command 'python ros_train.py ros_nerfacto --data /home/pf/CODE/NERF/nerf_bridge/nsros_config_sample.json', but I received an error message saying 'ros_train.py: error: unrecognized arguments: ros_nerfacto'. The image publishing frequency is 15Hz and the odometry frequency is 10Hz. Also, I have modified the camera intrinsic parameters and topics accordingly in the json file.May I know what are the possible solutions to this issue? Thank you!

Thank you for your good job! I have two questions:
1) When I run orbslam3 calling nerfstudio, only two image frames are displayed, but it seems to be able to render the approximate scene. Is this normal?

 2) Can the intrinsic parameters provided by the dataset be copied directly into the JSON file?

Thank you very much if you can help answer.

After running the ros_train. it trains but i can visualize the rendering. I am running the nerf_bridge in a docker container. Any solution availablle

I am having issues with the orbslam subscriber and also opencv problems

Thank you

Hi.
I noticed that you are currently working on the ros2 branch, can it work as well as the main branch？

Hi!

• Cloned the project
• Initialized "Nerfstudio" submodule
• Installed all of Nerfstudio's dependencies
• Executed "pip install -e ." inside the nerfstudio directory
• Run: "python ros_train.py ros_nerfacto --data nsros_config_sample.json"
• Getting parsing error:
  

It seems that the ros_nerfacto method is not recognized.
Does anybody know the fix for this?

Much appreciated!

(
camera_optimizer=CameraOptimizerConfig(

CameraOptimizerConfig has been moved from the DataManager to the Model.

/home/eiyike/NERF_BRIDGE_WRKSPACE/nerf_bridge/nsros/method_configs.py:36: FutureWarning: above message coming from
datamanager=ROSDataManagerConfig(

CameraOptimizerConfig has been moved from the DataManager to the Model.

/root/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/tyro/_calling.py:242: FutureWarning: above message coming from
return unwrapped_f(*positional_args, **kwargs), consumed_keywords # type: ignore
[04:23:25] Using --data alias for --data.pipeline.datamanager.dataparser.data ros_train.py:56
──────────────────────────────────────────────────────── Config ────────────────────────────────────────────────────────
ROSTrainerConfig(
_target=<class 'nsros.ros_trainer.ROSTrainer'>,
output_dir=PosixPath('outputs'),
method_name='ros_nerfacto',
experiment_name=None,
project_name='nerfstudio-project',
timestamp='2023-11-23_042325',
machine=MachineConfig(seed=42, num_devices=1, num_machines=1, machine_rank=0, dist_url='auto', device_type='cuda'),
logging=LoggingConfig(
relative_log_dir=PosixPath('.'),
steps_per_log=10,
max_buffer_size=20,
local_writer=LocalWriterConfig(
_target=<class 'nerfstudio.utils.writer.LocalWriter'>,
enable=True,
stats_to_track=(
<EventName.ITER_TRAIN_TIME: 'Train Iter (time)'>,
<EventName.TRAIN_RAYS_PER_SEC: 'Train Rays / Sec'>,
<EventName.CURR_TEST_PSNR: 'Test PSNR'>,
<EventName.VIS_RAYS_PER_SEC: 'Vis Rays / Sec'>,
<EventName.TEST_RAYS_PER_SEC: 'Test Rays / Sec'>,
<EventName.ETA: 'ETA (time)'>
),
max_log_size=10
),
profiler='basic'
),
viewer=ViewerConfig(
relative_log_filename='viewer_log_filename.txt',
websocket_port=None,
websocket_port_default=7007,
websocket_host='0.0.0.0',
num_rays_per_chunk=20000,
max_num_display_images=512,
quit_on_train_completion=False,
image_format='jpeg',
jpeg_quality=90,
make_share_url=False,
camera_frustum_scale=0.1,
default_composite_depth=True
),
pipeline=VanillaPipelineConfig(
_target=<class 'nerfstudio.pipelines.base_pipeline.VanillaPipeline'>,
datamanager=ROSDataManagerConfig(
_target=<class 'nsros.ros_datamanager.ROSDataManager'>,
data=None,
masks_on_gpu=False,
images_on_gpu=False,
dataparser=ROSDataParserConfig(
_target=<class 'nsros.ros_dataparser.ROSDataParser'>,
data=PosixPath('/home/eiyike/NERF_BRIDGE_WRKSPACE/nerf_bridge/nsros_config_sample.json'),
scale_factor=1.0,
aabb_scale=0.8
),
train_num_rays_per_batch=4096,
train_num_images_to_sample_from=-1,
train_num_times_to_repeat_images=-1,
eval_num_rays_per_batch=4096,
eval_num_images_to_sample_from=-1,
eval_num_times_to_repeat_images=-1,
eval_image_indices=(0,),
collate_fn=<function nerfstudio_collate at 0x7f814dae9310>,
camera_res_scale_factor=1.0,
patch_size=1,
camera_optimizer=CameraOptimizerConfig(
_target=<class 'nerfstudio.cameras.camera_optimizers.CameraOptimizer'>,
mode='SO3xR3',
trans_l2_penalty=0.01,
rot_l2_penalty=0.001,
optimizer=AdamOptimizerConfig(
_target=<class 'torch.optim.adam.Adam'>,
lr=0.0006,
eps=1e-08,
max_norm=None,
weight_decay=0.01
),
scheduler=None
),
pixel_sampler=PixelSamplerConfig(
_target=<class 'nerfstudio.data.pixel_samplers.PixelSampler'>,
num_rays_per_batch=4096,
keep_full_image=False,
is_equirectangular=False
),
publish_training_posearray=True,
data_update_freq=1.0,
num_training_images=500
),
model=NerfactoModelConfig(
_target=<class 'nerfstudio.models.nerfacto.NerfactoModel'>,
enable_collider=True,
collider_params={'near_plane': 2.0, 'far_plane': 6.0},
loss_coefficients={'rgb_loss_coarse': 1.0, 'rgb_loss_fine': 1.0},
eval_num_rays_per_chunk=32768,
prompt=None,
near_plane=0.05,
far_plane=1000.0,
background_color='last_sample',
hidden_dim=64,
hidden_dim_color=64,
hidden_dim_transient=64,
num_levels=16,
base_res=16,
max_res=2048,
log2_hashmap_size=19,
features_per_level=2,
num_proposal_samples_per_ray=(256, 96),
num_nerf_samples_per_ray=48,
proposal_update_every=5,
proposal_warmup=5000,
num_proposal_iterations=2,
use_same_proposal_network=False,
proposal_net_args_list=[
{'hidden_dim': 16, 'log2_hashmap_size': 17, 'num_levels': 5, 'max_res': 128, 'use_linear': False},
{'hidden_dim': 16, 'log2_hashmap_size': 17, 'num_levels': 5, 'max_res': 256, 'use_linear': False}
],
proposal_initial_sampler='piecewise',
interlevel_loss_mult=1.0,
distortion_loss_mult=0.002,
orientation_loss_mult=0.0001,
pred_normal_loss_mult=0.001,
use_proposal_weight_anneal=True,
use_average_appearance_embedding=True,
proposal_weights_anneal_slope=10.0,
proposal_weights_anneal_max_num_iters=1000,
use_single_jitter=True,
predict_normals=False,
disable_scene_contraction=False,
use_gradient_scaling=False,
implementation='tcnn',
appearance_embed_dim=32,
camera_optimizer=CameraOptimizerConfig(
_target=<class 'nerfstudio.cameras.camera_optimizers.CameraOptimizer'>,
mode='SO3xR3',
trans_l2_penalty=0.01,
rot_l2_penalty=0.001,
optimizer=None,
scheduler=None
)
)
),
optimizers={
'proposal_networks': {
'optimizer': AdamOptimizerConfig(
_target=<class 'torch.optim.adam.Adam'>,
lr=0.01,
eps=1e-15,
max_norm=None,
weight_decay=0
),
'scheduler': None
},
'fields': {
'optimizer': AdamOptimizerConfig(
_target=<class 'torch.optim.adam.Adam'>,
lr=0.01,
eps=1e-15,
max_norm=None,
weight_decay=0
),
'scheduler': None
}
},

Hi, thank you for your great work!

While utilizing the ros-depth-splatfacto, I encountered an issue triggered by the latest gsplat package when executing the following command:

ns-train ros-depth-splatfacto --data configs/desk.json --pipeline.datamanager.use-compressed-rgb True --pipeline.datamanager.dataparser.scene-scale-factor 0.5 --pipeline.datamanager.data-update-freq 8.0

This resulted in the following error:

AssertionError: block_width must be between 2 and 16

This error has been addressed in the issue raised on the gsplat GitHub repository, indicating changes in the code behavior.

The solution is to modify the installation command from:

pip install git+https://github.com/nerfstudio-project/gsplat.git

to

pip install git+https://github.com/nerfstudio-project/[email protected]

as documented in the last line of Installation section 4.

Hi.
I am trying to execute nerf-bridge in ros2 using the ros2 branch that was recently posted in this repo. I already have the json filled with the intrinsic camera parameters as well as the topics. I am publishing the topics:

• "image_topic": "/camera/color/image_raw", that is a message of type sensor_msgs/Image.msg
• "pose_topic": "/visual_slam/tracking/odometry", that is a message of type nav_msgs/Odometry.msg

Both topics are published at the same time, every second (for now, only for testing purposes). The info of these topics are obtained from the poster dataset in nerfstudio (the json containing camera intrinsics, paths to the images and transformation matrices).

I use ns-train ros-nerfacto --data /path/to/config.json as you recommend in the readme.

It shows "Killed" before even starting to train:

By doing some logging, I've found out that the process dies when it does the self.pipeline config setup inside trainer.py (inside nerfstudio itself), in the setup function.
This is what the data parser shows in the config:

Any hint on what could be going on? Could it be related to the version of nerfstudio I'm using? (I just installed it so it should be the newest available). Or maybe could it be related to how I'm launching the process? Or the data?

Thanks :)

Thanks for building this interesting project and thanks for opening source.

When I try to deploy this with realsense d455 camera with Vins-Fusion for localization. bridge_nerf shows (NSROS) Waiting for for image streaming to begin ....

Since Vins-Fusion doesn't come up with stamped_pose, I manually modify the source code to make it publish stamped_pose with same freq as image information published by realsense camera node (30 Hz). (check it out how to modify pose publish rate for VINS_FUSION here HKUST-Aerial-Robotics/VINS-Fusion#92 (comment))

My environment come up with:
ubuntu noetic
ros1
exact version of Nerfstudio as README

@javieryu Thank you for your work. It meets an issue with exporting the pointcloud. I look forward to your favorable reply.

Hi, thanks for your great work.
I`m trying to deploy NeRFBridge on an edge computing Nvidia Jetson AGX Orin 64GB platform.
firstly, I git clone the repo by

git clone --recursive https://github.com/javieryu/nerf_bridge.git

and then install the nerfstudio module successfully.
However when I continue to work and got some trouble.
To my personal understanding, I need to

roslaunch orb_slam3_ros euroc_mono_inertial.launch

rosbag play ~/data/MH_01_easy.bag

, and then

python ros_train.py ros_nerfacto --data /home/hello/code/nerf_bridge/ns_orb3_euroc.json

but I got this error

(nerf_bridge) hello@hello-desktop:~/code/nerf_bridge$ python ros_train.py ros_nerfacto --data /home/hello/code/nerf_bridge/ns_orb3_euroc.json 
/home/hello/mambaforge/envs/nerf_bridge/lib/python3.8/site-packages/tinycudann/modules.py:53: UserWarning: tinycudann was built for lower compute capability (86) than the system's (87). Performance may be suboptimal.
  warnings.warn(f"tinycudann was built for lower compute capability ({cc}) than the system's ({system_compute_capability}). Performance may be suboptimal.")
Traceback (most recent call last):
  File "ros_train.py", line 29, in <module>
    from nsros.method_configs import AnnotatedBaseConfigUnion
  File "/home/hello/code/nerf_bridge/nsros/method_configs.py", line 17, in <module>
    from nerfstudio.models.nerfacto import NerfactoModelConfig
  File "/home/hello/code/nerf_bridge/nerfstudio/nerfstudio/models/nerfacto.py", line 35, in <module>
    from nerfstudio.fields.density_fields import HashMLPDensityField
  File "/home/hello/code/nerf_bridge/nerfstudio/nerfstudio/fields/density_fields.py", line 26, in <module>
    from nerfstudio.data.scene_box import SceneBox
ModuleNotFoundError: No module named 'nerfstudio.data.scene_box'

Could you give me some suggestions? Thanks very much.

Hello, about the code, there are some parts that I don't understand and I hope to get your help. At line 60 of ros_trainer.py, "self.pipeline.datamanager.train_image_dataloader.msg_status", we know that msg_status is a method of the ROSDataloader class. The ROSDataManager class instantiates an instance of ROSDataloader as train_image_dataloader in its setup_train method to call msg_status, but I don't understand why the instance of ROSDataloader, train_image_dataloader, can be called by datamanager because there doesn't seem to be any connection between datamanager and train_image_dataloader.

Hi, I'm confusied when I run orb_slam3_ros with nerf_brige, the viewer 's image stay still in the center and never have another image shows up, and the nerf nerver work good, is there anything I do wrong? Or is there any rosbag could verify the configuration is right?

optimizer is no longer specified in the CameraOptimizerConfig, it is now defined with the rest of the param groups
inside the config file under the name 'camera_opt'

/home/eiyike/NERF_BRIDGE_WRKSPACE/nerf_bridge/nsros/method_configs.py:42: FutureWarning: above message coming from
camera_optimizer=CameraOptimizerConfig(

CameraOptimizerConfig has been moved from the DataManager to the Model.

/home/eiyike/NERF_BRIDGE_WRKSPACE/nerf_bridge/nsros/method_configs.py:36: FutureWarning: above message coming from
datamanager=ROSDataManagerConfig(

CameraOptimizerConfig has been moved from the DataManager to the Model.

/usr/local/lib/python3.8/dist-packages/tyro/_calling.py:245: FutureWarning: above message coming from
return unwrapped_f(*positional_args, **kwargs), consumed_keywords # type: ignore
[01:16:16] Using --data alias for --data.pipeline.datamanager.dataparser.data ros_train.py:56
──────────────────────────────────────────────────────── Config ────────────────────────────────────────────────────────
ROSTrainerConfig(
_target=<class 'nsros.ros_trainer.ROSTrainer'>,
output_dir=PosixPath('outputs'),
method_name='ros_nerfacto',
experiment_name=None,
project_name='nerfstudio-project',
timestamp='2024-02-27_011616',
machine=MachineConfig(seed=42, num_devices=1, num_machines=1, machine_rank=0, dist_url='auto', device_type='cuda'),
logging=LoggingConfig(
relative_log_dir=PosixPath('.'),
steps_per_log=10,
max_buffer_size=20,
local_writer=LocalWriterConfig(
_target=<class 'nerfstudio.utils.writer.LocalWriter'>,
enable=True,
stats_to_track=(
<EventName.ITER_TRAIN_TIME: 'Train Iter (time)'>,
<EventName.TRAIN_RAYS_PER_SEC: 'Train Rays / Sec'>,
<EventName.CURR_TEST_PSNR: 'Test PSNR'>,
<EventName.VIS_RAYS_PER_SEC: 'Vis Rays / Sec'>,
<EventName.TEST_RAYS_PER_SEC: 'Test Rays / Sec'>,
<EventName.ETA: 'ETA (time)'>
),
max_log_size=10
),
profiler='basic'

Hi, I was wondering if you have any plans to update this to ROS2 eventually as ROS1 will eventually become deprecated. Thanks!

Which parameters should I provide if I use mine pose topic？

I am a student from China, I can not buy the camera you introduced, could you please tell me what other type of camera I can choose.

Great work! It seems that nerfStudio's pointcloud or mesh export does not retain the best quality when viewed in the web viewer. Can you show me the exported mesh file of the demo video? I need them to display on mesh file readers like meshlab or threejs.
Thanks.

Hello, when trying to train the the NeRF Bridge model using the command given to train the ROS2 bag I get an error that looks like the one pasted below. To install GSPLAT I used the recommended pip command to install the repo from source is there any way to resolve this?

(nerfbridge) charl@charl-XPS-8950:~$ ns-train ros-depth-nerfacto --data configs/desk.json --pipeline.datamanager.use-compressed-rgb True --pipeline.datamanager.dataparser.scene-scale-factor 0.5 --pipeline.datamanager.data-update-freq 8.0
Traceback (most recent call last):
File "/home/charl/anaconda3/envs/nerfbridge/bin/ns-train", line 5, in
from nerfstudio.scripts.train import entrypoint
File "/home/charl/anaconda3/envs/nerfbridge/lib/python3.10/site-packages/nerfstudio/scripts/train.py", line 62, in
from nerfstudio.configs.method_configs import AnnotatedBaseConfigUnion
File "/home/charl/anaconda3/envs/nerfbridge/lib/python3.10/site-packages/nerfstudio/configs/method_configs.py", line 50, in
from nerfstudio.engine.trainer import TrainerConfig
File "/home/charl/anaconda3/envs/nerfbridge/lib/python3.10/site-packages/nerfstudio/engine/trainer.py", line 45, in
from nerfstudio.viewer.viewer import Viewer as ViewerState
File "/home/charl/anaconda3/envs/nerfbridge/lib/python3.10/site-packages/nerfstudio/viewer/viewer.py", line 35, in
from nerfstudio.models.splatfacto import SplatfactoModel
File "/home/charl/anaconda3/envs/nerfbridge/lib/python3.10/site-packages/nerfstudio/models/splatfacto.py", line 28, in
from gsplat._torch_impl import quat_to_rotmat
ModuleNotFoundError: No module named 'gsplat._torch_impl'