chungmin99 / garfield Goto Github PK

When exporting cropped Gaussian Splats, it requires a camera path. Is this camera from colmap or JSON from data_parser from the outputs directory?

Hi.
I installed the latest nerfstudio in conda environment, does that mean I could also install Garfield inside conda nerfstudio env?

I tried to run conda install -c rapidsai -c conda-forge -c nvidia cuml inside conda nerfstudio env but it doesn't work properly. Has anyone encountered the same issue? Here is the error log for your reference:

(base) C:\Users\lzlal>conda activate nerfstudio

(nerfstudio) C:\Users\lzlal>cd nerfstudio

(nerfstudio) C:\Users\lzlal\nerfstudio>conda install -c rapidsai -c conda-forge -c nvidia cuml
Channels:
 - rapidsai
 - conda-forge
 - nvidia
 - defaults
 - pytorch
 - nvidia/label/cuda-11.8.0
Platform: win-64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - cuml

Current channels:

  - https://conda.anaconda.org/rapidsai
  - https://conda.anaconda.org/conda-forge
  - https://conda.anaconda.org/nvidia
  - defaults
  - https://conda.anaconda.org/pytorch
  - https://conda.anaconda.org/nvidia/label/cuda-11.8.0/win-64

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.



(nerfstudio) C:\Users\lzlal\nerfstudio>

Is it possible to export point clouds with semantic labels or instance labels after instance segmentation?

Thanks for this exciting work! I am wondering if you could provide the LERF Dataset with segmentation ground truth you used for evaluation?

Thank you!

Hello @chungmin99

is there any description on how to run the complete Garfield Gaussian Process?

The read me only mention ns train with gauss but there is no info on how to prepare the data, how to create the required checkpoints, segmentation etc.

I would appreciate if there is a guide on how to use it.

Thanks for your patience. I appreciate your work. :-)

EDIT: I understand now, I have to run the garfield process first before I even can run the gauss variant.

Hi, Thank you for sharing your amazing project :)

1. Can I check the training results using the viewer after training is finished? If I finish training and quit nerf-studio, is there a way to return to the last state of training?
2. It looks like mesh extraction was performed in the asset extraction of the project page. How can I extract the mesh?

Hello

Is there any way I can make a render (e.g from a camera path) with the instance colours ?

I'm guessing that since the instance colours change with every camera viewpoint change, the answer is not- but thought it worth asking the question

Andrew

Hi, I want to use garfield locally without using NeRF studio.

It aimed at obtaining the final segmentation result local (like assume to use on server).

Can we do this by this code?

Hi, dear author, thanks for your amazing work. I have run the nerfstudio project successfully before. But this time when I installed garfield and tried to run 'ns-train garfield', I was informed 'tcnn is not defined'. I tried reinstalling or upgrading torch, python, cuda toolkit and tiny-cuda-nn but it didnt help, and now i cant even run 'ns-train nerfacto'. Any idea how to solve this problem? I'd very much appreciate it.

Hi, Thanks for your excellent work!
When I trained on my own dataset, I met the inf loss problem. In several steps starting from 2000 steps, instance_loss_4 and total_loss are inf. After training for a few steps, Loss returned to normal. Is this a normal phenomenon?
Looking forward to your reply, Thank you!

Hi Authors,
I'm getting an index error while calculating 3D masks. (ns-train garfield)
Please let me know what's wrong.
Here's the full error Traceback:

I try to run the command ns-process-data images --data=/Users/pascal/Desktop/tumorimages --output-dir=/Users/pascal/Desktop/processed_tumor, nerfstudio is installed. and this is the output is No usable images in the data folder. process_data_utils.py:411. The images are in the same directory as the code and are just some png files.

Hello,

I am attempting to perform gaussian splat segmentation. When I press select any method for clustering, I just get a single cluster for the whole gaussian splat (single color). What could the reason be?

Training was successfully performed with garfield, and then I used the checkpoint from it to train the garfield-gauss.

Using latest Nerfstudio 1.13.0 with gsplat 1.0.0.

Any input is appreciated. Thanks! @chungmin99

Hi,

Is there any suggestion for how to reduce gpu memory usage? I've been attempting to try this with custom data but I run out of gpu memory during the mask generation. I have an rtx a5000 (24 gb vram).

Thanks!

Hi, thanks for your excellent work! I wonder if is it possible to give us a data folder example to illustrate how to organize our data to run ns-train garfield --data xxxx , and wether we should pre-process our own data. Very thx for your help!

How to do asset extraction like this in the demo?
drop.webm

Hello, me and a group of people are using Garfield for an object recognition research project. This includes us uploading custom data into Garfield and utilizing the software to manipulate scenes. When training this data with an ADA6000 Nvidia GPU we see load times of 45-90 minutes. Is there any way we can cut this time down, while not sacrificing our quality too much?

While I have successfully run ns-train garfield --data /your/data/here and got the GARField's affinity field, the experiments fail when I try to run ns-train garfield-gauss --data /your/data/here --pipeline.garfield-ckpt outputs/your/data/garfield/.../config.yml.
Namely, I can use the Global clustering function, but when I click on an instance and use Crop to click, the following error occurs:

Exception in thread Thread-3:
Traceback (most recent call last):
File "/home/%/miniconda3/envs/garfield2/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/home/%/nerfstudio/nerfstudio/viewer/render_state_machine.py", line 218, in run
outputs = self._render_img(action.camera_state)
File "/home/%/nerfstudio/nerfstudio/viewer/render_state_machine.py", line 164, in _render_img
outputs = self.viewer.get_model().get_outputs_for_camera(camera, obb_box=obb)
File "/home/%/miniconda3/envs/garfield2/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/%/nerfstudio/nerfstudio/models/splatfacto.py", line 866, in get_outputs_for_camera
outs = self.get_outputs(camera.to(self.device))
File "/home/%/nerfstudio/nerfstudio/models/splatfacto.py", line 766, in get_outputs
depth_im = rasterize_gaussians( # type: ignore
File "/home/%/miniconda3/envs/garfield2/lib/python3.8/site-packages/gsplat/rasterize.py", line 67, in rasterize_gaussians
return _RasterizeGaussians.apply(
File "/home/%/miniconda3/envs/garfield2/lib/python3.8/site-packages/torch/autograd/function.py", line 539, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/%/miniconda3/envs/garfield2/lib/python3.8/site-packages/gsplat/rasterize.py", line 108, in forward
num_intersects, cum_tiles_hit = compute_cumulative_intersects(num_tiles_hit)
File "/home/%/miniconda3/envs/garfield2/lib/python3.8/site-packages/gsplat/utils.py", line 117, in compute_cumulative_intersects
num_intersects = cum_tiles_hit[-1].item()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Every time, the same error occurs. I encountered this while using CUDA 11.7 on an NVIDIA A40 GPU.

First of all, thank you for sharing your exciting work! The GARField paper and code are both very impressive and have great potential for improving the world around us.

In my attempts to follow along at home, I've run into a couple of mentions of the fully trained Garfield checkpoint- I see it is require for Gaussian Fields and a required field in the Data scripts, but I do not see any link to the fully trained Garfield checkpoint either here, or on the popular model repository Huggingface.

If the checkpoint is hidden somewhere in this repo, or there is a link that I have overlooked, please let me know. Otherwise, please share the checkpoint! I would be happy to help by going through the process of getting it up on Huggingface and pointing the documentation over, if that would be any help.

Hi- Thanks for this interesting work. What license is the code using? Is it Apache 2.0 like NerfStudio?

Currently, it's unclear what the dimensionality + type of the call function should be!

Need to define this before adding support for additional grouping types.

In particular:

• the resolution of the mask should equal resolution of the image
• the mask should be boolean (not 0/1s)
• the masks should be ordered in increasing 2D size

Thank you for your great work!
I trained a garfield by ns-train, and it was successful.
Then, I tried running the garfield-gauss, but it seems the ns-train does not have garfield-gauss. The checkpoint path is correct.
I had installed the main branch of nerfstudio, so I have no idea how to fix this😭

Thanks for this exciting work. I wonder if you could release pre-trained models on LERF dataset, or just the rendered segmentations?

The previous process in secceed for ''Running GARField''. But sth. went wrong when I use the command ''ns-train garfield-gauss --data /your/data/here --pipeline.garfield-ckpt outputs/y/.../config.yml''. Error is as following:

[5/5] c++ bindings.cuda.o backward.cuda.o forward.cuda.o ext.o -shared -L/home/moonlight/.conda/envs/nerfstudio/lib/python3.8/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/home/moonlight/.conda/envs/nerfstudio/lib64 -lcudart -o gsplat_cuda.so
FAILED: gsplat_cuda.so
c++ bindings.cuda.o backward.cuda.o forward.cuda.o ext.o -shared -L/home/moonlight/.conda/envs/nerfstudio/lib/python3.8/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/home/moonlight/.conda/envs/nerfstudio/lib64 -lcudart -o gsplat_cuda.so
/usr/bin/ld: cannot find -lcudart: No such file or directory
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

can you afford some help?thank you

Hi, thank you for the awesome work and the well-documented code. I am really impressed by the design and the amazing grouping quality.

I have a small question regarding the next_group function:

Here the same random number is used in line 274-278 for inverse CDF sampling and line 290-307 for scale densification. Then only a fraction of the scale range is sampled during training for a specific mask. For example, let's say one pixel has a mask CDF of [0.2, 0.5, 1.0]. The smallest mask with scale s0 will be selected when a random number p < 0.2 is generated. Then for training with this mask, only scale p*s0 < 0.2s0 will be used.

I wonder if I misunderstood something or if this is intended?

Dear Author:
after ''Running GARField''I use the command ''ns-train garfield-gauss --data my/data/ --pipeline.garfield-ckpt outputs/2015-3-522/.../config.yml''. Error is as following:

Optimizer config for 'xyz' 、‘features_dc’ ... is missing from the yml provided. Is there anything wrong?? The previous step
is successful, but quality is not good enough. Thanks for your reply!

Hello @chungmin99 @kerrj

It seems that the new Nerfstudio update breaks this version of Garfield. For example, gsplat has received an update that wraps some gsplat functions as legacy.

When I run Garfield train, it caches the images and then just says “killed”. Any suggestions? Thanks!!

EDIT: Found the issue, when running in WSL2 environment, there was not enough RAM to accomodate this. I had to enable swapfile for this,