jun-cen / segmentanyrgbd

Segment Any RGBD

Hey there! 👋 I'm Jun CEN.

👨🏻‍💻 About Me

• 🎓   I'm currently a PhD candidate in Robotics Institute, HKUST.
• 📕   I'm working on open-world problem (open-set recognition + incremental learning) in computer vision, including 2D/3D semantic segmentation, 3D object detection, and action recognition et al.
• 🌱   Enthusiast in distributed learning and blockchain.
• 👯   I’m looking to collaborate on any related topics about open-world problem.
• 📫   How to reach me: jcenaa at connect dot ust dot hk.
• 💻   My personal webset: cen-jun.com

⭐️ From devSouvik

segmentanyrgbd's Issues

修改深度权重问题

您好,很抱歉打扰您了，首先感谢您杰出的工作，其次，我想咨询一下关于深度图的训练权重问题，项目中是否能够更改训练时深度图的权重？我该怎么进行修改？

Run on own data?

Hi, I am trying to run this on the KITTI dataset, and when I load the image I get 'error' displayed in all the windows.

How can I run this repo on my own data?

Do i just need color and depth images?

Thanks!

Suggestion - Integrate MobileSAM into the pipeline for lightweight and faster inference

Reference: https://github.com/ChaoningZhang/MobileSAM

Our project performs on par with the original SAM and keeps exactly the same pipeline as the original SAM except for a change on the image encode, therefore, it is easy to Integrate into any project.

MobileSAM is around 60 times smaller and around 50 times faster than original SAM, and it is around 7 times smaller and around 5 times faster than the concurrent FastSAM. The comparison of the whole pipeline is summarzed as follows:

Best Wishes,

Qiao

The error of running ui.py

The error output:

Traceback (most recent call last):
File "ui.py", line 12, in
import gradio as gr
File "xxx/anaconda3/envs/segment-anything/lib/python3.8/site-packages/gradio/init.py", line 3, in
import gradio.components as components
File "xxx/envs/segment-anything/lib/python3.8/site-packages/gradio/components.py", line 55, in
from gradio import processing_utils, utils
File "xxx/envs/segment-anything/lib/python3.8/site-packages/gradio/utils.py", line 497, in
class AsyncRequest:
File "xxx/envs/segment-anything/lib/python3.8/site-packages/gradio/utils.py", line 516, in AsyncRequest
client = httpx.AsyncClient()
File "xxx/envs/segment-anything/lib/python3.8/site-packages/httpx/_client.py", line 1395, in init
proxy_map = self._get_proxy_map(proxies, allow_env_proxies)
File "xxx/envs/segment-anything/lib/python3.8/site-packages/httpx/_client.py", line 216, in _get_proxy_map
return {
File "xxx/envs/segment-anything/lib/python3.8/site-packages/httpx/_client.py", line 217, in
key: None if url is None else Proxy(url=url)
File "xxx/envs/segment-anything/lib/python3.8/site-packages/httpx/_config.py", line 336, in init
raise ValueError(f"Unknown scheme for proxy URL {url!r}")
ValueError: Unknown scheme for proxy URL URL('socks://127.0.0.1:1089/')

Could you please help me solve this problem?

UserWarning: /root/.cache/clip/ViT-L-14.pt exists, but the SHA256 checksum does not match; re-downloading the file

WARNING [10/01 22:33:04 fvcore.common.config]: Loading config configs/ovseg_swinB_vitL_demo.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.
/root/miniconda3/envs/ovseg/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/root/autodl-tmp/Seg-RGBD/SegmentAnyRGBD/third_party/CLIP/clip/clip.py:60: UserWarning: /root/.cache/clip/ViT-L-14.pt exists, but the SHA256 checksum does not match; re-downloading the file
warnings.warn(
3%|▉ | 23.8M/933M [04:06<3:38:21, 69.4kiB/s]^CKeyboard interruption in main thread... closing server.

Export results?

Hi, and thanks for making this code available!

Is it possible to export the 3D results to be imported into another programme such as After Effects?

And can it run on high resolution (6k) images?

ui.py error on colab

Thanks for great work!
I would like to ui.py on google colab.
but, when I ran it, I met the following error on GradIO

Unexpected token '<', "<html> <h"... is not valid JSON

On Google colab console

['/tmp/c0c30d2f119e2c9f55e079cff868436b0489eb5d/tmpawyn8wax.png']
  0% 0/1 [00:00<?, ?it/s]Using SAM to generate segments for the RGB image
Using SAM to generate segments for the Depth map
  0% 0/1 [01:06<?, ?it/s]

SARGBD cannot generate RGB image for semantic segmentation...

Please tell me how to solve this problem.

Problem with ui.py

Hello, thanks for u great job.
When i run ui.py, in my own GPU , i meet a problem like loading the yaml
WARNING [06/14 10:03:14 fvcore.common.config]: Loading config configs/ovseg_swinB_vitL_demo.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.
And the bug like

Traceback (most recent call last):
  File "/home/sport/.local/lib/python3.8/site-packages/gradio/routes.py", line 422, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/sport/.local/lib/python3.8/site-packages/gradio/blocks.py", line 1323, in process_api
    result = await self.call_function(
  File "/home/sport/.local/lib/python3.8/site-packages/gradio/blocks.py", line 1051, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/sport/.local/lib/python3.8/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/sport/.local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/home/sport/.local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "ui.py", line 93, in greet_sailvos3d
    demo = VisualizationDemo(cfg)
  File "/home/sport/3d_seg/SegmentAnyRGBD/open_vocab_seg/utils/predictor.py", line 151, in __init__
    self.predictor = OVSegPredictor(cfg)
  File "/home/sport/3d_seg/SegmentAnyRGBD/open_vocab_seg/utils/predictor.py", line 27, in __init__
    super().__init__(cfg)
  File "/home/sport/anaconda3/envs/ovseg/lib/python3.8/site-packages/detectron2/engine/defaults.py", line 288, in __init__
    checkpointer.load(cfg.MODEL.WEIGHTS)
  File "/home/sport/anaconda3/envs/ovseg/lib/python3.8/site-packages/detectron2/checkpoint/detection_checkpoint.py", line 52, in load
    ret = super().load(path, *args, **kwargs)
  File "/home/sport/anaconda3/envs/ovseg/lib/python3.8/site-packages/fvcore/common/checkpoint.py", line 143, in load
    checkpoint = self._load_file(path)
  File "/home/sport/anaconda3/envs/ovseg/lib/python3.8/site-packages/detectron2/checkpoint/detection_checkpoint.py", line 88, in _load_file
    loaded = super()._load_file(filename)  # load native pth checkpoint
  File "/home/sport/anaconda3/envs/ovseg/lib/python3.8/site-packages/fvcore/common/checkpoint.py", line 240, in _load_file
    return torch.load(f, map_location=torch.device("cpu"))
  File "/home/sport/anaconda3/envs/ovseg/lib/python3.8/site-packages/torch/serialization.py", line 608, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/sport/anaconda3/envs/ovseg/lib/python3.8/site-packages/torch/serialization.py", line 777, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
EOFError: Ran out of input

Can you help me about this issue?

ERROR: Exception in ASGI application when running ui .py

Thanks for u great work
when I run ui.py, I meet a problem:

I can run demo_scannet.py, so I don't know where the problem is. Can you help me

Online demo doesn't work.

The hugging face demo on MMLAB-NTU is an empty page.
The demo on jcenaa got a 'Runtime Error'

Is that the problem of my network connection or the deployment?