The free3d from lyndonzheng

先个大神点个赞，然后说说我的理解，特别和“终极理想立体视觉”模型之间的距离。

核心其实就一点，太严重的依赖于比较准的P(CameraPose)了。

我理解的，最终那个“终极理想立体视觉”，输入应该没有P，更不应该要求很精确的P。

要达到这种效果，应该是算法具有超级强大的先验，无论是diff-hash, memory-network, diffusion-parameters还是啥，已经有超级多超级强的先验，要重构的那些立体几乎都是存在的。给出的这张输入图片，就是对最接近的那些立体，在shape上做微调整，对应上texture。

即使算法过程中，需要P，那么也应该弱化P，不要那么精确，能够容忍一定程度的不准。
如果能够把P去掉，或者至少做到算法内部有一个estimation的模块，最好不是colmap，或者3d model上给定pose渲染，是个differentiable的pose-esti，还能做到比较好的效果，我觉得算法就更牛逼一些。

the details about pseudo-attention

Hi, congrats on Free3D, great work!
I saw your paper does not demonstrate the details about pseudo-3DAttention, such as the channel of the Linear Project Layer.

Pseudo 3D attention checkpoint

Hi! Remarkable work done here!
I'm working on a project similar to your Free3D and would really benefit from accessing the checkpoint of your pseudo 3D attention. Could you please share the checkpoint file or let me know how I can get it?
Thanks a lot!

Approximate time of code releasing

Hi, there. Remarkable work done here! Wondering is there an approximate time of code releasing?

Error when training on multiple GPU

Hi,
great work and thanks for sharing the code.
I am trying to run the training script. With the original command, it works fine.

However, when I try to add mutiple gpus, it does not work.
Here is the script, where I just added the gpus id:


python main.py \
    -t \
    --base configs/objaverse.yaml \
    --logdir /work/cxzheng/diff3d/test/logs \
    --name test \
    --gpus 0,1,2,3 \
    --scale_lr False \
    --num_nodes 1 \
    --seed 42 \
    --check_val_every_n_epoch 10 \
    --finetune_from /work/cxzheng/code/zero123_old/zero123/105000.ckpt

I get the following error


Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 75, in _wrap
    fn(i, *args)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py", line 139, in _wrapping_function
    results = function(*args, **kwargs)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1103, in _run
    results = self._run_stage()
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1182, in _run_stage
    self._run_train()
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1205, in _run_train
    self.fit_loop.run()
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 194, in run
    self.on_run_start(*args, **kwargs)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 160, in on_run_start
    _ = iter(data_fetcher)  # creates the iterator inside the fetcher
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 179, in __iter__
    self._apply_patch()
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 120, in _apply_patch
    apply_to_collections(self.loaders, self.loader_iters, (Iterator, DataLoader), _apply_patch_fn)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 156, in loader_iters
    return self.dataloader_iter.loader_iters
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/trainer/supporters.py", line 556, in loader_iters
    self._loader_iters = self.create_loader_iters(self.loaders)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/pytorch_lightning/trainer/supporters.py", line 596, in create_loader_iters
    return apply_to_collection(loaders, Iterable, iter, wrong_dtype=(Sequence, Mapping))
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/lightning_utilities/core/apply_func.py", line 52, in apply_to_collection
    return _apply_to_collection_slow(
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/lightning_utilities/core/apply_func.py", line 96, in _apply_to_collection_slow
    return function(data, *args, **kwargs)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 439, in __iter__
    return self._get_iterator()
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 387, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1040, in __init__
    w.start()
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/dense/miniconda3/envs/free3d/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'ObjaverseDataset.__init__.<locals>.<lambda>

As mentioned above, with one GPU the training start smoothly

Thanks for the help

lyndonzheng / free3d Goto Github PK

free3d's People

Contributors

Stargazers

Watchers

Forkers

free3d's Issues

先个大神点个赞，然后说说我的理解，特别和“终极理想立体视觉”模型之间的距离。

the details about pseudo-attention

Pseudo 3D attention checkpoint

Approximate time of code releasing

Error when training on multiple GPU

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

Jobs