cangumeli / roca Goto Github PK

Thanks for your open source wonderful work, I keep getting an error reported when I try to run demo.py and I can't fix it!

What should I do to fix this?

Hello @cangumeli,
Sorry to bother you again, but I am getting following error while training the ROCA model.
Appreciate any help.

[09/07 15:47:04 d2.evaluation.evaluator]: Inference done 5388/5436. 0.0699 s / img. ETA=0:00:03
[09/07 15:47:07 d2.evaluation.evaluator]: Total inference time: 0:07:07.043241 (0.078631 s / img per device, on 1 devices)
[09/07 15:47:07 d2.evaluation.evaluator]: Total inference pure compute time: 0:06:19 (0.069837 s / img per device, on 1 devices)

Starting per-frame evaluation
Frame: 0/5436
Frame: 500/5436
Frame: 1000/5436
Frame: 1500/5436
Frame: 2000/5436
Frame: 2500/5436
Frame: 3000/5436
Frame: 3500/5436
Frame: 4000/5436
Frame: 4500/5436
Frame: 5000/5436
Traceback (most recent call last):
  File "C:\Users\Anaconda3\envs\roca\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "D:\research\code\roca\network\roca\engine\trainer.py", line 207, in cad_context
    yield
  File "D:\research\code\roca\network\roca\engine\trainer.py", line 180, in test
    results = super().test(cfg, model, evaluators)
  File "d:\research\code\detectron2-0.3\detectron2\engine\defaults.py", line 534, in test
    results_i = inference_on_dataset(model, data_loader, evaluator)
  File "d:\research\code\detectron2-0.3\detectron2\evaluation\evaluator.py", line 176, in inference_on_dataset
    results = evaluator.evaluate()
  File "d:\research\code\detectron2-0.3\detectron2\evaluation\evaluator.py", line 91, in evaluate
    result = evaluator.evaluate()
  File "D:\research\code\roca\network\roca\evaluation\per_frame_evaluation.py", line 81, in evaluate
    compute_ap(scores, labels, npos).item() * 100,
AttributeError: 'float' object has no attribute 'item'

Hi! I tried the demo by following the instructions in README and it works successfully, except that command should be

$ python demo.py --model_path $MODEL_DIR/model_best.pth --data_dir $DATA_DIR/Dataset --config_path $MODEL_DIR/config.yaml

where --model_dir is changed to --model_path and --config_dir is changed to --config_path.

Please correct me if I am wrong. Thanks!

Hello @cangumeli ,

Thank you for sharing the code.

I obtained the dataset based on the code and sh run.sh
I am getting following error while rendering the ROCA data.

Appreciate any help.
Process Process-1:
Traceback (most recent call last):
File "/home/zhou/anaconda3/envs/roca/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/zhou/anaconda3/envs/roca/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "render.py", line 296, in worker
scene_alignments, scene_images = render_scene(
File "render.py", line 181, in render_scene
rasterizer.add_mesh(mesh, idx, camera_to_model)
File "/home/zhou/下载/ROCA-main/renderer/utils/raster.py", line 88, in add_mesh
self.add_model(
File "/home/zhou/下载/ROCA-main/renderer/utils/raster.py", line 123, in add_model
super().add_model(triangles, vertices, idx, to_noc, normals)
TypeError: add_model(): incompatible function arguments. The following argument types are supported:
1. (self: scan2cad_rasterizer.Rasterizer, arg0: numpy.ndarray[numpy.uint64], arg1: numpy.ndarray[numpy.float32], arg2: int) -> None
2. (self: scan2cad_rasterizer.Rasterizer, arg0: numpy.ndarray[numpy.uint64], arg1: numpy.ndarray[numpy.float32], arg2: int, arg3: numpy.ndarray[numpy.float32]) -> None

skipping vacant point sample ('02933112', '37e5fcf70007bc26788f926f4d51e733')...

WARNING - 2022-05-30 21:59:28,861 - obj - unable to load materials from: model_normalized.mtl

WARNING - 2022-05-30 21:59:28,868 - obj - specified material (material_52_24) not loaded!

Hi dear author,

Thanks for your wonderful job and making the source code public so quickly. I think the idea in the paper of learning a joint embedding between depth object and cad model is clever and easy to understand. But it seems that you are not explaining in paper how to find postive and negative exemples in joint embedding learning. I would be appreciate if you can explain more details about it.

Thanks a lot : )

Can I find out where the CAD model is saved

I followed all your steps for installation and it is successfully finished but when I try to run your demo.py example I get error:
"anaconda3/envs/roca/lib/python3.8/site-packages/torch/lib/../../../../libcublas.so.11: undefined symbol: free_gemm_select, version libcublasLt.so.11"
According to google, people who had this error suggested to change version of pytorch or cudatoolkit but when I change it the rest of code fails and I get other errors.
What could it be to cause this error and how to solve it?

Hello @cangumeli ,

Thank you for sharing the code.

I have a question about the code written in demo.py. On line 25, why do you use scene names from ScanNet dataset?

I want to try the demo code for images taken by phone camera. Could you please tell what steps I need to follow for preprocessing? What scene names I need to choose to write on line 25 of demo.py to work on such not-in-dataset images?

Thank you,
Supriya

Hello, your project is fantastic. It was very helpful to me, but, when I run "python demo.py", the following issues occurred during the operation process:

unable to load materials from: model_normalized.mtl
specified material (material_52_24) not loaded!
specified material (material_37_24) not loaded!
specified material (material_27_24) not loaded!
specified material (material_47_24) not loaded!
specified material (material_26_24) not loaded!
specified material (material_24_24) not loaded!
specified material (material_15_24) not loaded!
specified material (material_6_24) not loaded!
specified material (material_1_24) not loaded!

Hi,

Thanks for this great work, it looks very promising and exciting!
I am doing some tests. I did not have major issues with installation and the demo runs. Congrats!

I want to adapt the code to get a simple CLI with an input image and its intrinsics. It would be great if such a demo was part of the codebase IMHO.

To this end, could you kindly explain, in demo.py, what is the "scene" argument here ?

   for name, scene in zip(
        ('3m', 'sofa', 'lab', 'desk'),
        ('scene0474_02', 'scene0207_00', 'scene0378_02', 'scene0474_02')
    ):

Thanks
Thibault

Hi，thank for your excellent work on single image secen understanding.
I found a small bug in line 165 of network/roca/modeling/retrieval_head/retrieval_ops.py.

 except ValueError:
        assert len(feats) == 0 #feats.numel() == 0 is invalid, since feats is a List.

I tried to train the model from scratch using both the provided dataset and processed data following the instructions. But I met the following error.

`[06/13 16:27:47 d2.data.datasets.coco]: Loaded 5436 images in COCO format from /workspace/ROCA/dataset/Data/Dataset/scan2cad_instances_val.json
[06/13 16:27:47 d2.data.common]: Serializing 5436 elements to byte tensors and concatenating them all ...
[06/13 16:27:47 d2.data.common]: Serialized dataset takes 11.65 MiB
[06/13 16:27:54 d2.evaluation.evaluator]: Start inference on 5436 images
[06/13 16:27:55 d2.evaluation.evaluator]: Inference done 11/5436. 0.0710 s / img. ETA=0:07:35
[06/13 16:28:00 d2.evaluation.evaluator]: Inference done 72/5436. 0.0697 s / img. ETA=0:07:21
[06/13 16:28:05 d2.evaluation.evaluator]: Inference done 134/5436. 0.0693 s / img. ETA=0:07:13
[06/13 16:28:10 d2.evaluation.evaluator]: Inference done 199/5436. 0.0683 s / img. ETA=0:07:01
[06/13 16:28:15 d2.evaluation.evaluator]: Inference done 263/5436. 0.0681 s / img. ETA=0:06:54
[06/13 16:28:20 d2.evaluation.evaluator]: Inference done 326/5436. 0.0681 s / img. ETA=0:06:50
[06/13 16:28:25 d2.evaluation.evaluator]: Inference done 390/5436. 0.0679 s / img. ETA=0:06:43
[06/13 16:28:30 d2.evaluation.evaluator]: Inference done 446/5436. 0.0691 s / img. ETA=0:06:46
ERROR [06/13 16:28:34 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
File "/workspace/ROCA/network/roca/modeling/retrieval_head/retrieval_ops.py", line 162, in voxelize_nocs
volumes = add_pointclouds_to_volumes(points, volumes)
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/pytorch3d/ops/points_to_volumes.py", line 275, in add_pointclouds_to_volumes
raise ValueError("'pointclouds' have to have their 'features' defined.")
ValueError: 'pointclouds' have to have their 'features' defined.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 135, in train
self.after_step()
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 165, in after_step
h.after_step()
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/hooks.py", line 353, in after_step
self._do_eval()
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/hooks.py", line 328, in _do_eval
results = self._func()
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/defaults.py", line 366, in test_and_save_results
self._last_eval_results = self.test(self.cfg, self.model)
File "/workspace/ROCA/network/roca/engine/trainer.py", line 180, in test
results = super().test(cfg, model, evaluators)
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/defaults.py", line 534, in test
results_i = inference_on_dataset(model, data_loader, evaluator)
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/evaluation/evaluator.py", line 141, in inference_on_dataset
outputs = model(inputs)
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/workspace/ROCA/network/roca/modeling/meta_arch/meta_arch.py", line 40, in forward
return self.inference(batched_inputs)
File "/workspace/ROCA/network/roca/modeling/meta_arch/meta_arch.py", line 124, in inference
results, extra_outputs = self.roi_heads(
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/workspace/ROCA/network/roca/modeling/roi_heads/roi_heads.py", line 132, in forward
pred_instances, alignment_outputs = self._forward_alignment(
File "/workspace/ROCA/network/roca/modeling/roi_heads/roi_heads.py", line 180, in _forward_alignment
return self._forward_alignment_inference(
File "/workspace/ROCA/network/roca/modeling/roi_heads/roi_heads.py", line 282, in _forward_alignment_inference
predictions, extra_outputs = self.alignment_head(
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/workspace/ROCA/network/roca/modeling/alignment_head/alignment_head.py", line 137, in forward
return self.forward_inference(*args, **kwargs)
File "/workspace/ROCA/network/roca/modeling/alignment_head/alignment_head.py", line 333, in forward_inference
predictions, extra_outputs = self._forward_retrieval_inference(
File "/workspace/ROCA/network/roca/modeling/alignment_head/alignment_head.py", line 803, in _forward_retrieval_inference
cad_ids, pred_indices = self.retrieval_head(
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/workspace/ROCA/network/roca/modeling/retrieval_head/retrieval_head.py", line 201, in forward
return self._embedding_lookup(
File "/workspace/ROCA/network/roca/modeling/retrieval_head/retrieval_head.py", line 353, in _embedding_lookup
noc_embeds = self.embed_nocs(shape_code, noc_points, pred_masks)
File "/workspace/ROCA/network/roca/modeling/retrieval_head/retrieval_ops.py", line 165, in voxelize_nocs
assert len(feats) == 0
AssertionError
[06/13 16:28:34 d2.engine.hooks]: Overall training speed: 7497 iterations in 1:56:25 (0.9318 s / it)
[06/13 16:28:34 d2.engine.hooks]: Total training time: 2:18:02 (0:21:37 on hooks)
[06/13 16:28:34 d2.utils.events]: eta: 18:51:20 iter: 7499 total_loss: 5.823 loss_cls: 0.3354 loss_box_reg: 0.482
3 loss_image_depth: 0.3101 loss_mask: 0.3888 loss_mask_iou: 0.3305 loss_roi_depth: 0.2764 loss_mean_depth: 0.2694
loss_scale: 0.3106 loss_depth_min: 0.04275 loss_depth_max: 0.05285 loss_trans: 0.4288 loss_noc: 0.9828 loss_pro
c: 0.5804 loss_trans_proc: 0.3947 loss_noc_comp: 0.1564 loss_triplet: 0.2907 loss_rpn_cls: 0.03125 loss_rpn_loc:
0.01378 time: 0.9317 data_time: 0.0259 lr: 0.001 max_mem: 4627M
Traceback (most recent call last):
File "/workspace/ROCA/network/roca/modeling/retrieval_head/retrieval_ops.py", line 162, in voxelize_nocs
volumes = add_pointclouds_to_volumes(points, volumes)
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/pytorch3d/ops/points_to_volumes.py", line 275, in add_pointclouds_to_volumes
raise ValueError("'pointclouds' have to have their 'features' defined.")
ValueError: 'pointclouds' have to have their 'features' defined.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 175, in
main(parse_args())
File "main.py", line 171, in main
train_or_eval(args, cfg)
File "main.py", line 164, in train_or_eval
trainer.train()
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/defaults.py", line 413, in train
super().train(self.start_iter, self.max_iter)
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 135, in train
self.after_step()
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/train_loop.py", line 165, in after_step
h.after_step()
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/hooks.py", line 353, in after_step
self._do_eval()
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/hooks.py", line 328, in _do_eval
results = self._func()
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/defaults.py", line 366, in test_and_save_results
self._last_eval_results = self.test(self.cfg, self.model)
File "/workspace/ROCA/network/roca/engine/trainer.py", line 180, in test
results = super().test(cfg, model, evaluators)
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/engine/defaults.py", line 534, in test
results_i = inference_on_dataset(model, data_loader, evaluator)
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/detectron2/evaluation/evaluator.py", line 141, in inference_on_dataset
outputs = model(inputs)
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/workspace/ROCA/network/roca/modeling/meta_arch/meta_arch.py", line 40, in forward
return self.inference(batched_inputs)
File "/workspace/ROCA/network/roca/modeling/meta_arch/meta_arch.py", line 124, in inference
results, extra_outputs = self.roi_heads(
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/workspace/ROCA/network/roca/modeling/roi_heads/roi_heads.py", line 132, in forward
pred_instances, alignment_outputs = self._forward_alignment(
File "/workspace/ROCA/network/roca/modeling/roi_heads/roi_heads.py", line 180, in _forward_alignment
return self._forward_alignment_inference(
File "/workspace/ROCA/network/roca/modeling/roi_heads/roi_heads.py", line 282, in _forward_alignment_inference
predictions, extra_outputs = self.alignment_head(
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/workspace/ROCA/network/roca/modeling/alignment_head/alignment_head.py", line 137, in forward
return self.forward_inference(*args, **kwargs)
File "/workspace/ROCA/network/roca/modeling/alignment_head/alignment_head.py", line 333, in forward_inference
predictions, extra_outputs = self._forward_retrieval_inference(
File "/workspace/ROCA/network/roca/modeling/alignment_head/alignment_head.py", line 803, in _forward_retrieval_inference
cad_ids, pred_indices = self.retrieval_head(
File "/root/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/workspace/ROCA/network/roca/modeling/retrieval_head/retrieval_head.py", line 201, in forward
return self._embedding_lookup(
File "/workspace/ROCA/network/roca/modeling/retrieval_head/retrieval_head.py", line 353, in _embedding_lookup
noc_embeds = self.embed_nocs(shape_code, noc_points, pred_masks)
File "/workspace/ROCA/network/roca/modeling/retrieval_head/retrieval_head.py", line 226, in embed_nocs
noc_points = voxelize_nocs(grid_to_point_list(noc_points, mask))
File "/workspace/ROCA/network/roca/modeling/retrieval_head/retrieval_ops.py", line 165, in voxelize_nocs
assert len(feats) == 0
AssertionError

`

I am using pytorch3d 0.6.2, pytorch 1.7.0

in file predictor.py :
i can't import these as follows:
from roca.config import roca_config
from roca.data import CADCatalog
from roca.data.constants import CAD_TAXONOMY, COLOR_BY_CLASS
from roca.data.datasets import register_scan2cad
from roca.structures import Intrinsics
from roca.utils.alignment_errors import translation_diff
from roca.utils.linalg import make_M_from_tqs
the errors say: Unresolved reference "roca"
it 's a basic issues however, i can't solve .

Hello @cangumeli,
Sorry to bother you, but I am getting the following error while running the demo.py with model_best.pth
Appreciate any help!

File "/home/aston/Desktop/python/ROCA-main/network/roca/engine/predictor.py", line 44, in __init__
    model.load_state_dict(backup['model'])
  File "/home/aston/anaconda3/envs/roca/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1604, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ROCA:
	Unexpected key(s) in state_dict: "pixel_mean", "pixel_std", "proposal_generator.anchor_generator.cell_anchors.0", "proposal_generator.anchor_generator.cell_anchors.1", "proposal_generator.anchor_generator.cell_anchors.2", "proposal_generator.anchor_generator.cell_anchors.3", "proposal_generator.anchor_generator.cell_anchors.4". 

Process finished with exit code 1