kdexd / probnmn-clevr Goto Github PK

I followed all the steps as mentioned in the document but While running the command for Phase: module_training.

python scripts/train.py
--config-yml configs/module_training.yml
--phase module_training
--config-override CHECKPOINTS.QUESTION_CODING checkpoints/question_coding_ours/question_coding_best.pth
--gpu-ids 0 1 2 3
--serialization-dir checkpoints/module_training

I'm getting this error: Input type (torch.cuda.DoubleTensor) and weight type (torch.cuda.FloatTensor) should be the same

on line: output_dict = self._nmn(batch["image"], pg_output_dict["predictions"], batch["answer"]) in module_training_trainer.py

When I change the type of batch["image"] to batch["image"].float() i got error: TypeError: zip argument #1 must support iteration

I tried different ways to convert batch["image"] of type torch.cuda.FloatTensor but every time i got it is not iterable. I also tried to change the get_item from dataset to return an image of type float but still, it is not working.

I am pasting the completing error for clear reference.

Batch['image'] type : torch.cuda.DoubleTensor

> Traceback (most recent call last):
>   File "scripts/train.py", line 138, in <module>
>     trainer.step(iteration)
>   File "/data/vivek/neural-symbolic/probnmn-clevr/probnmn/trainers/_trainer.py", line 147, in step
>     output_dict = self._do_iteration(batch)
>   File "/data/vivek/neural-symbolic/probnmn-clevr/probnmn/trainers/module_training_trainer.py", line 96, in _do_iteration
>     output_dict = self._nmn(batch["image"], pg_output_dict["predictions"], batch["answer"])
>   File "/data/vivek/anaconda3/envs/probnmn/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/data/vivek/anaconda3/envs/probnmn/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 143, in forward
>     outputs = self.parallel_apply(replicas, inputs, kwargs)
>   File "/data/vivek/anaconda3/envs/probnmn/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 153, in parallel_apply
>     return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
>   File "/data/vivek/anaconda3/envs/probnmn/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
>     raise output
>   File "/data/vivek/anaconda3/envs/probnmn/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
>     output = module(*input, **kwargs)
>   File "/data/vivek/anaconda3/envs/probnmn/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/data/vivek/neural-symbolic/probnmn-clevr/probnmn/models/nmn.py", line 183, in forward
>     feat_input_volume = self.stem(features)
>   File "/data/vivek/anaconda3/envs/probnmn/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/data/vivek/anaconda3/envs/probnmn/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
>     input = module(input)
>   File "/data/vivek/anaconda3/envs/probnmn/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/data/vivek/anaconda3/envs/probnmn/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 320, in forward
>     self.padding, self.dilation, self.groups)
> RuntimeError: Input type (torch.cuda.DoubleTensor) and weight type (torch.cuda.FloatTensor) should be the same
> 
> 

Hi，I'm getting this error TypeError: Can't broadcast (128, 200704) -> (128, 1024, 14, 14) when running extract_features.py. The error is at line 133 output_h5["features"][counter : counter + len(batch)] = output_features.cpu().numpy()

Hi, in the second training phase, it's fine to use only one gpu, but when using multiple gpus it would run into problems. The layout is as follows. How can I handle this?

2020-11-16 02:03:30.277 | INFO | probnmn.utils.checkpointing:load:156 - Checkpointables not found in file: [] 2020-11-16 02:03:30.335 | INFO | probnmn.utils.checkpointing:load:131 - Loading checkpoint from checkpoints/question_coding_ours/checkpoint_best.pth 2020-11-16 02:03:30.367 | INFO | probnmn.utils.checkpointing:load:153 - optimizer not found in checkpointables. 2020-11-16 02:03:30.368 | INFO | probnmn.utils.checkpointing:load:153 - scheduler not found in checkpointables. 2020-11-16 02:03:30.368 | INFO | probnmn.utils.checkpointing:load:141 - Loading program_generator from checkpoints/question_coding_ours/checkpoint_best.pth 2020-11-16 02:03:30.371 | INFO | probnmn.utils.checkpointing:load:153 - question_reconstructor not found in checkpointables. 2020-11-16 02:03:30.371 | INFO | probnmn.utils.checkpointing:load:156 - Checkpointables not found in file: [] training: 0%| | 0/80000 [00:11<?, ?it/s] Traceback (most recent call last): File "scripts/train.py", line 136, in <module> trainer.step(iteration) File "/localscratch/zelin/batch_soft_reason/baselines/probnmn-clevr/probnmn/trainers/_trainer.py", line 148, in step output_dict = self._do_iteration(batch) File "/localscratch/zelin/batch_soft_reason/baselines/probnmn-clevr/probnmn/trainers/module_training_trainer.py", line 90, in _do_iteration output_dict = self._nmn(batch["image"], pg_output_dict["predictions"], batch["answer"]) File "/localscratch/ksamel3/anaconda3/envs/soft_reason/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/localscratch/ksamel3/anaconda3/envs/soft_reason/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 153, in forward return self.gather(outputs, self.output_device) File "/localscratch/ksamel3/anaconda3/envs/soft_reason/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 165, in gather return gather(outputs, output_device, dim=self.dim) File "/localscratch/ksamel3/anaconda3/envs/soft_reason/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather res = gather_map(outputs) File "/localscratch/ksamel3/anaconda3/envs/soft_reason/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 62, in gather_map for k in out)) File "/localscratch/ksamel3/anaconda3/envs/soft_reason/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 62, in <genexpr> for k in out)) File "/localscratch/ksamel3/anaconda3/envs/soft_reason/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 62, in gather_map for k in out)) File "/localscratch/ksamel3/anaconda3/envs/soft_reason/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 62, in <genexpr> for k in out)) File "/localscratch/ksamel3/anaconda3/envs/soft_reason/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map return type(out)(map(gather_map, zip(*outputs))) TypeError: zip argument #1 must support iteration