knazeri / edge-informed-sisr

Edge-Informed Single Image Super-Resolution, ICCVW 2019 https://arxiv.org/abs/1909.05305

I'm Kamyar

I am a coder for life with expertise in C++, C#, Go, Python, JavaScript, TypeScript, and most recently with a focus on Cloud Architecture, Deep Learning in Computer Vision, and Generative Models.

edge-informed-sisr's Issues

Pretrained model

@knazeri, thank you for the project!
Could you provide a link to the pretrained model, please?

TypeError: torch.FloatTensor is not a Module subclass

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

start training...

Training epoch: 1
/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py:92: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
img = torch.from_numpy(np.array(pic, np.float32, copy=False))
/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py:92: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
img = torch.from_numpy(np.array(pic, np.float32, copy=False))
/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py:92: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
img = torch.from_numpy(np.array(pic, np.float32, copy=False))
/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py:92: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
img = torch.from_numpy(np.array(pic, np.float32, copy=False))
Traceback (most recent call last):
File "main.py", line 120, in
main(mode=1)
File "main.py", line 50, in main
model.train()
File "/content/drive/MyDrive/Colab Notebooks/SR/edge-informed-sisr-master/src/edge_match.py", line 133, in train
self.sr_model.backward(gen_loss, dis_loss)
File "/content/drive/MyDrive/Colab Notebooks/SR/edge-informed-sisr-master/src/models.py", line 283, in backward
gen_loss.backward()
File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/usr/local/lib/python3.6/dist-packages/torch/autograd/init.py", line 132, in backward
allow_unreachable=True) # allow_unreachable flag

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 512, 4, 4]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

---

somebody know this error and answer??

What's the training set?

invalid argument 0: Sizes of tensors must match except in dimension 1. Got 256 and 130 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71

Training epoch: 1
Traceback (most recent call last):
File "main.py", line 118, in
main()
File "main.py", line 48, in main
model.train()
File "/workspace/aniket/edge-informed-sisr/src/edge_match.py", line 120, in train
hr_images_pred, gen_loss, dis_loss, logs = self.sr_model.process(lr_images, hr_images, lr_edges, hr_edges_pred)
File "/workspace/aniket/edge-informed-sisr/src/models.py", line 210, in process
outputs = self(lr_images, hr_edges)
File "/workspace/aniket/edge-informed-sisr/eisr-env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/workspace/aniket/edge-informed-sisr/src/models.py", line 263, in forward
inputs = torch.cat((hr_images, hr_edges), dim=1)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 256 and 130 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71

BrokenPipeError: [Errno 32] Broken pipe

start training...

Training epoch: 1
Traceback (most recent call last):
File "", line 1, in
Traceback (most recent call last):
File "train.py", line 2, in
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\spawn.py", line 105, in spawn_main
main(mode=1)
File "D:\SR\edge-informed-sisr-master\main.py", line 50, in main
exitcode = _main(fd)
model.train()
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\spawn.py", line 114, in _main
File "D:\SR\edge-informed-sisr-master\src\edge_match.py", line 95, in train
prepare(preparation_data)
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\spawn.py", line 225, in prepare
for items in train_loader:
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\site-packages\torch\utils\data\dataloader.py", line 352, in iter
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="mp_main")
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\runpy.py", line 263, in run_path
return self._get_iterator()
pkg_name=pkg_name, script_name=fname)
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\site-packages\torch\utils\data\dataloader.py", line 294, in _get_iterator
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\runpy.py", line 85, in _run_code
return _MultiProcessingDataLoaderIter(self)
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\site-packages\torch\utils\data\dataloader.py", line 801, in init
exec(code, run_globals)
File "D:\SR\edge-informed-sisr-master\train.py", line 2, in
w.start()
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\process.py", line 112, in start
main(mode=1)
File "D:\SR\edge-informed-sisr-master\main.py", line 50, in main
model.train()
File "D:\SR\edge-informed-sisr-master\src\edge_match.py", line 95, in train
for items in train_loader:
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\site-packages\torch\utils\data\dataloader.py", line 352, in iter
return self._get_iterator()
self._popen = self._Popen(self)
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\context.py", line 223, in _Popen
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\site-packages\torch\utils\data\dataloader.py", line 294, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\site-packages\torch\utils\data\dataloader.py", line 801, in init
w.start()
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\context.py", line 322, in _Popen
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
return Popen(process_obj)
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\popen_spawn_win32.py", line 46, in init
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\popen_spawn_win32.py", line 89, in init
prep_data = spawn.get_preparation_data(process_obj._name)
reduction.dump(process_obj, to_child)
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\reduction.py", line 60, in dump
_check_not_importing_main()
File "C:\Users\SYM\AppData\Local\conda\conda\envs\deepcamp\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.    ForkingPickler(file, protocol).dump(obj)

BrokenPipeError: [Errno 32] Broken pipe

---

what is wrong with me? i cant find any solution

Missing details

Please could you add the pytorch version that you used?

Sizes of tensors must match except in dimension 0. Got 3 and 1 in dimension 1

Encounter training phase problem with Places2 256 size dataset , looking forward help to solve this.

Traceback (most recent call last):
File "train.py", line 2, in
main(mode=1)
File "/home/user/workspace/user/performance/edge-informed-sisr/main.py", line 50, in main
model.train()
File "/home/user/workspace/user/performance/edge-informed-sisr/src/edge_match.py", line 150, in train
self.sample()
File "/home/user/workspace/user/performance/edge-informed-sisr/src/edge_match.py", line 252, in sample
items = next(self.sample_iterator)
File "/home/user/workspace/user/performance/edge-informed-sisr/src/dataset.py", line 146, in create_iterator
for item in sample_loader:
File "/home/user/anaconda3/envs/user_workspace/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 346, in next
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/user/anaconda3/envs/user_workspace/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/home/user/anaconda3/envs/user_workspace/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 79, in default_collate
return [default_collate(samples) for samples in transposed]
File "/home/user/anaconda3/envs/user_workspace/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 79, in
return [default_collate(samples) for samples in transposed]
File "/home/user/anaconda3/envs/user_workspace/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
return torch.stack(batch, 0, out=out)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 3 and 1 in dimension 1 at /opt/conda/conda-bld/pytorch_1570710743984/work/aten/src/TH/generic/THTensor.cpp:689

Note : This problem encounter while evaluation dataset not training dataset

Can you share your model？thanks

TypeError: torch.FloatTensor is not a Module subclass

Traceback (most recent call last):
File "train.py", line 2, in
main(mode=1)
File "/root/edge-informed-sisr/main.py", line 42, in main
model = EdgeMatch(config)
File "/root/edge-informed-sisr/src/edge_match.py", line 27, in init
self.sr_model = SRModel(config).to(config.DEVICE)
File "/root/edge-informed-sisr/src/models.py", line 179, in init
self.add_module('scale_kernel', kernel_weight)
File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 179, in add_module
torch.typename(module)))
TypeError: torch.FloatTensor is not a Module subclass

how to use this cide?

how to use this cide?
could you write readme about it?

hr_image and hr_edge are in different sizes

The shape of hr_image is 3,256,256 however hr_edge is 1,70,70. I couldn't find anything relatable with 70. I made the change with cuda as in issue #1 . What might be the problem here? Where does 70 come. I use edge enhancer and sr model together in mode 3 and super resolution scale is 8.

Traceback (most recent call last):
File "main.py", line 120, in
main()
File "main.py", line 50, in main
model.train()
File "/scratch/users/nacar14/edge-informed-sisr/src/edge_match.py", line 120, in train
hr_images_pred, gen_loss, dis_loss, logs = self.sr_model.process(lr_images, hr_images, lr_edges, hr_edges_pred)
File "/scratch/users/nacar14/edge-informed-sisr/src/models.py", line 214, in process
outputs = self(lr_images, hr_edges)
File "/kuacc/users/nacar14/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/scratch/users/nacar14/edge-informed-sisr/src/models.py", line 269, in forward
inputs = torch.cat((hr_images, hr_edges), dim=1)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 256 and 70 in dimension 2 at /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/THC/g
eneric/THCTensorMath.cu:71