zhouyanzhao / prm

Weakly Supervised Instance Segmentation using Class Peak Response, in CVPR 2018 (Spotlight)

Weakly Supervised Instance Segmentation using
Class Peak Response

[Home] [Project] [Paper] [Supp] [Poster] [Presentation]

PyTorch Implementation

The pytorch branch contains:

• the pytorch implementation of Peak Response Mapping (Stimulation and Backprop).
• the PASCAL-VOC demo (training, inference, and visualization).

Please follow the instruction below to install it and run the experiment demo.

Prerequisites

• System (tested on Ubuntu 14.04LTS and Win10)

• NVIDIA GPU + CUDA CuDNN (CPU mode is also supported but significantly slower)

• Python>=3.5

• PyTorch>=0.4

• Jupyter Notebook and ipywidgets (required by the demo):

  # enable the widgetsnbextension before you start the notebook server
  jupyter nbextension enable --py --sys-prefix widgetsnbextension

Installation

1. Install Nest, a flexible tool for building and sharing deep learning modules:

   I created Nest in the process of refactoring PRM's pytorch implementation. It aims at encouraging code reuse and ships with a bunch of useful features. PRM is now implemented as a set of Nest modules; thus you can easily install and use it as demonstrated below.

   $ pip install git+https://github.com/ZhouYanzhao/Nest.git

2. Install PRM via Nest's CLI tool:

   # note that data will be saved under your current path
   $ nest module install github@ZhouYanzhao/PRM:pytorch prm
   # verify the installation
   $ nest module list --filter prm
   # Output:
   #
   # 3 Nest modules found.
   # [0] prm.fc_resnet50 (1.0.0)
   # [1] prm.peak_response_mapping (1.0.0)
   # [2] prm.prm_visualize (1.0.0)

Run demo

1. Install Nest's build-in Pytorch modules:

   To increase reusability, I abstracted some features from the original code, such as network trainer, to build Nest's built-in pytorch module set.

   $ nest module install github@ZhouYanzhao/Nest:pytorch pytorch

2. Download the PASCAL-VOC2012 dataset:

   mkdir ./PRM/demo/datasets
   cd ./PRM/demo/datasets
   # download and extract data
   wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
   tar xvf VOCtrainval_11-May-2012.tar

3. Run the demo experiment via demo/main.ipynb

   

Citation

If you find the code useful for your research, please cite:

@INPROCEEDINGS{Zhou2018PRM,
    author = {Zhou, Yanzhao and Zhu, Yi and Ye, Qixiang and Qiu, Qiang and Jiao, Jianbin},
    title = {Weakly Supervised Instance Segmentation using Class Peak Response},
    booktitle = {CVPR},
    year = {2018}
}

prm's Issues

How to find the Json files in the data folder?

Hi, can anyone help me, how to get the json files in the data folder?

another eoor when executing run_tasks

from nest import modules, run_tasks works well for me,
while when executing, I got this error:

AttributeError: type object 'tqdm' has no attribute '_lock'

Is there anyone who ever met this error?

RuntimeError: CUDA error: out of memory

Hi,zhuoyanZhao
I met a problem when I run the demo,
here is the error:

RuntimeError Traceback (most recent call last)
in
----> 1 run_tasks('./config.yml')

~/.local/lib/python3.6/site-packages/nest/parser.py in run_tasks(config_file, param_file, verbose)
135 logger.info('Finished (%s).' % (U.format_elapse(seconds=(end_time - param_start_time).total_seconds())))
136 else:
--> 137 resolved_config = parse_config(config, env_vars=env_vars)
138 check_all_resolved(resolved_config)
139

~/.local/lib/python3.6/site-packages/nest/parser.py in parse_config(config, env_vars, global_vars)
68 return nest_module(**config)
69 else:
---> 70 return nest_module(**config, delay_resolve=True)
71
72 return config

~/.local/lib/python3.6/site-packages/nest/modules.py in call(self, *args, **kwargs)
160 try:
161 self._check_params(resolved_params)
--> 162 returns = self.func(**resolved_params)
163 except KeyError as exc_info:
164 if 'Nest module' in str(exc_info):

~/Nest-pytorch/install/trainer.py in network_trainer(data_loaders, model, criterion, optimizer, parameter, meters, hooks, max_epoch, test_interval, resume, log_path, device, use_data_parallel, use_cudnn_benchmark, random_seed)
81
82 # setup model
---> 83 model = model.to(device)
84
85 # multi-gpu support

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in to(self, *args, **kwargs)
423 return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
424
--> 425 return self._apply(convert)
426
427 def register_backward_hook(self, hook):

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in _apply(self, fn)
199 def _apply(self, fn):
200 for module in self.children():
--> 201 module._apply(fn)
202
203 def compute_should_use_set_data(tensor, tensor_applied):

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in _apply(self, fn)
199 def _apply(self, fn):
200 for module in self.children():
--> 201 module._apply(fn)
202
203 def compute_should_use_set_data(tensor, tensor_applied):

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in _apply(self, fn)
199 def _apply(self, fn):
200 for module in self.children():
--> 201 module._apply(fn)
202
203 def compute_should_use_set_data(tensor, tensor_applied):

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in _apply(self, fn)
221 # with torch.no_grad():
222 with torch.no_grad():
--> 223 param_applied = fn(param)
224 should_use_set_data = compute_should_use_set_data(param, param_applied)
225 if should_use_set_data:

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in convert(t)
421
422 def convert(t):
--> 423 return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
424
425 return self._apply(convert)

RuntimeError: CUDA error: out of memory

I have one GPU: GeForce RTX 2060
python :3.6.9
It is really weird it has out of memory before training. could you please give me some suggestions?

Does this PRM project support training？

Hi Yanzhao，Does this PRM project support training？
I want to training on VOC 2007. But I did not find the training codes.

用训练好的model测试其他图像时，.json格式的文件如何生成

Some details about Peak back-propagation

Your works is amazing!
Still, I'm puzzled about some details of the peak back-propagation process in your paper.
In the paper, a few demo results of your model show a promising performance that PRM could work well on most cases where objects in the image are not so dense. I'm wondering that what if the input image is crowded with objects, for example a group of people or a traffic jam. The reason I propose this issue is that as far as I'm concerned, there will be many problems when it encountering this kind of situations.
First, since the model extracts peaks of the feature map (which called Peak stimulation), if many objects are overlapping or very close to each other, then it is very likely of your model being not able to locate precise peaks of each of them, because their features are probably messed up with each other.
Second, I've noticed that in paper, some results of peak back-propagation are shown, where different colors are used and making it fancy. My question is, according to the results, peak back-propagation could only recover part of objects, no matter what the object is, in paper, there are two kids playing with a car, and none of them are fully recovered after peak back-propagation. So, how do they turn into the final result?
I know you are very busy, sorry for this long issue. My last question is, what is the speed (frame per second) of the whole model during inference?

Understand peak threshold and how to set it?

Hi authors,

Can you please explain the peak_threshold and how to find a suitable value for a different dataset.

PRM/install/modules/peak_response_mapping.py

Line 173 in 1bf2068

def forward(self, input, class_threshold=0, peak_threshold=30, retrieval_cfg=None):

Pankesh

Wrong peak back-propagation implementation?

about the 'No such file or directory: './snapshots/model_latest.pt''

Hi @ZhouYanzhao !
I have a problem when I run this "state = torch.load('./snapshots/model_latest.pt')"
problem: "No such file or directory: './snapshots/model_latest.pt' ".?
How can I get this file?

Evaluation metrics: ABO and mAP

Thanks for the code!

What scripts did you use to compute the mAP and the average best overlapping (ABO) metrics?

Thanks!

Peak Stimulation

Hi @ZhouYanzhao

May I ask how did you implement Peak Stimulation (local max) ?
Did you build a custom layer or by existing pytorch function ?
I'm trying to build one, but still haven't come up with a solution which makes it differentiable yet.

about the "pre-computed object proposals"

hi,@ZhouYanzhao,I am confused that How to prepare the "pre-computed object proposals".
Hope to receive your reply!Tks

Pretrained model weights not found at ./snapshots/model_latest.pt

When I run demo/main.ipynb

state = torch.load('./snapshots/model_latest.pt')

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-13-d7aa072378d3> in <module>()
      1 # loaded pre-trained weights
      2 model = nn.DataParallel(model)
----> 3 state = torch.load('./snapshots/model_latest.pt')
      4 model.load_state_dict(state['model'])
      5 model = model.module.cuda()

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in load(f, map_location, pickle_module)
    354             (sys.version_info[0] == 3 and isinstance(f, pathlib.Path)):
    355         new_fd = True
--> 356         f = open(f, 'rb')
    357     try:
    358         return _load(f, map_location, pickle_module)

FileNotFoundError: [Errno 2] No such file or directory: './snapshots/model_latest.pt'

Can't achieve the performance reported in the paper

Hi,

This is an interesting work.
But we can't achieve the performance reported in PRM (mAP50: 26.8 with MCG proposals).
We can only get 11.5 mAP50 with MCG proposals downloaded from https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/mcg/
and 21.5 mAP50 with COB proposals downloaded from http://www.vision.ee.ethz.ch/~cvlsegmentation/cob/code.html.

We use the default parameters of PRM (https://github.com/ZhouYanzhao/PRM/blob/pytorch/demo/config.yml) to train the classification network, (change the train_splits from trainval to trianaug, of course). But we notice that both the quality of the peaks and the instance masks are worse than those reported in the paper,

So we wonder if you use other hyper-parameter settings in your experiments?

Besides, according to our observations, the MCG proposals from https://data.vision.ee.ethz.ch/jpont/mcg/MCG-Pascal-Segmentation_trainvaltest_2012-proposals.tgz are much worse than the those shown in the paper and the supplement material. So do we need to retrain MCG with PASCAL train set to generate better proposals?

Would you please point out the differences between my experiments and yours that may results in the gap? or could you give us some advice to boost the performance?

Thanks a lot.

Is there a downloadable link to the model snapshot?

I am trying to load ./snapshots/model_latest.pt to run the main.ipynb notebook. Is there a link to the latest model?

Thank you.

ran out of input

---

EOFError Traceback (most recent call last)
in
----> 1 run_tasks('./config.yml')

~/.conda/envs/prm/lib/python3.6/site-packages/nest/parser.py in run_tasks(config_file, param_file, verbose)
135 logger.info('Finished (%s).' % (U.format_elapse(seconds=(end_time - param_start_time).total_seconds())))
136 else:
--> 137 resolved_config = parse_config(config, env_vars=env_vars)
138 check_all_resolved(resolved_config)
139

~/.conda/envs/prm/lib/python3.6/site-packages/nest/parser.py in parse_config(config, env_vars, global_vars)
58 elif isinstance(val, dict):
59 config[key] = parse_config(
---> 60 val, env_vars=env_vars, global_vars=global_vars)
61 if key == '_var':
62 U.merge_dict(global_vars, config[key], union=True)

~/.conda/envs/prm/lib/python3.6/site-packages/nest/parser.py in parse_config(config, env_vars, global_vars)
58 elif isinstance(val, dict):
59 config[key] = parse_config(
---> 60 val, env_vars=env_vars, global_vars=global_vars)
61 if key == '_var':
62 U.merge_dict(global_vars, config[key], union=True)

~/.conda/envs/prm/lib/python3.6/site-packages/nest/parser.py in parse_config(config, env_vars, global_vars)
68 return nest_module(**config)
69 else:
---> 70 return nest_module(**config, delay_resolve=True)
71
72 return config

~/.conda/envs/prm/lib/python3.6/site-packages/nest/modules.py in call(self, *args, **kwargs)
160 try:
161 self._check_params(resolved_params)
--> 162 returns = self.func(**resolved_params)
163 except KeyError as exc_info:
164 if 'Nest module' in str(exc_info):

~/PRM-pytorch/install/prm.py in fc_resnet50(num_classes, pretrained)
17 """FC ResNet50.
18 """
---> 19 model = FC_ResNet(models.resnet50(pretrained), num_classes)
20 return model
21

~/.conda/envs/prm/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg/torchvision/models/resnet.py in resnet50(pretrained, **kwargs)
186 model = ResNet(Bottleneck, [3, 4, 6, 3], **kwargs)
187 if pretrained:
--> 188 model.load_state_dict(model_zoo.load_url(model_urls['resnet50']))
189 return model
190

~/.conda/envs/prm/lib/python3.6/site-packages/torch/utils/model_zoo.py in load_url(url, model_dir, map_location, progress)
64 hash_prefix = HASH_REGEX.search(filename).group(1)
65 _download_url_to_file(url, cached_file, hash_prefix, progress=progress)
---> 66 return torch.load(cached_file, map_location=map_location)
67
68

~/.conda/envs/prm/lib/python3.6/site-packages/torch/serialization.py in load(f, map_location, pickle_module)
301 f = open(f, 'rb')
302 try:
--> 303 return _load(f, map_location, pickle_module)
304 finally:
305 if new_fd:

~/.conda/envs/prm/lib/python3.6/site-packages/torch/serialization.py in _load(f, map_location, pickle_module)
457 f.seek(0)
458
--> 459 magic_number = pickle_module.load(f)
460 if magic_number != MAGIC_NUMBER:
461 raise RuntimeError("Invalid magic number; corrupt file?")

EOFError: Ran out of input

Has the code been deleted by the author? Where can I download it?

nest module install github@ZhouYanzhao/PRM:pytorch prm

My server can't connect to the external network, I have no way to nest, I have downloaded the zip to the local and uploaded it to the server, how should I install PRM?

Nest module

Hi Yanzhao,
Thanks for the code! Could you explain how to set up the nest modules? I tried 'nest module list' without prm filter, but it still only gives the three modules under prm instead of all 25 modules. So when I try to run the demo, it gives this error: 'Could not find a Nest module matches query "*pascal_voc_classification".' How to set it up properly?

Thanks!

=====
Reinstalled and solved! thanks

Always "No class peak response detected"

Hi, I fine-tuned the model with my own dataset. I just fine-tuned the classification layer freezing the feature layers, with a very small dataset of 48 images. I have only 6 classes.
However, while testing I am always getting "No class peak response detected". Can anyone please suggest me the probable reason for that.

Version of Python

Will te be possible to run in python 3.5.2

I want to test with pascal voc2012. How should I generate JSON files？

how to train my dataset?

Do anyone known how to train my dataset by using this repo?

KeyError: 'Could not find a Nest module matches query "*fc_resnet50".'

In the demo file, when I try to run : run_tasks('./config.yml'). I am getting the following error.
KeyError: 'Could not find a Nest module matches query "*fc_resnet50".'

Could you please help me solve this.

Unable to execute run_tasks

Hi,

I ran into this problem

TypeError: The param "transform" of Nest module "fetch_data" should be type of "Union[Callable, NoneType]". Got
image_transform(
[✓] image_size: Union[int, List[int]],
augmentation: dict,
[✓] mean: List[float] = [0.485, 0.456, 0.406],
[✓] std: List[float] = [0.229, 0.224, 0.225]) -> Callable
Please check if some important params of Nest module "image_transform" have been forgotten in use.

Anyone of you know how to solve this? Or alternativelty anyone of you has the training code without Nest?

Thanks

the point of 'sub_pixel_locating_factor'

Dear Yanzhao,
Sorry to bother but I have some issues when running the code. In the paper, there is a description as "We first upsample the class response maps to the size of the image via bilinear interpolation." However, in the code , the size of upsampling class response map is 112*112. It doesn't match the above description for the input image is of size 448 * 448. And I don't understand the meaning of the parameter sub_pixel_locating_factor.
Hope for your reply.

Where is modules.rle_decode?

In demo/main.ipynb, In[11] line 6, proposals = list(map(modules.rle_decode, json.load(f))). I can not find rle_decode in config.yml. Does any one know where can I find it? Thanks.

Hook module not installed

when running this commans, hook modules such as 'print_state' are not installing!

nest module install github@ZhouYanzhao/Nest:pytorch pytorch

Can not pass the verification

Execute " nest module list --filter prm" and get the following result.(Ubuntu16)

(pt-py3) root@ebe535e9a1a8:~# nest module list --filter prm
Traceback (most recent call last):
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/modules.py", line 346, in _import_nest_modules_from_file
spec.loader.exec_module(py_module)
File "", line 665, in exec_module
File "", line 222, in _call_with_frames_removed
File "/root/PRM/install/prm.py", line 30, in
filter_type: Union[str, int, float] = 'median') -> nn.Module:
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/modules.py", line 262, in _register
return create_module(args[0])
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/modules.py", line 259, in create_module
return type('NestModule', (NestModule,), dict(slots=(), doc=doc))(func, nest_meta)
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/modules.py", line 69, in init
self._check_definition()
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/modules.py", line 80, in _check_definition
if v.default is not inspect.Parameter.empty and not U.is_annotation_matched(v.default, v.annotation):
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/logger.py", line 59, in wrapper
res = func(*args, **kwargs)
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/utils.py", line 236, in is_annotation_matched
sub_annotation = annotation.args
AttributeError: type object 'Union' has no attribute 'args'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/Envs/pt-py3/bin/nest", line 11, in
sys.exit(main())
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/main.py", line 4, in main
CLI()
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/cli.py", line 71, in init
getattr(self, 'cmd_' + args.command)('nest ' + args.command, sys.argv[2:])
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/cli.py", line 162, in cmd_module
module_info = ['%s (%s)' % (k, v.meta.get('version', 'version')) for k, v in module_manager]
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/modules.py", line 702, in iter
self._update_modules()
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/modules.py", line 692, in _update_modules
ModuleManager._import_nest_modules_from_dir(meta['module_path'], namespace, self.py_modules, self.nest_modules, meta)
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/modules.py", line 407, in _import_nest_modules_from_dir
ModuleManager._import_nest_modules_from_file(file_path, namespace, py_modules, nest_modules, meta)
File "/root/Envs/pt-py3/lib/python3.5/site-packages/nest/modules.py", line 356, in _import_nest_modules_from_file
if (type(exc_info) is ModuleNotFoundError or type(exc_info) is ImportError) and exc_info.name is not None:
NameError: name 'ModuleNotFoundError' is not defined

能不能脱离nest运行code？

普通pytorch code并不是很复杂，要是有能不依赖nest工具的运行版本就好了。毕竟为了运行一个code去熟悉另外一套封装，加大了复杂度，与其它code也不易整合。

关于使用cpu的问题

作者您好，我看你的介绍时说可以使用cpu训练，但速度很慢，现在我受条件限制只能用cpu，在运行demo时报了这个错：
~\Nest-pytorch\install\trainer.py in network_trainer(data_loaders, model, criterion, optimizer, parameter, meters, hooks, max_epoch, test_interval, resume, log_path, device, use_data_parallel, use_cudnn_benchmark, random_seed)
74 device = torch.device(device)
75 if device.type == 'cuda':
---> 76 assert torch.cuda.is_available(), 'CUDA is not available.'
77 torch.backends.cudnn.benchmark = use_cudnn_benchmark
很明显这是没装cuda的结果，但nest作为一个网上基本查不到资料的工具，我也不知道该怎么改才能继续在cpu状态训练，希望您能回答我，谢谢。

Stack overflow and OOM

Hi,

Very cool work! However I run into some issues trying to experiment with it.
How much GPU/CPU memory is recommended for training and inference?

I have a pretty powerful machine (GPU RTX 24 GB RAM) but I could only train with a batch size of 1.

Also at inference time I get stack overflow at the line:
proposals = list(map(modules.rle_decode, json.load(f)))

Any advice on how to fix it would be appreciated.

Thanks

run_tasks('./config.yml') TypeError

TypeError occurs when I run the cell " run_tasks('./config.yml') ".The details are as follows.Everything is OK when I install the nest and PRM modules.Hope someone could help me solve this problem.Please~
帮帮我~

TypeError: The param "transform" of Nest module "fetch_data" should be type of "Union[Callable, NoneType]". Got
image_transform(
[✓] image_size: Union[int, List[int]],
augmentation: dict,
[✓] mean: List[float] = [0.485, 0.456, 0.406],
[✓] std: List[float] = [0.229, 0.224, 0.225]) -> Callable
Please check if some important params of Nest module "image_transform" have been forgotten in use.

Any news on the implementation?

Will the implementation be open sourced soon? Amazing work btw.

How to run it for your own set of images/videos?

Hi,
I love the work you have done. But how can we run this for our own set of images/videos?
Thanks