hongje / hmmn

Hierarchical Memory Matching Network for Video Object Segmentation (ICCV 2021)

Hierarchical Memory Matching Network for Video Object Segmentation

Hongje Seong, Seoung Wug Oh, Joon-Young Lee, Seongwon Lee, Suhyeon Lee, Euntai Kim

ICCV 2021

This is the implementation of HMMN.
This code is based on STM (ICCV 2019): [link].
Please see our paper for the details: [paper]

Dependencies

• Python 3.8
• PyTorch 1.8.1
• numpy, opencv, pillow

Trained model

• Download pre-trained weights into the same folder with demo scripts
  Link: [weights]

Code

• DAVIS-2016 validation set (Single-object)

python eval_DAVIS.py -g '0' -s val -y 16 -D [path/to/DAVIS]

• DAVIS-2017 validation set (Multi-object)

python eval_DAVIS.py -g '0' -s val -y 17 -D [path/to/DAVIS]

Pre-computed Results

We also provide pre-computed results for benchmark sets.

• [DAVIS-16-val]
• [DAVIS-17-val]
• [DAVIS-17-test-dev]
• [YouTube-VOS-18-valid]
• [YouTube-VOS-19-valid]

Bibtex

@inproceedings{seong2021hierarchical,
  title={Hierarchical Memory Matching Network for Video Object Segmentation},
  author={Seong, Hongje and Oh, Seoung Wug and Lee, Joon-Young and Lee, Seongwon and Lee, Suhyeon and Kim, Euntai},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  year={2021}
}

Terms of Use

This software is for non-commercial use only. The source code is released under the Attribution-NonCommercial-ShareAlike (CC BY-NC-SA) Licence (see this for details)

hmmn's Issues

Regarding the results on YouTube-VOS

Hi,

THX for releasing the inference code of DAVIS.

May I ask what frame rate did you use in the evaluation of HMMN and KMN on YouTube-VOS? Validation or Validation All Frames? 6FPS or 30FPS?

Best.

How to train your model？

I noticed that this code only included evaluation part and lack the training part. It would be really helpful if you could proide the training code. Another question is that the model in this code has some differences compared to the paper. In the top-k guided memory matching module, the model in this code don't have the VQ3 add to the result which come out from dropout. Instead, it directly uses results after the second ''matmul'' as the output of the top-k module (in Figure.6 in the paper this is Z4). Could you please explain the reason? Moreover, I wonder if I need to add a dropout layer, a conv3 layer and a ''VQ3 add.'' layer when training the model. Of course if you can provide me with your training code, it would be most helpful! Thank you for your time!

Dropout in the top-k guided memory matching module and the value of query(r2,r3)

Excellent work!
I have some question about the dropout in the top-k guided memory matching module and the value of query(r2,r3).
In your code, you doesn't use the value of query of r2 and r3 stage, but in the paper you use the value to generate Z3. The operation in the code is different from the description in the paper.

question about the top-k guided memory matching

thanks for your sharing . wonderful job,
but why i didn't see the dropout in top-k guided memory matching and the conv3*3 before it in your code.
and the KV_Q_r3 = KeyValue(512, keydim=128, valdim=-1, only_key=True) means there is no need for values.
What do I understand it ?

training code？

Training code

Are you also planning to share code for training (e.g. based on DAVIS data)?

thanks in advance