pku-icst-mipl / cm-gans_tomm2019

Source code of our TOMM 2019 paper "CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning".

Introduction

This is the source code of our TOMM 2019 paper "CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning", Please cite the following paper if you find our code useful.

Yuxin Peng, Jinwei Qi, "CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning", ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Vol.15, No.1, pp.22:1-22:24, 2019. [PDF]

Preparation

Our code is based on torch, and tested on Ubuntu 14.04.5 LTS, Lua 5.1.

Usage

Data Preparation: we use pascal dataset as example, and the data should be put in ./data/. The data files can be download from the link and unzipped to the above path.

run sh run.sh to train models, extract features and calculate mAP.

Our Related Work

If you are interested in cross-media retrieval, you can check our recently published overview paper on IEEE TCSVT:

Yuxin Peng, Xin Huang, and Yunzhen Zhao, "An Overview of Cross-media Retrieval: Concepts, Methodologies, Benchmarks and Challenges", IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), Vol.28, No.9, pp.2372-2385, 2018. [PDF]

Welcome to our Benchmark Website and Laboratory Homepage for more information about our papers, source codes, and datasets.

cm-gans_tomm2019's Issues

How to implement Weight -sharing Constrains based on PyTorch?

Thanks for your code! It helps me a lot. And I want to re-implement your code based on PyTorch. However, I have problem with Weight -sharing Constrains. It seems like that it is quite simple in torch by function share:
encoder_img2:share(encoder_txt2,'weight','bias','gradWeight','gradBias')

I do not know how to implement Weight -sharing Constrains based on PyTorch. Could you tell me if you know it?

Thanks again.

What does err_L1 do?

It seems like err_L1 is just to print? But in annotation, It says "L1 loss which forces features generated by TG and IG to be closer". How does it do that?

Is data the features generated by VGG19 and TextCNN?

In paper, you say the image/text is first through VGG19/TextCNN. However, I can not find the VGG19 or TextCNN in train.lua. So I guess the data you used here is not original data but the features generated by VGG19 and TextCNN offline. Is that true?