mahdiabavisani / deep-multimodal-subspace-clustering-networks

Tensorflow implementation of "Deep Multimodal Subspace Clustering Networks"

Deep multimodal subspace clustering networks

Overview

This repository contains the implementation of the paper "Deep multimodal subspace clustering networks" by Mahdi Abavisani and Vishal M. Patel. The paper was posted on arXiv in May 2018.

"Deep multimodal subspace clustering networks" (DMSC) investigated various fusion methods for the task of multimodal subspace clustering, and suggested a new fusion technique called "affinity fusion" as the idea of integrating complementary information from two modalities with respect to the similarities between datapoints across different modalities.

Citation

Please use the following to refer to this work in publications:

@ARTICLE{8488484, 
author={M. {Abavisani} and V. M. {Patel}}, 
journal={IEEE Journal of Selected Topics in Signal Processing}, 
title={Deep Multimodal Subspace Clustering Networks}, 
year={2018}, 
volume={12}, 
number={6}, 
pages={1601-1614}, 
doi={10.1109/JSTSP.2018.2875385}, 
ISSN={1932-4553}, 
month={Dec},}

Setup:

Dependencies:

Tensorflow, numpy, sklearn, munkres, scipy.

Data preprocessing:

Resize the input images of all the modalities to 32 × 32, and rescale them to have pixel values between 0 and 255. This is for keeping the hyperparameter selections suggested in Deep subspace clustering networks valid.

Save the data in a .mat file that includes verctorized modalities as separate matrices with the names modality_0,modality_1, ... ; labels in a vector with the name Labels; and number of modalities in the variable num_modalities.

A sample preprocessed dataset is available in: Data/EYB_fc.mat

Running the code

Affinity fusion :

Run affinity_fusion.py to do mutlimodal subspace clustering. For demo a pretrained model trained on EYB_fc is avilable in models/EYBfc_af.ckpt

Run the demo as:

python affinity_fusion.py --mat EYB_fc --model EYBfc_af

Pretraining:

Run pretrain_affinity_fusion.py to pretrain your networks.

For example:

python pretrain_affinity_fusion.py --mat EYB_fc --model mymodel --epoch 100000

deep-multimodal-subspace-clustering-networks's Issues

diagonal of co-eff matrix is non zero

Hi Mahdi,
I was wondering as in \eq(6) where the optimization function requires diag(\theta_s)=0, I could not find how you were forcing the same in the code.
Could you please point me to that line in the code.?
Thank you very much

Dataset

Could you release the dataset ARL?

Help

The coefficient matrix is saved in the variable "C" in a .mat file in the models_DSC folder (see the last line of the code). As to the digits, have you changed the kernel size, and hidden units to match the network reported in the paper? Is the number of samples around 2k samples? If not the learning rate or the max_step of iterations should be changed. From the results you posted, it seems that your learning rate is too high. Try a smaller learning rate (maybe 10e-5? ) or smaller max number of iterations (perhaps max_step =100 and display_step = 10) ... .

But the figure does not only need coefficient matrix,maybe needs predict label,right?

Also, I redo the setting as you said above and all other are the same as reported in the paper,it still went bad.

Could share the digit dataset with me ,so that I can find the problem that I met.

Originally posted by @HuangQinJian in #4 (comment)

need help

I train on the digit dataset,the result is bad:

the result decreases as the epoch growing,why?

I have checked the dataset and the network,and the vision of restruction picture is good

Originally posted by @HuangQinJian in #3 (comment)

dimension mismatch according to the paper

Hello, I wanna to reproduce the experiment, but I find some problem in dimension.

the decoder can not reconstruct the original data according to the paper.