linardos / salema

Simple vs complex temporal recurrences for video saliency prediction (BMVC 2019)

SalEMA

SalEMA is a video saliency prediction network. It utilizes a moving average of convolutional states to produce state of the art results. The architecture has been trained on DHF1K.

Publication

Find the published version of our work here: arXiv, or check our friendly summary on medium. Also, if this project has been helpful to your work, please consider citing us:

@inproceedings{linardos2019simple,
  author    = {Panagiotis Linardos and
               Eva Mohedano and
               Juan Jos{\'{e}} Nieto and
               Noel E. O'Connor and
               Xavier Gir{\'{o}}{-}i{-}Nieto and
               Kevin McGuinness},
  title     = {Simple vs complex temporal recurrences for video saliency prediction},
  booktitle = {30th British Machine Vision Conference 2019, {BMVC} 2019, Cardiff,
               UK, September 9-12, 2019},
  pages     = {182},
  publisher = {{BMVA} Press},
  year      = {2019},
  url       = {https://bmvc2019.org/wp-content/uploads/papers/0952-paper.pdf},
  timestamp = {Thu, 30 Apr 2020 17:36:09 +0200},
  biburl    = {https://dblp.org/rec/conf/bmvc/LinardosMNONM19.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

Model

• Download our best configuration of the SalEMA model here (364MB)
• Download our best configuration of the SalCLSTM model here (580MB)
• Download our baseline, the SalBCE model here (364MB)

Sample video (click to be redirected to youtube):

Installation

• Clone the repo:

git clone https://github.com/Linardos/SalEMA

• Install requirements

pip install -r requirements.txt

Inference

You may use our pretrained model for inference on either of the 3 datasets: DHF1K [link], Hollywood-2 [link], UCF-sports [link] or your own dataset so long as it follows a specific folder structure:

To perform inference on DHF1K validation set:

python inference.py -dataset=DHF1K -pt_model=SalEMA30.pt -alpha=0.1 -start=600 -end=700 -dst=/path/to/output -src=/path/to/DHF1K

To perform inference on Hollywood-2 or UCF-sports test set (because of the way the dataset is structured, it's convenient to use the same path for dst and src):

python inference.py -dataset=Hollywood-2 -pt_model=SalEMA30.pt -alpha=0.1 -dst=/path/to/Hollywood-2/testing -src=/path/to/Hollywood-2/testing

python inference.py -dataset=UCF-sports -pt_model=SalEMA30.pt -alpha=0.1 -dst=/path/to/UCF-sports/testing -src=/path/to/UCF-sports/testing

To perform inference on your own dataset make sure to follow a simple folder structure (one superfolder given as root at the input, which includes folders of frames) and use the tag "other":

python inference.py -dataset=other -alpha=0.1 -pt_model=SalEMA30.pt -dst=/path/to/output -src=/path/to/superfolder/frames

If your dataset follows a more quirky structure you might need to manipulate the data_loader source code.

Training

To train on DHF1K using CUDA:

python train.py -dataset=DHF1K -pt_model=False -new_model=SalEMA -ema_loc=30 -start=1 -end=4 -src=/path/to/DHF1K -use_gpu='gpu' -epochs=7

To train on Hollywood-2, UCF-sports using CUDA. For fine-tuning a pretrained model, use a higher number of epochs, the training commences from the epoch number where it stopped on:

python train.py -dataset=Hollywood-2 -pt_model=SalEMA30.pt -new_model=SalEMA -ema_loc=30 -src=/path/to/Hollywood-2 -use_gpu='gpu' -epochs=10 -lr=0.0000001

salema's Issues

Some bugs in train.py

Hi, in your train.py line 163
` if args.pt_model != False:
# In truth it's not None, we default to SalGAN or SalBCE (JuanJo's)weights
# By setting strict to False we allow the model to load only the matching layers' weights
if SALGAN_WEIGHTS == 'model_weights/gen_model.pt':
model.salgan.load_state_dict(torch.load(SALGAN_WEIGHTS), strict=False)
else:

        model.salgan.load_state_dict(torch.load(SALGAN_WEIGHTS)['state_dict'], strict=False)
    start_epoch = 1
else:
    # Load an entire pretrained model
    checkpoint = load_weights(model, args.pt_model)
    model.load_state_dict(checkpoint, strict=False)
    start_epoch = torch.load(args.pt_model, map_location='cpu')['epoch']
    #optimizer.load_state_dict(torch.load(args.pt_model, map_location='cpu')['optimizer'])

    print("Model loaded, commencing training from epoch {}".format(start_epoch))`

the type of "args.pt_model' is "str", so no matter how we set the pt_model is "False" or "True", It will always use the weight of SalGAN or SalBCE right?

Error while running the inference.py

Hi @Linardos
I am trying to run the inference.py file for UCF data and I got the following error, can you please tell me what is the problem..

I run the following command python inference.py -dataset=UCF-sports -pt_model=/home/dsvk/MTP1/SalEMA/SalEMA30.pt -alpha=0.1 -dst=/home/dsvk/MTP1/SalEMA/data/UCF-sports/testing -src=/home/dsvk/MTP1/SalEMA/data/UCF-sports/testing

I get the following:

Output directory /home/dsvk/MTP1/SalEMA/data/UCF-sports/testing
Commencing inference for dataset UCF-sports
Size of test set is 47
Model initialized, EMA located at Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
Pre-trained model /home/dsvk/MTP1/SalEMA/SalEMA30.pt loaded succesfully
Alpha tuned to tensor([0.1000])
Be warned, you are about to write on an existing folder /home/dsvk/MTP1/SalEMA/data/UCF-sports/testing. If this is not intentional cancel now.
Destination: /home/dsvk/MTP1/SalEMA/SalEMA30_predictions
The final destination is /home/dsvk/MTP1/SalEMA/SalEMA30_predictions/Diving-Side-0000001.png. Cancel now if this is incorrect
Traceback (most recent call last):
File "inference.py", line 302, in
main(args)
File "inference.py", line 230, in main
state, saliency_map = model.forward(input_ = clip[idx], prev_state = state)
File "/home/dsvk/MTP1/SalEMA/src/model/SalEMA.py", line 108, in forward
current_state = sigmoid(self.alpha)*self.salganself.ema_loc+(1-sigmoid(self.alpha))*prev_state
RuntimeError: expected device cpu and dtype Float but got device cuda:0 and dtype Float

My Cuda and torch versions are:
cuda: 9.2, V9.2.148
torch : 1.2.0
torchvision : 0.4.0a0+9232c4a

Wrong usage of dropout

Hey dude, you've used dropout incorrectly

if self.dropout is True:
    x = dropout2d(x)

nn.Dropout performs different during train and test phase, your code will result in incorrect inference

License

You are missing a LICENSE file,
What is the license of this code repository and model?

Thank you

request for SalCLSTM model

Train with CUDA meet Error: raise ValueError("can't optimize a non-leaf Tensor")

Hi, When I train this model use "python train.py -dataset=DHF1K -pt_model=False -new_model=SalEMA -ema_loc=30 -start=1 -end=10 -src=F:/2/DHF1K -use_gpu='gpu' -epochs=7", It repoert error like this:
Commencing training on dataset DHF1K Size of train set is 1 Initial alpha set to: tensor([0.2500], device='cuda:0', grad_fn=<CopyBackwards>) Model initialized, EMA located at Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) Initialized SalEMA with residual set to False and dropout set to True Traceback (most recent call last): File "train.py", line 518, in <module> main(args) File "train.py", line 147, in main optimizer = torch.optim.Adam([{'params':model.salgan.parameters() , 'lr': args.lr, 'weight_decay':weight_decay},{'params':model.alpha, 'lr': 0.1}]) File "D:\App\Anaconda\lib\site-packages\torch\optim\adam.py", line 42, in __init__ super(Adam, self).__init__(params, defaults) File "D:\App\Anaconda\lib\site-packages\torch\optim\optimizer.py", line 50, in __init__ self.add_param_group(param_group) File "D:\App\Anaconda\lib\site-packages\torch\optim\optimizer.py", line 200, in add_param_group raise ValueError("can't optimize a non-leaf Tensor") ValueError: can't optimize a non-leaf Tensor
Can you help me? The environment is the same as the requirement.

request for model_weights/salgan_salicon.pt file

Hi, thanks for releasing the source code for your work!

Could you please also release the salgan_salicon.pt file mentioned in line 33 of train.py?

Thanks,
Ekta

Training on custom dataset

@Linardos
Hi, I want to train the model on a custom dataset, can you describe the steps involved and details like the structure and format of the training input and annotation data, the training parameters you used for the training etc.

thank you.