GithubHelp home page GithubHelp logo

cvlab-yonsei / mnad Goto Github PK

View Code? Open in Web Editor NEW
332.0 12.0 82.0 1021 KB

An official implementation of "Learning Memory-guided Normality for Anomaly Detection" (CVPR 2020) in PyTorch.

Python 96.60% CSS 3.40%

mnad's Introduction

PyTorch implementation of "Learning Memory-guided Normality for Anomaly Detection"

no_imageno_image

This is the implementation of the paper "Learning Memory-guided Normality for Anomaly Detection (CVPR 2020)".

For more information, checkout the project site [website] and the paper [PDF].

Dependencies

  • Python 3.6
  • PyTorch 1.1.0
  • Numpy
  • Sklearn

Datasets

These datasets are from an official github of "Future Frame Prediction for Anomaly Detection - A New Baseline (CVPR 2018)".

Download the datasets into dataset folder, like ./dataset/ped2/

Update

  • 02/04/21: We uploaded the codes based on reconstruction method, and pretrained wieghts for Ped2 reconstruction, Avenue prediction and Avenue reconstruction.

Training

  • The training and testing codes are based on prediction method
  • Now you can implemnet the codes based on both prediction and reconstruction methods.
  • The codes are basically based on the prediction method, and you can easily implement this as
git clone https://github.com/cvlab-yonsei/projects
cd projects/MNAD/code
python Train.py # for training
  • You can freely define parameters with your own settings like
python Train.py --gpus 1 --dataset_path 'your_dataset_directory' --dataset_type avenue --exp_dir 'your_log_directory'
  • For the reconstruction task, you need to newly set the parameters, e.g,, the target task, the weights of the losses and the number of the time sequence.
python Train.py --method recon --loss_compact 0.01 --loss_separate 0.01 --t_length 1 # for training

Evaluation

  • Test your own model
  • Check your dataset_type (ped2, avenue or shanghai)
python Evaluate.py --dataset_type ped2 --model_dir your_model.pth --m_items_dir your_m_items.pt
  • For the reconstruction task, you need to set the parameters as
python Evaluate.py --method recon --t_length 1 --alpha 0.7 --th 0.015 --dataset_type ped2 --model_dir your_model.pth --m_items_dir your_m_items.pt
  • Test the model with our pre-trained model and memory items
python Evaluate.py --dataset_type ped2 --model_dir pretrained_model.pth --m_items_dir m_items.pt

Pre-trained model and memory items

Bibtex

@inproceedings{park2020learning,
  title={Learning Memory-guided Normality for Anomaly Detection},
  author={Park, Hyunjong and Noh, Jongyoun and Ham, Bumsub},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={14372--14381},
  year={2020}
}

mnad's People

Contributors

hyunjp avatar njyoun avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mnad's Issues

IndexError: list index out of range

I test the model with your pre-trained model and memory items, but it prompts the following error:

Traceback (most recent call last):
File "Evaluate.py", line 118, in
label_length += videos[videos_list[video_num].split('/')[-1]]['length']
IndexError: list index out of range

RuntimeError

Prediction code is well utilized.

Thank you for the intuitive usage.

I am trying to learn about Reconstruction this time, but I encountered the following error.

RuntimeError: Given groups=1, weight of size [64, 3, 3, 3], expected input[4, 15, 256, 256] to have 3 channels, but got 15 channels instead

The sample code you provided
I entered python Train.py --method recon --loss_compact 0.01 --loss_separate 0.01 as it is, but I get the same error as above.

Is there any solution?

About pytorch bug

As I know, pytorch ver. 1.1 has a bug(I don't know it is really bug) that it doesn't track the mean and variance from training set at evaluation step.
And it makes results over repeated evaluation different.
At first evaluation, futhermore, the evaluation result become too poor because there is no tracked mean & var.
I also tried to replicate your result by your own code, but it couldn't generate 88.5(avenue) or 97.0(ped2).
Did you notice the pytorch bug?
How could you acheive the result even with this bug?
I 'll appreciate your reply. Thank you.

Visualization

Thanks for the perfect code.
The evaluation was able to proceed without any difficulty.
Could you please explain how to visualize the result as a picture as suggested in the paper?

About t-SNE

Can I get the code for this part of t-SNE?

AttributeError: 'convAE' object has no attribute 'clone'

I used pytorch1.1.0,and no change the code,but show this issue:

Traceback (most recent call last):
File "/home/lh/MNAD-master1/Evaluate.py", line 121, in
m_items_test = m_items.clone()
File "/home/lh/anaconda3/envs/torch11/lib/python3.6/site-packages/torch/nn/modules/module.py", line 539, in getattr
type(self).name, name))
AttributeError: 'convAE' object has no attribute 'clone'

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

Hi, i`m trying to experiment your model to the Shanghaitech dataset. but i got a error when evaluating the code.
"ValueError: Input contains NaN, infinity or a value too large for dtype('float64')."
It would be very much appreciated if you could provide me some help.
this is my error
스크린샷 2020-08-25 오후 6 57 40

there is a nan value in the anomaly_score_tatal_list of the Evaluate.py.
how can i solve this error??

All the best.

Unable to save recreated images and run model real time on a video

Hello!
Thank you for your great contribution towards anomaly detection. I am currently getting started with your repo and have trained the model on the UCSD dataset. However, how do I create a demo showing the anomaly scores real time as the video plays and how do I generate the reconstructions. Is a script available for the same? Thank you!

Faulty evaluation calculation?

Isn't the evaluation calculation wrong?
Considering Ped2, why are the scores normalized between 0 and 1 for each of the 12 video clips individually? This isn't correct as the camera view is the same in all of the clips.

RuntimeError:

Traceback (most recent call last):
File "D:/python/pycharm/MNAD-master/Evaluate.py", line 139, in
outputs, feas, updated_feas, m_items_test, softmax_score_query, softmax_score_memory, _, _, _, compactness_loss = model.forward(imgs[:,0:3*4], m_items_test, False)
File "D:\python\pycharm\MNAD-master\model\final_future_prediction_with_memory_spatial_sumonly_weight_ranking_top1.py", line 135, in forward
fea, skip1, skip2, skip3 = self.encoder(x)
File "D:\python\Anaconda\envs\MNAD\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "D:\python\pycharm\MNAD-master\model\final_future_prediction_with_memory_spatial_sumonly_weight_ranking_top1.py", line 46, in forward
tensorConv1 = self.moduleConv1(x)
File "D:\python\Anaconda\envs\MNAD\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "D:\python\Anaconda\envs\MNAD\lib\site-packages\torch\nn\modules\container.py", line 141, in forward
input = module(input)
File "D:\python\Anaconda\envs\MNAD\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "D:\python\Anaconda\envs\MNAD\lib\site-packages\torch\nn\modules\conv.py", line 446, in forward
return self._conv_forward(input, self.weight, self.bias)
File "D:\python\Anaconda\envs\MNAD\lib\site-packages\torch\nn\modules\conv.py", line 443, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size [64, 12, 3, 3], expected input[1, 3, 256, 256] to have 12 channels, but got 3 channels instead

Pretrained models

Hi, the link to the pretrained files seems to be broken. Can you please point to the locations.

about prediction task codes ?

can someone tell me that how U-Net is training in this whole code because i am not able to see any code which is used to train U-Net which is predicting a new frame.

This code can be used in not video frame detection conditions?

Hello!
First, thank you so much for this contribution.
My question:
This code is for abnormal video frame detection, I wonder if it can be used in other conditions, e.g., one small abnormal group of images different from most normal images, not video frames, but medical images etc.
Thank you !

replicate results on CUHK Avenue dataset

Hi, Thanks a lot for your great work!

I want to replicate your results on the CUHK Avenue dataset, but I can only get around 70% anomaly detection accuracy. Then I give another read on the paper. I found there are several places that I am not quite sure:

  1. I trained the model using PyTorch 1.1.0, with 60 epochs, initial learning rate 2e-4 which is decayed by the cosine annealing method. However, it seems that the Separateness jumps up and down, so I am wondering is it because of my learning rate too high?

  2. On page 6, it's said that "We set initial learning rates to 2e-5 and 2e-4, respectively, for reconstruction and prediction tasks", but from what I have seen from the code, it seems that there is only one learning rate.

Could you please give me some help with this? Thanks a lot in advance!

NameError: name 'loss_pixel' is not defined

I followed the steps in the readme and reported the error as follows:
/home/anaconda3/envs/MNAD/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Traceback (most recent call last):
File "Train.py", line 148, in
print('Loss: Prediction {:.6f}/ Compactness {:.6f}/ Separateness {:.6f}'.format(loss_pixel.item(), compactness_loss.item(), separateness_loss.item()))
NameError: name 'loss_pixel' is not defined

How to solve it? Thank you

I want to check for 0 or 1 per frame.

I found that the anomaly score was calculated for each frame in Evaluate.py.
Finally, the AUC value was also confirmed.

However, for qualitative analysis of results, i want to identify TP, TN, FP and FN for each frame.
In conclusion, in order ro know which frame is judged as normal or anomaly, i want to know which part of the code to look at and understand.

Thank you.

evaluate.py

when i use the model provided by the author to evaluate it, "ValueError: Found input variables with inconsistent numbles of samples:[1966,7056]". Has someone else had the same problem?

Zero values for compactness loss

Hi,

Thank you very much for the code and interesting paper!

We are trying to run your method on our dataset, but seem to often get zero values for compactness loss.
This is specifically a problem when we try to normalize the "feature_distance_list" during inference.

Could you please advise?

Also, if you have any suggestions how to scale the number of epochs with the size of the dataset, please share.

Niv.

Error using my own dataset

Hello, when I use my own dataset for training, the following error occurs, but this error does not occur when I use CUHK Avenue. My dataset directory is as follows,can you tell me why and how to solve it? Thank you!
image
image

about shanghai dataset on evaluation

The model can be trained normally during training, but an error is reported during testing. Does anyone know how to solve it?

File "F:\PycharmProjects\MNAD\Evaluate1.py", line 114, in valrec1
labels_list = np.append(labels_list,labels[0][4+label_length:videos[video_name]['length']+label_length])
IndexError: invalid index to scalar variable.
This error is also reported on Linux

The following is the structure of Shanghai data set:
图片

Problem about trainset of ShanghaiTech

I found that all videos from ped1, Avenue and the test set of Shanghaitech of science and technology were extracted into frames. However, Shanghaitech's trainsets are in video format. How to extract it?

How to make the reconstruction more accurate?

Thank you for sharing this project.

I am using this project for medical images, and the reconstruction result is:

image

In the above figure, the left image is the original CT image, and the right image is the reconstructed image. The reconstructed image is blur. I used the default setting. How can I make the reconstruction result more accurate?

The only way I can think of is to make the encoder/decoder deeper. Is there any other method to make the reconstruction result more accurate? It's appreciated for any suggestion.

An error occurred while training the Shanghai dataset

Traceback (most recent call last):
File "Train.py", line 154, in
for j,(imgs) in enumerate(train_batch):
File "/root/userfolder/software/anaconda3/envs/MemG/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 819, in next
return self._process_data(data)
File "/root/userfolder/software/anaconda3/envs/MemG/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/root/userfolder/software/anaconda3/envs/MemG/lib/python3.6/site-packages/torch/_utils.py", line 369, in reraise
raise self.exc_type(msg)
IndexError: Caught IndexError in DataLoader worker process 1.
Original Traceback (most recent call last):
File "/root/userfolder/software/anaconda3/envs/MemG/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/root/userfolder/software/anaconda3/envs/MemG/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/root/userfolder/software/anaconda3/envs/MemG/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/root/userfolder/code/projects1/MNAD/FFPMem/model/utils.py", line 71, in getitem
image = np_load_frame(self.videos[video_name]['frame'][frame_name+i], self._resize_height, self._resize_width)
IndexError: list index out of range

How to decide number of memory items "m" ?

Hi, great work. There is something I would like to know. How do you decide the number of memory items? Like in your paper you used 10 memory items How do you decide that number ? Is there a rationale behind it ? I'll be very grateful to get an answer. Thank you

CUHK03 dataset and DukeMTMC-reID dataset

Hi,the work RRID is excellent, and I has followed your framework. However, recently when I try to train CUHK03 dataset,it cannot to be uploaded. Could you please provide the dataset link with its json data? Thank you very much!
Looking forward to your reply!

A question about the structure of reconstruction network

In the Encoder of reconstruction network, some code is:

        def Basic(intInput, intOutput):
            return torch.nn.Sequential(
                torch.nn.Conv2d(in_channels=intInput, out_channels=intOutput, kernel_size=3, stride=1, padding=1),
                torch.nn.BatchNorm2d(intOutput),
                torch.nn.ReLU(inplace=False),
                torch.nn.Conv2d(in_channels=intOutput, out_channels=intOutput, kernel_size=3, stride=1, padding=1),
                torch.nn.BatchNorm2d(intOutput),
                torch.nn.ReLU(inplace=False)
            )

        def Basic_(intInput, intOutput):
            return torch.nn.Sequential(
                torch.nn.Conv2d(in_channels=intInput, out_channels=intOutput, kernel_size=3, stride=1, padding=1),
                torch.nn.BatchNorm2d(intOutput),
                torch.nn.ReLU(inplace=False),
                torch.nn.Conv2d(in_channels=intOutput, out_channels=intOutput, kernel_size=3, stride=1, padding=1),
            )

I wonder why the Basic_ do not include BatchNorm2d and ReLU for the second convolution kernel.

Any suggestion is appreciated~

display abnormal region

Thanks for the perfect code.
I want to display the prediction errot,how can I display it?
thanks for any suggestion!

some questions about Evaluate

Traceback (most recent call last):
File "D:\Pycharm\PyCharm Community Edition 2021.3.2\plugins\python-ce\helpers\pydev\pydevd.py", line 1483, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "D:\Pycharm\PyCharm Community Edition 2021.3.2\plugins\python-ce\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "E:/huyi/MNAD/Evaluate.py", line 184, in
main()
File "E:/huyi/MNAD/Evaluate.py", line 142, in main
outputs, feas, updated_feas, m_items_test, softmax_score_query, softmax_score_memory, _, _, _, compactness_loss = model.forward(imgs[:, 0:3 * 4], m_items_test, False)
File "E:\huyi\MNAD\model\final_future_prediction_with_memory_spatial_sumonly_weight_ranking_top1.py", line 150, in forward
updated_fea, keys, softmax_score_query, softmax_score_memory,query, top1_keys, keys_ind, compactness_loss = self.memory(fea, keys, train)
File "D:\Anaconda\envs\MNAD\lib\site-packages\torch\nn\modules\module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "E:\huyi\MNAD\model\memory_final_spatial_sumonly_weight_ranking_top1.py", line 148, in forward
compactness_loss, query_re, top1_keys, keys_ind = self.gather_loss(query,keys, train)
File "E:\huyi\MNAD\model\memory_final_spatial_sumonly_weight_ranking_top1.py", line 215, in gather_loss
softmax_score_query, softmax_score_memory = self.get_score(keys, query)
File "E:\huyi\MNAD\model\memory_final_spatial_sumonly_weight_ranking_top1.py", line 120, in get_score
score = torch.matmul(query, torch.t(mem))# b X h X w X m
RuntimeError: cublas runtime error : the GPU program failed to execute at C:/w/1/s/tmp_conda_3.6_035809/conda/conda-bld/pytorch_1556683229598/work/aten/src/THC/THCBlas.cu:259

my environment is windows+A5000+python 3.6.2+pytorch 1.1.0
Could you tell me how I can solve it?

dataset path error

ValueError: num_samples should be a positive integer value, but got num_samples=0

In this code,the path is right,while the videos list can't be achieved by 'glob.glob(os.path.join(self.dir, '*'))' in model/utils.py.

I conducted the experiment according to the paper, but there is a gap between the experimental results and the paper. I hope to get your help.

Hello author:
I conducted a reconstruction experiment, but there was a gap between the results of the paper.
I set parameters as :
method recon;loss_compact 0.01;loss_separate 0.01;t_length 1;alpha 0.7;
th 0.015 .
Experimental AUC results : ped2 87. 355%; avenue 72.122%; shanghai 68.086% .
How can I achieve the expected experimental results. Thank you for your attention.

License

May I know what is the license for this code?

About the number of epoch for training on ShanghaiTech dataset.

Hi,
Thank you for your great word!
I try to reproduce the performance on the ShanghaiTech through the source code. The training settings are totally following the Implementation details. However, I only get AUC 67.8.
The questions are below:

  1. why can not I get AUC 72 on Shanghai Tech? My pytorch version is 1.1.0.
  2. The training data in ShanghaiTech are more than that in Ped2 or Avenue, but the training epoch num is set as 10. Is it enough for convergence?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.