wangjk666 / pydeepfakedet

PyDeepFakeDet is an integrated and scalable tool for Deepfake detection.

PyDeepFakeDet

An integrated and scalable library for Deepfake detection research.

Introduction

PyDeepFakeDet is an integrated and scalable Deepfake detection tool developed by Fudan Vision and Learning Lab. The goal is to provide state-of-the-art Deepfake detection Models as well as interfaces for the training and evaluation of new Models on commonly used Deepfake datasets.

This repository includes implementations of both CNN-based and Transformer-based methods:

• CNN Models

  • ResNet
  • Xception
  • EfficientNet
  • MesoNet
  • GramNet
  • Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues
  • Multi-attentional Deepfake Detection

• Transformer Models

  • ViT
  • M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection

Model Zoo and Baselines

The baseline Models on three versions of FF-DF dataset are provided.

Method	RAW	C23	C40	Model
ResNet50	97.61	94.87	84.95	RAW / C23 / C40
Xception	97.84	95.24	86.27	RAW / C23 / C40
EfficientNet-b4	97.89	95.61	87.12	RAW / C23 / C40
Meso4	85.14	77.14	60.13	RAW / C23 / C40
MesoInception4	95.45	84.13	71.31	RAW / C23 / C40
GramNet	97.65	95.16	86.21	RAW / C23 / C40
F3Net	99.95	97.52	90.43	RAW / C23 / C40
MAT	97.90	95.59	87.06	RAW / C23 / C40
ViT	96.72	93.45	82.97	RAW / C23 / C40
M2TR	99.50	97.93	92.89	RAW / C23 / C40

The baseline Models on Celeb-DF is also available.

Method	Celeb-DF	Model
ResNet50	98.51	CelebDF
Xception	99.05	CelebDF
EfficientNet-b4	99.44	CelebDF
Meso4	73.04	CelebDF
MesoInception4	75.87	CelebDF
GramNet	98.67	CelebDF
F3Net	96.47	CelebDF
MAT	99.02	CelebDF
ViT	96.73	CelebDF
M2TR	99.76	CelebDF

Installation

• We use Python == 3.9.0, torch==1.11.0, torchvision==1.12.0.

• Install the required packages by:

  pip install -r requirements.txt

Data Preparation

Please follow the instructions in DATASET.md to prepare the data.

Quick Start

Specify the path of your local dataset in ./configs/resnet50.yaml, and then run:

python run.py --cfg resnet50.yaml

Visualization tools

Please refer to VISUALIZE.md for detailed instructions.

Contributors

PyDeepFakeDet is written and maintained by Wenhao Ouyang, Chao Zhang, Zhenxin Li, and Junke Wang.

License

PyDeepFakeDet is released under the MIT license.

Citations

@inproceedings{wang2021m2tr,
  title={M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection},
  author={Wang, Junke and Wu, Zuxuan and Ouyang, Wenhao and Han, Xintong and Chen, Jingjing and Lim, Ser-Nam and Jiang, Yu-Gang},
  booktitle={ICMR},
  year={2022}
}

pydeepfakedet's Issues

Parameters and FLOPs

Thank you very much for providing this codebase, which is very useful. Can you provide the parameters and FLOPs of these models?

'EfficientNet' object has no attribute 'extract_textures'

Hey , I compared the code with your other repository "m2tr", I found that your deleted the def extract_textures in efficientnet.py, and this caused the code to report an error —— 'EfficientNet' object has no attribute 'extract_textures'

about the reproduced results

Hi,

Thanks for your excellent summary.
I have reproduced the related results on the dataset "raw" of FF-DF.
But I got the results as shown:


The overall values are on the high side, where meso4, MesoInception4 is not the correct one.

PS: My dataset where the original is real, manipulated is fake, where fake and real each have 1000 videos * 128 frames.
I follow your default setup of the code for extracting frames from the video.
Is it normal? Or is there something I ignore?

thanks~

Training stuck for MAT

when I training MAT, The training is stuck, and it's not over. I checked a lot, but I didn't find the relevant information.

CPU shows running, Do you know what the problem is?

RuntimeError: CUDA error: invalid device ordinal

RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Excuse me, how to solve this error.

可视化中，如果我的模型是MAT，我该如何构造--layer

您好，我在可视化图片时遇到了一些问题，我不知道--layers后面应该构造什么层，请问一下这个该怎么解决啊？

Number of frames during testing

Hi,

For FFDF dataset, how many frame images do you extract from per video during testing?

RuntimeError: Error(s) in loading state_dict for M2TR

We tried to load the corresponding weights for the model but found them to be inconsistent, including c23 and c40. We don't understand why, it looks like the keys are wrong.

RuntimeError: Error(s) in loading state_dict for M2TR:
Missing key(s) in state_dict: "layers.2.0.t.attention.query_embedding.weight", "layers.2.0.t.attention.query_embedding.bias", "layers.2.0.t.attention.value_embedding.weight", "layers.2.0.t.attention.value_embedding.bias", "layers.2.0.t.attention.key_embedding.weight", "layers.2.0.t.attention.key_embedding.bias", "layers.2.0.t.attention.output_linear.0.weight", "layers.2.0.t.attention.output_linear.0.bias", "layers.2.0.t.attention.output_linear.1.weight", "layers.2.0.t.attention.output_linear.1.bias", "layers.2.0.t.attention.output_linear.1.running_mean", "layers.2.0.t.attention.output_linear.1.running_var", "layers.2.0.t.feed_forward.conv.0.weight", "layers.2.0.t.feed_forward.conv.0.bias", "layers.2.0.t.feed_forward.conv.1.weight", "layers.2.0.t.feed_forward.conv.1.bias", "layers.2.0.t.feed_forward.conv.1.running_mean", "layers.2.0.t.feed_forward.conv.1.running_var", "layers.2.0.t.feed_forward.conv.3.weight", "layers.2.0.t.feed_forward.conv.3.bias", "layers.2.0.t.feed_forward.conv.4.weight", "layers.2.0.t.feed_forward.conv.4.bias", "layers.2.0.t.feed_forward.conv.4.running_mean", "layers.2.0.t.feed_forward.conv.4.running_var", "layers.2.1.filter.complex_weight", "layers.2.1.feed_forward.conv.0.weight", "layers.2.1.feed_forward.conv.0.bias", "layers.2.1.feed_forward.conv.1.weight", "layers.2.1.feed_forward.conv.1.bias", "layers.2.1.feed_forward.conv.1.running_mean", "layers.2.1.feed_forward.conv.1.running_var", "layers.2.1.feed_forward.conv.3.weight", "layers.2.1.feed_forward.conv.3.bias", "layers.2.1.feed_forward.conv.4.weight", "layers.2.1.feed_forward.conv.4.bias", "layers.2.1.feed_forward.conv.4.running_mean", "layers.2.1.feed_forward.conv.4.running_var", "layers.2.2.conv1.weight", "layers.2.2.conv1.bias", "layers.2.2.conv2.weight", "layers.2.2.conv2.bias", "layers.2.2.conv3.weight", "layers.2.2.conv3.bias", "layers.2.2.conv4.0.weight", "layers.2.2.conv4.0.bias", "layers.2.2.conv4.1.weight", "layers.2.2.conv4.1.bias", "layers.2.2.conv4.1.running_mean", "layers.2.2.conv4.1.running_var", "layers.3.0.t.attention.query_embedding.weight", "layers.3.0.t.attention.query_embedding.bias", "layers.3.0.t.attention.value_embedding.weight", "layers.3.0.t.attention.value_embedding.bias", "layers.3.0.t.attention.key_embedding.weight", "layers.3.0.t.attention.key_embedding.bias", "layers.3.0.t.attention.output_linear.0.weight", "layers.3.0.t.attention.output_linear.0.bias", "layers.3.0.t.attention.output_linear.1.weight", "layers.3.0.t.attention.output_linear.1.bias", "layers.3.0.t.attention.output_linear.1.running_mean", "layers.3.0.t.attention.output_linear.1.running_var", "layers.3.0.t.feed_forward.conv.0.weight", "layers.3.0.t.feed_forward.conv.0.bias", "layers.3.0.t.feed_forward.conv.1.weight", "layers.3.0.t.feed_forward.conv.1.bias", "layers.3.0.t.feed_forward.conv.1.running_mean", "layers.3.0.t.feed_forward.conv.1.running_var", "layers.3.0.t.feed_forward.conv.3.weight", "layers.3.0.t.feed_forward.conv.3.bias", "layers.3.0.t.feed_forward.conv.4.weight", "layers.3.0.t.feed_forward.conv.4.bias", "layers.3.0.t.feed_forward.conv.4.running_mean", "layers.3.0.t.feed_forward.conv.4.running_var", "layers.3.1.filter.complex_weight", "layers.3.1.feed_forward.conv.0.weight", "layers.3.1.feed_forward.conv.0.bias", "layers.3.1.feed_forward.conv.1.weight", "layers.3.1.feed_forward.conv.1.bias", "layers.3.1.feed_forward.conv.1.running_mean", "layers.3.1.feed_forward.conv.1.running_var", "layers.3.1.feed_forward.conv.3.weight", "layers.3.1.feed_forward.conv.3.bias", "layers.3.1.feed_forward.conv.4.weight", "layers.3.1.feed_forward.conv.4.bias", "layers.3.1.feed_forward.conv.4.running_mean", "layers.3.1.feed_forward.conv.4.running_var", "layers.3.2.conv1.weight", "layers.3.2.conv1.bias", "layers.3.2.conv2.weight", "layers.3.2.conv2.bias", "layers.3.2.conv3.weight", "layers.3.2.conv3.bias", "layers.3.2.conv4.0.weight", "layers.3.2.conv4.0.bias", "layers.3.2.conv4.1.weight", "layers.3.2.conv4.1.bias", "layers.3.2.conv4.1.running_mean", "layers.3.2.conv4.1.running_var", "classifier.projection.weight", "classifier.projection.bias".
Unexpected key(s) in state_dict: "classifier.weight", "classifier.bias".
size mismatch for model._fc.weight: copying a param with shape torch.Size([1, 1792]) from checkpoint, the shape in current model is torch.Size([2, 1792]).
size mismatch for model._fc.bias: copying a param with shape torch.Size([1]) from checkpoint, the shape in current model is torch.Size([2]).

Hope this can be fixed soon.

Duplicate code in efficientnet

Duplicate code in efficientnet at line 530

how to use multiple gpu, can u show example ?

@wangjk666 greetings, please show example for multi gpu setting in same cluster. Single Machine with 8 GPUs I have.

Some wrong about the output is through two softmax?

PyDeepFakeDet/tools/test.py

Line 45 in 631d781

preds = F.softmax(outputs['logits'], dim=1)[:, 1]

in this line, the output feature is by softmax again.
In the f3net，

PyDeepFakeDet/PyDeepFakeDet/models/f3net.py

Line 44 in 631d781

f = F.softmax(f, dim=-1)

Duplicate code in efficientnet

Duplicate 530 lines in efficientnet