dariusaf / mesonet Goto Github PK
View Code? Open in Web Editor NEW"MesoNet: a Compact Facial Video Forgery Detection Network" (D. Afchar, V. Nozick) - IEEE WIFS 2018
License: Apache License 2.0
"MesoNet: a Compact Facial Video Forgery Detection Network" (D. Afchar, V. Nozick) - IEEE WIFS 2018
License: Apache License 2.0
Hi @DariusAf
Can you share with us code how did you train your models Meso4.
Thanks.
Hi, I can't open the download data set link in the description, could you please share me an [email protected]
I am receiving an error when running this project with the pretrained weights.
The message I receive is as follows....
2021-01-11 13:47:20.220685: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open weights/Meso4_DF: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
Is there any solution to this problem?
I am running the ./example.py script on a virtualenv Python 3.8.7 environment.
Below are the packages that are installed in the virtualenv:
absl-py==0.11.0
astunparse==1.6.3
cachetools==4.2.0
certifi==2020.12.5
chardet==4.0.0
click==7.1.2
cmake==3.18.4.post1
cycler==0.10.0
dlib==19.21.1
face-recognition==1.3.0
face-recognition-models==0.3.0
flatbuffers==1.12
gast==0.3.3
google-auth==1.24.0
google-auth-oauthlib==0.4.2
google-pasta==0.2.0
grpcio==1.32.0
h5py==2.10.0
idna==2.10
imageio==2.9.0
imageio-ffmpeg==0.4.3
Keras==2.4.3
Keras-Preprocessing==1.1.2
kiwisolver==1.3.1
Markdown==3.3.3
matplotlib==3.3.3
numpy==1.19.5
oauthlib==3.1.0
opt-einsum==3.3.0
Pillow==8.1.0
protobuf==3.14.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
python-dateutil==2.8.1
PyYAML==5.3.1
requests==2.25.1
requests-oauthlib==1.3.0
rsa==4.7
scipy==1.6.0
six==1.15.0
tensorboard==2.4.0
tensorboard-plugin-wit==1.7.0
tensorflow==2.4.0
tensorflow-estimator==2.4.0
termcolor==1.1.0
typing-extensions==3.7.4.3
urllib3==1.26.2
Werkzeug==1.0.1
wrapt==1.12.1
Can someone explain why I am receiving this error? It appears to be related to tensorflow not able to read the pre-trained sets supplied with the project.
Thank you for any help!
Hi, I can't access the dataset from the link you have mentioned. It would be great if you could just check the link for any discrepancies or provide any alternate way for me to access your dataset.
Thanks!
Hi, thanks for this work, I wonder if you can tell me how to get the datasets you used, I an really interested in this work and want to try something by myself....
Thanks!
Hi, after downloading the dataset, I saw 2 folders named: train::test and validation. I know that the dataset usually must contain 3 parts: train-set, valid-set and test-set.
I wonder how u use those 2 folders in training model and evaluate model.
Hi,
where is the training code? If I want to train the code on my dataset
The output which you shown is very far different from what we are getting,
Could u please show us the implementation process of yours in a video.
As we have seen, There is an interface ,But we are not getting that
We are just getting the values as below
Dealing with video Tom Cruise VIRAL Deepfakes on TikTok Causing Security Concerns.mp4
Face extraction warning : 0 - no face
Face extraction warning : 304 - no face
Face extraction warning : 608 - found face in full frame [(207, 669, 251, 624)]
Face extraction warning : 912 - found face in full frame [(141, 636, 234, 543)]
Face extraction warning : 1824 - no face
Face extraction warning : 2128 - no face
Face extraction warning : 2432 - found face in full frame [(382, 723, 516, 589)]
Face extraction warning : 3344 - found face in full frame [(277, 585, 303, 559)]
Face extraction warning : 3648 - no face
Face extraction warning : 3952 - no face
Face extraction warning : 4256 - found face in full frame [(159, 679, 293, 545)]
Face extraction warning : 4560 - no face
Face extraction warning : 4864 - no face
Face extraction warning : 5168 - found face in full frame [(367, 708, 501, 575)]
Face extraction warning : 6384 - found face in full frame [(139, 878, 472, 545)]
Face extraction warning : 6688 - found face in full frame [(165, 660, 218, 606)]
Face extraction warning : 7296 - found face in full frame [(145, 738, 256, 627)]
Face extraction warning : 7600 - found face in full frame [(387, 761, 548, 601)]
Face extraction warning : 7904 - no face
Face extraction warning : 8208 - found face in full frame [(251, 718, 296, 674)]
Face extraction warning : 8512 - found face in full frame [(382, 753, 516, 619)]
Face extraction report of not_found : 21
Face extraction report of no_face : 9
Predicting Tom Cruise VIRAL Deepfakes on TikTok Causing Security Concerns.mp4
Hi Darius,
I am currently working on a Deepfake-related project. Would you be able to share the Deepfake dataset you collected? I have sent you an email.
Using TensorFlow backend.
Traceback (most recent call last):
File "/home/aimannnhakim/PycharmProjects/project1/example.py", line 2, in
from classifiers import *
File "/home/aimannnhakim/PycharmProjects/project1/classifiers.py", line 3, in
from keras.models import Model as KerasModel
File "/home/aimannnhakim/PycharmProjects/MesoNet/venv/lib/python3.5/site-packages/keras/init.py", line 3, in
from . import utils
File "/home/aimannnhakim/PycharmProjects/MesoNet/venv/lib/python3.5/site-packages/keras/utils/init.py", line 6, in
from . import conv_utils
File "/home/aimannnhakim/PycharmProjects/MesoNet/venv/lib/python3.5/site-packages/keras/utils/conv_utils.py", line 9, in
from .. import backend as K
File "/home/aimannnhakim/PycharmProjects/MesoNet/venv/lib/python3.5/site-packages/keras/backend/init.py", line 81, in
from .tensorflow_backend import *
File "/home/aimannnhakim/PycharmProjects/MesoNet/venv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 514
"""Returns whether x
is a placeholder.
^
SyntaxError: invalid syntax
Hello, I am a student of master. I work for my research project deepfake detection. I wanted to test your code with other deepfake dataset. I used your code Meso4 and MesoInception to test other deepfake dataset. It doesn't work as better as writed in the papier. I think it's problem of weight. Can I know what kind of method that you used to get your weight, please. Did you use backpropagation to get your weight and save them as file ?
I have tested your code, but I find it doesn't appear like what you claimed.
Maybe I have something wrong in test method.
If you want my test dataset, you can leave you email or send the email to [email protected]
I am very interested in the project you are working on, but lack training data. I want to ask you for the data set. Thank you
I am facing issues while loading weights from Meso4_DF.h5 file. Can you help me figure it out this error?
AttributeError Traceback (most recent call last)
/var/folders/_d/3lw9778n4x507_np6pxqkgsc0000gn/T/ipykernel_11257/4180714191.py in
1 # Instantiate a MesoNet model with pretrained weights
2 classifier = Meso4()
----> 3 classifier.load('./weights/Meso4_DF.h5')
/var/folders/_d/3lw9778n4x507_np6pxqkgsc0000gn/T/ipykernel_11257/2096412517.py in load(self, path)
15
16 def load(self, path):
---> 17 self.model.load_weights(path)
~/opt/anaconda3/envs/tf/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in load_weights(self, filepath, by_name)
179 raise ValueError('Load weights is not yet supported with TPUStrategy '
180 'with steps_per_run greater than 1.')
--> 181 return super(Model, self).load_weights(filepath, by_name)
182
183 @trackable.no_automatic_dependency_tracking
~/opt/anaconda3/envs/tf/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py in load_weights(self, filepath, by_name)
1175 saving.load_weights_from_hdf5_group_by_name(f, self.layers)
1176 else:
-> 1177 saving.load_weights_from_hdf5_group(f, self.layers)
1178
1179 def _updated_config(self):
~/opt/anaconda3/envs/tf/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py in load_weights_from_hdf5_group(f, layers)
649 """
650 if 'keras_version' in f.attrs:
--> 651 original_keras_version = f.attrs['keras_version'].decode('utf8')
652 else:
653 original_keras_version = '1'
AttributeError: 'str' object has no attribute 'decode'
For the training, how was the data loaded? I am especially interested in the part where the architecture works for 2D images so if the pre-processed version of Face2Face is having 300 frames per video(assume), then how is the network tackling that. Are you stacking so many images together one on top of other to create a sort of "voxel"? Any help shall be highly appreciated.
Hi,
it seems that your implementation has a leaky ReLU between two fully connected layers in both Meso4 and MesoInception4.
But figure 4 in your paper does not have a leaky ReLU.
Why the difference?
Thanks
I've tried to run the example but it seems your weights are too old and cannot be used with the TensorFlow version(1.13.1), Keras 2.3.1.
Cuda 10.0.130
Cudnn 7.3.1
Using TensorFlow backend.
/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tensorflow:From /home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2020-02-27 16:42:29.428823: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-02-27 16:42:29.435453: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-27 16:42:29.436706: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x2bc55c0 executing computations on platform CUDA. Devices:
2020-02-27 16:42:29.436720: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce RTX 2060 SUPER, Compute Capability 7.5
2020-02-27 16:42:29.471700: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3696000000 Hz
2020-02-27 16:42:29.472057: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x3289690 executing computations on platform Host. Devices:
2020-02-27 16:42:29.472070: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2020-02-27 16:42:29.472140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce RTX 2060 SUPER major: 7 minor: 5 memoryClockRate(GHz): 1.71
pciBusID: 0000:01:00.0
totalMemory: 7.79GiB freeMemory: 7.12GiB
2020-02-27 16:42:29.472150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2020-02-27 16:42:29.472992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-27 16:42:29.473004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2020-02-27 16:42:29.473008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2020-02-27 16:42:29.473054: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6927 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
Found 4 images belonging to 2 classes.
2020-02-27 16:42:31.830396: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2020-02-27 16:42:33.494656: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-02-27 16:42:33.500889: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
File "/home/irfan/PycharmProjects/MesoNet/example.py", line 25, in <module>
print('Predicted :', classifier.predict(X), '\nReal class :', y)
File "/home/irfan/PycharmProjects/MesoNet/classifiers.py", line 14, in predict
return self.model.predict(x)
File "/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/keras/engine/training.py", line 1462, in predict
callbacks=callbacks)
File "/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/keras/engine/training_arrays.py", line 324, in predict_loop
batch_outs = f(ins_batch)
File "/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3076, in __call__
run_metadata=self.run_metadata)
File "/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1439, in __call__
run_metadata_ptr)
File "/home/irfan/Documents/Upwork/.upwork/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node conv2d_1/convolution}}]]
Process finished with exit code 1
I am trying to run example.py on a video of my own.
The only modification made by me is for the path to the folder with the videos :
Line 31 in 61fe5cd
I have the same python and keras versions as mentioned in the readme.
The error trace:
Using TensorFlow backend.
2019-05-03 14:32:26.071187: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Found 4 images belonging to 2 classes.
Predicted : [[0.99782073]]
Real class : [1.]
Dealing with video fadg0-fram1-roi93.mov
Traceback (most recent call last):
File "/home/aneesh/Work/MesoNet/example.py", line 31, in <module>
predictions = compute_accuracy(classifier, '/home/aneesh/Work/data/deepfaketimit/DeepfakeTIMIT/')
File "/home/aneesh/Work/MesoNet/pipeline.py", line 305, in compute_accuracy
skipstep = max(floor(face_finder.length / frame_subsample_count), 0)
OverflowError: cannot convert float infinity to integer
Fatal Python error: could not acquire lock for <_io.BufferedReader name=8> at interpreter shutdown, possibly due to daemon threads
Thread 0x00007f08e77fe700 (most recent call first):
File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/site-packages/imageio_ffmpeg/_parsing.py", line 61 in run
File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/threading.py", line 884 in _bootstrap
Current thread 0x00007f0949657740 (most recent call first):
File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/subprocess.py", line 1557 in _communicate
File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/subprocess.py", line 863 in communicate
File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/site-packages/imageio_ffmpeg/_io.py", line 193 in read_frames
File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/site-packages/imageio/plugins/ffmpeg.py", line 342 in _close
File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/site-packages/imageio/core/format.py", line 252 in close
File "/home/aneesh/SoftwareInstall/miniconda3/envs/lab_env/lib/python3.6/site-packages/imageio/core/format.py", line 241 in __del__
I've tried debugging it. It seems to be stemming from
Line 305 in 61fe5cd
where face_finder.length is returning infinity.
I found a potential solution here : https://stackoverflow.com/questions/54778001/how-to-to-tackle-overflowerror-cannot-convert-float-infinity-to-integer
I may have fixed it in my fork :
https://github.com/aneesh-joshi/MesoNet/blob/19749db10561901b0622009d5266046bc967bb10/pipeline.py#L25
But on running it with this "fix", I get 100% confidence of fakeness in real and fake videos. I suspect I'm doing something wrong.
Could you help me out @DariusAf
Thanks!
I'm trying to detect fake video and it does not work. Here is the video I'm using for fake detection
https://youtu.be/r1jng79a5xc
Here is output
Dealing with video Jennifer Lawrence-Buscemi on her favorite housewives [Deepfake].mp4
Face extraction warning : 0 - found face in full frame [(173, 797, 334, 636)]
Face extraction warning : 1444 - found face in full frame [(156, 690, 316, 529)]
Face extraction report of not_found : 2
Face extraction report of no_face : 0
Predicting Jennifer Lawrence-Buscemi on her favorite housewives [Deepfake].mp4
2020-03-13 12:18:34.535467: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 86507520 exceeds 10% of system memory.
2020-03-13 12:18:34.554189: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 86507520 exceeds 10% of system memory.
2020-03-13 12:18:34.586315: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 86507520 exceeds 10% of system memory.
`Jennifer Lawrence-Buscemi on her favorite housewives [Deepfake]` video class prediction : 1.0
Hi, @DariusAf,
I'm trying to replicate the results of this model but can't seem to get the accuracy to be anywhere near the numbers claimed in the paper. I'm using the dataset provided in the readme.
I believe that my training code is correct and there's nothing wrong with it. The only "novel" thing in the code is a custom implementation of a learning rate schedule which decreases the learning rate by an order of magnitude every 1000 iterations, as stated in the paper.
Still, my validation accuracy is stuck at somewhere around ~75% and doesn't budge from that point.
Can you help me, please?
Hi, how can I present the result in table 4 in your paper? Your download link does not obtain face2face dataset!Thanks!
Hi,
nice paper!
Are you planing to push pipeline code? Also, these links for datasets aren't working.
Thanks,
Aleksandar
Are your deepfake images (from training dataset) generated using autoencoders? Currently GAN (Generative adversarial networks) and diffusion model (for example DDPM) are also popular to generate deepfake images/videos. How accurate do you think your model is when detecting GAN or diffusion model generated deepfakes?
Many thanks!
Can you provide one example which has used this for training a mesonet model. I have found PyTorch implementation, but need a Keras one.
the issue is "batch = np.concatenate((batch, np.expand_dims(self.resize_patch(patch), axis = 0)),axis = 0)
ValueError: all the input array dimensions except for the concatenation axis must match exactly" in pipeline.py when running example.py, I print the shape of batch is (1, 256, 256, 3),but the shape of np.expand_dims(self.resize_patch(patch), axis = 0) isn't (1, 256, 256, 3). How should I modify it?
Dear Darius:
I am a undergraduate from Beijing Univeraity of Chemical Technology,China, Currenty I want to begin my personal research in Deepfake Detection because of my interest,I really admire to your algorithm in this paper and want to run your code,so would you pleased sent me your dataset?I would appreciate it if you send me your dataset!!My email aderss is [[email protected]].Thank you very much!BEST WISHES!
Hi @DariusAf
It would be a good idea to create a operation steps for your model. For beginner like me, I really don't know where to start.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.