Comments (23)
@ZHUANG-JLU
Hello,
Shortly I'm sharing a complete piece of code which'll help you perform face verification based on the DeepFace pre-trained model.
Sorry for the delay in reply.
from deepface.
Hello, thanks for your work first.
I am reading DeepFace, and I am quite confused about the 3D alignment step. To be specific, I cannot tell what is the affine 3D-to-2D camera P (what are the 8 unknowns mean?). Please help me. Thank you a lot.
from deepface.
Thank you so much for your sharing.
Indeed, I have 2 questions about the 3D-to-2D process.
First, what are the 8 unknows in the affine camera P mean? Location, gesture and lens?
Second, the author added the residual r into the 3d coordinates, but this operation can be done only if the coordinates systems are aligned, how could that be done?
I would check the resources you shared and try to re-implement the whole process. By the way, actually, in the modern network, the strong 3D alignment seems not to be a necessity. I am interested in the 3D alignment only because that I am working in the 3D reconstruction of face.
I would let you know when I solve the two questions above.
Thanks again.
from deepface.
You need to use the preprocess_image
function which is an essential part of the train and test data pipeline. In your case the function can be called in the following way.
# required imports
from tensorflow.keras.preprocessing.image import load_img, img_to_array
from deepface import create_deepface
from deepface import Dataset
model = create_deepface()
model.load_weights('VGGFace2_DeepFace_weights_val-0.9034.h5')
# fine tune the deepface model for your dataset
# for fine tuning, remove last dense layer only; (as per paper)
# set all pretrained parameters to non trainable
# add a dense layer with dim = num_classes (your dataset)
# set dense layer to trainable and train the model
img = load_img(path)
img = img.resize((152,152))
img_array = img_to_array(img)
img_array = img_array.reshape((-1, 152, 152, 3))
res = model.predict(img_array)
If there are any issues regarding the fine tuning step, please do let me know.
from deepface.
For training the DeepFace network, 2D aligned (2D frontalisation using 5 fiducial points similar to the one specified in paper) images from the VGGFace2 dataset have been used. It is expected that we use the same alignment technique (2D alignment of tight face crops) to make accurate predictions when using DeepFace feature vectors.
Shortly, will update the repository with the code required to obtain 2D aligned tight face crops from images in dataset. 2D-alignment in our case had been achieved using dlib.get_face_chips
method using shape_predictor_5_face_landmarks.
Currently, I'm working on the 3D frontalisation step to replicate the exact 3D alignment step as specified in paper. Hope to add include those changes and train the network using 3D frontalised images soon.
from deepface.
Thank you 👍
Now I have code like this:
from keras.preprocessing.image import load_img
from keras.preprocessing.image import img_to_array
from keras.preprocessing.image import array_to_img
import numpy as np
import dlib;
from keras_retinanet.utils.image import read_image_bgr, preprocess_image, resize_image
detector = dlib.get_frontal_face_detector()
sp = dlib.shape_predictor('shape_predictor_5_face_landmarks.dat')
def processdeepface(path):
img = dlib.load_rgb_image(path)
dets = detector(img, 1)
for k, d in enumerate(dets):
shape = sp(img, d)
face_chip = dlib.get_face_chip(img, shape, size=152)
image = preprocess_image(face_chip)
img_array = img_to_array(image)
img_array = img_array.reshape((1, IMAGE_SIZE[0], IMAGE_SIZE[1], 3))
res = model.predict(img_array)
Am I doing it in proper way?
Faces belonging to the same person should have the same output up? Am I correct?
Thanks
from deepface.
The code for the 2D face alignment seems fine.
The only limitation being that the model.predict
method can make accurate predictions only after the model has been fine tuned (last dense layer only) on your required dataset.
The original implementation (which actually represents the pretrained weights) makes use of a softmax classifier trained on 8631 classes from the VGGFace2 dataset. So, it is unlikely to make predictions the same class label unless the face crop being used belongs to one of those classes.
DeepFace doesn't use a triplet loss based implementation unlike FaceNet by Scroff et al. which can be directly used for a face verification task. Instead, the DeepFace feature vector obtained from the second last dense layer from the network can be effectively used to learn a face verification problem.
from deepface.
Thank you
Now I have code like this:
from keras.preprocessing.image import load_img from keras.preprocessing.image import img_to_array from keras.preprocessing.image import array_to_img import numpy as np import dlib; from keras_retinanet.utils.image import read_image_bgr, preprocess_image, resize_image detector = dlib.get_frontal_face_detector() sp = dlib.shape_predictor('shape_predictor_5_face_landmarks.dat') def processdeepface(path): img = dlib.load_rgb_image(path) dets = detector(img, 1) for k, d in enumerate(dets): shape = sp(img, d) face_chip = dlib.get_face_chip(img, shape, size=152) image = preprocess_image(face_chip) img_array = img_to_array(image) img_array = img_array.reshape((1, IMAGE_SIZE[0], IMAGE_SIZE[1], 3)) res = model.predict(img_array)Am I doing it in proper way?
Faces belonging to the same person should have the same output up? Am I correct?Thanks
@swghosh @lukkowal5 Hi,how can i process the 8631 output to a image? Thanks.
from deepface.
@ZHUANG-JLU,
What is the actual problem that you're motivated to solve?
Is it a face verification task (relating to matching a pair of images whether they belong to same person or not) or a face recognition task (relating to a multi class classification problem with some k number of people/identities in dataset)?
FYI, the last layer of the CNN i.e. the F8 layer is trained to classify samples of the VGGFace2 dataset in this case. The pre-final layer, F7 layer with 4096 dimensions is used to extract the DeepFace feature vector.
The simplest workaround for a face recognition problem is to replace the F8 layer with a new dense layer of dimension equal to number of classes in new dataset, and retrain the network for a few epochs (eg. 20) with all weights frozen upto F7. [some similar work to this is available here: FineTuning DeepFace on a smaller dataset.ipynb]
Please confirm about the same then I can get back to you with some code that can perform either based on your requirement.
from deepface.
@ZHUANG-JLU,
What is the actual problem that you're motivated to solve?Is it a face verification task (relating to matching a pair of images whether they belong to same person or not) or a face recognition task (relating to a multi class classification problem with some k number of people/identities in dataset)?
FYI, the last layer of the CNN i.e. the F8 layer is trained to classify samples of the VGGFace2 dataset in this case. The pre-final layer, F7 layer with 4096 dimensions is used to extract the DeepFace feature vector.
The simplest workaround for a face recognition problem is to replace the F8 layer with a new dense layer of dimension equal to number of classes in new dataset, and retrain the network for a few epochs (eg. 20) with all weights frozen upto F7. [some similar work to this is available here: FineTuning DeepFace on a smaller dataset.ipynb]Please confirm about the same then I can get back to you with some code that can perform either based on your requirement.
Face Detection/Recognition/Alignments.Thanks for your reply.
from deepface.
@ZHUANG-JLU,
Did you look into this code?
FineTuning DeepFace on a smaller dataset.ipynb
from deepface.
@ZHUANG-JLU,
Did you look into this code?
FineTuning DeepFace on a smaller dataset.ipynb
I will see it.Thanks.
from deepface.
Hey there, I am new in this field can you tell me how to run this code.
from deepface.
Also is there any way I can run it on my custom dataset.
from deepface.
@ZHUANG-JLU,
Did you look into this code?
FineTuning DeepFace on a smaller dataset.ipynb
Hi @thepranaygaur,
Can you refer to the notebook provided in the link to run the face recognition model on a custom dataset?
Assuming that you've a directory containing pictures of faces of people (tight face crops, aligned) segregated into sub directories basis their class labels / identity of the person, you can use the notebook for end to end code.
Do let me know if you have any queries.
For face detection and alignment please use dlib.get_frontal_face_detector
and dlib.get_face_chips
respectively as given in #1 (comment)
from deepface.
Hi @FunkyKoki, thanks for reaching out!
In my repository, the DeepFace model which have been trained on VGGFace2 dataset uses tight face crops with 2D alignment only (atleast, as of now) using dlib.get_face_chips
(works reasonably well under various poses, except poses that lack frontal face which is the reason I had remove from training the face pictures that lacked frontal poses). I went through the 3D alignment process as discussed in the original paper but failed to replicate the work primarily due to lack of publicly available 3D face datasets and level of difficulty associated with it.
The methodology discussed uses 3D to 2D mapping of face keypoints with the help of an affine camera (not really sure how that can be done with regular 2D face crops) and a trained 3D face model. As a starting point, I'd recommend you to get started with these resources which should help you develop more clarity on this topic.
- Viewing Real-World Faces in 3D (https://talhassner.github.io/home/publication/2013_ICCV)
- face-frontalization repo by dougsouza (https://github.com/dougsouza/face-frontalization) - this is a Python port of the original MATLAB implementation of the paper Effective Face Frontalization in Unconstrained Images (https://talhassner.github.io/home/publication/2015_CVPR_1)
Aforementioned, includes work done by Tal Hassner who has relevant work in this field (also, cited in DeepFace paper). - Introductory details about camera models and 3D affine cameras (https://www-users.cs.umn.edu/~hspark/CSci5980/CameraModel1.pdf)
Let me know if you are willing to collaborate with me on this at a suitable time / looking forward to contributing some code that is related to 3D face frontalization and related model training. Will be more than happy to be of any help.
Good luck!
from deepface.
@swghosh Hello, I'm trying to use https://github.com/serengil/deepface deepFace facial attribute analysis, but the performance toward certain age group is bad. So I'm looking for a way to train/ tune the age/gender/race classifier with my own dataset. I'm wondering if this is possible to do? I saw this repo is using VGG face for pretrain weight. Can I retrain with my dataset? May you give me some insight? Thank you!
from deepface.
Hey @zzzzmoya,
That sounds like an interesting concept. Indeed, the original VGG-Face2 dataset focuses on a detailed study about various facial attributes. I'm not sure about what serengil/deepface offers with regard to Facial Attribute Analysis (but it does use the weights from my repo for what they call as FB-DeepFace
). I'm skeptical that the original residual architecture mentioned (if I remember correctly) in the VGGFace2 paper should help you with get a head start on your work. And, I'd assume that the VGGFace2 dataset is inclusive enough irrespective of age groups so you can train your own model with it and later tune it on your dataset. Feel free to ping me if there's related additional discussion to be done on this regard (preferably, by creating a separate issue).
Unfortunately a no to using those weights for facial attribute analysis as I'm not sure if it's a great fit with the existing weights ATM. (as the weights are classification task specific; also I'm not sure about the effects of using the DeepFace architecture for such tasks)
I saw this repo is using VGG face for pretrain weight. Can I retrain with my dataset? May you give me some insight?
from deepface.
Hey @swghosh, awesome post! I have a question:
Could the classification task be repurposed to say facial analysis? Eg: Using the same data hierarchical structure, what if instead of matching (one to many) subjects, the labels would be race-ethnicity identification where folders would be labeled into a limited number of categories?
From your reply to @ZHUANG-JLU, I'm thinking of creating a new F8 layer with the number of categories for my task and train the model first as a classifier (i.e. training just the F8 layer) using VGGFace2 pre-trained weights and then fine-tuning setting a portion of the DeepFace model to trainable. Curious to know your thoughts? Would you by any chance have any sample code for creating an additional F8 layer? Thanks!
from deepface.
Hey @swghosh, awesome post! I have a question:
Could the classification task be repurposed to say facial analysis? Eg: Using the same data hierarchical structure, what if instead of matching (one to many) subjects, the labels would be race-ethnicity identification where folders would be labeled into a limited number of categories?
From your reply to @ZHUANG-JLU, I'm thinking of creating a new F8 layer with the number of categories for my task and train the model first as a classifier (i.e. training just the F8 layer) using VGGFace2 pre-trained weights and then fine-tuning setting a portion of the DeepFace model to trainable. Curious to know your thoughts? Would you by any chance have any sample code for creating an additional F8 layer? Thanks!
-
I am certain that the output of F7 layer can be used as features for other facial related tasks as well, Facial Analysis in your case. Yes, there are quite a lot of chances that adding a new layer on top F7 or even experimenting the same with the output of L6 and then stacking custom Dense layers should help you fine-tune the model on a new facial task with new categories.
In case you want a model that performs generic facial analysis, you should be able to do that using a MTCNN (multi-task CNN) which would provide for multi-label classification as well as regression tasks under a common end-to-end neural network.
For just race/ethnicity classification, I'd hope that output of features from either F7 or L6 (can try experimenting with either: F7 preferred) should suffice given there is one or more Dense layer(s) stacked on top (subject to hyper-parameter tuning and experiments). -
Would this notebook not be sufficient for you to guide you on training a new face model with new classification categories? (in your cases, categories would be race/ethnicity labels; everything else code-wise should be the same).
FineTuning DeepFace on a smaller dataset.ipynb
You could also refer to any resources where generic fine-tuning tasks using Keras models have been described for that purpose. Should be pretty easy to load the weights and just remove the last layer to use your own.
As an extra, if you'd like to fine-tune the complete model (inclusive of Conv layers, Locally-connected layers) but please note that it is extremely computationally expensive and would either need a GPU cluster or a TPU. (Google Colab provides for free TPUs which should suffice)
from deepface.
hi ,
i used FineTuning DeepFace on a smaller dataset.ipynb with my dataset but i can't get good accuracy and it give me just 10%
thnx for help
from deepface.
Hi @saidafef: Apologies for the delayed response.
You could probably check for any red flags in any of the following aspects:
- Pre-processing is same as that used in training or not
- The size of the dataset and number of samples per class should be reasonable as the DeepFace feature classifier would perform only as good as the number of samples it is trained on for each class (unlike one-shot face verification tasks like FaceNet)
- Occlusions / necessary variations among your training and validation datasets
- Learning rate can be tuned significantly along with dropout value in range 0.2-0.5 for some performance improvements
- Full fine tuning with a very low exponentially decayed learning rate? (only you have cumbersome resources available because training the locally connected layers is too much resource hungry)
From my personal standpoint I have been able to achieve 95+% accuracy on the test by training it on a custom face dataset created from scratch, with very different set of occlusions on the train and testing sets. (training set had very less variety in terms of occlusions/protrusions whereas the test set had a lot of them and were quite different in terms of lighting as well; suprisingly the features transfer learned from DeepFace were able to capture correlation in facial structure giving good results even on a very different test set)
Feel free to ping with any additional details.
from deepface.
Hi @swghosh , I am looking for a notebook of face verification problem (given two images identify if they are of the same individual or of different people). Do you have a similar implementation like this (https://colab.research.google.com/drive/1otl-4tJaCfyCJF33bvnjzeuUdldcWysA) for my case. We want to fine tune the model to identify images of same people in a given dataset.
from deepface.
Related Issues (10)
- LFW accuracy HOT 1
- Can not load the finetuned model in keras HOT 1
- Masked face HOT 1
- Dataset Preparation and Training on GPU HOT 22
- Face Verification using Pre-Trained Model HOT 4
- Problem with TFRecords Preparation HOT 2
- Weights are not of the same shape, "Shapes are incompatible" error HOT 3
- Error in realtime recogniton HOT 1
- Triplet Loss HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepface.