polariszhao / pfld-pytorch

PFLD pytorch Implementation

PFLD-pytorch

Implementation of PFLD A Practical Facial Landmark Detector by pytorch.

1. install requirements

pip3 install -r requirements.txt

2. Datasets

• WFLW Dataset Download

​ Wider Facial Landmarks in-the-wild (WFLW) is a new proposed face dataset. It contains 10000 faces (7500 for training and 2500 for testing) with 98 fully manual annotated landmarks.

1. WFLW Training and Testing images [Google Drive] [Baidu Drive]
2. WFLW Face Annotations
3. Unzip above two packages and put them on ./data/WFLW/
4. move Mirror98.txt to WFLW/WFLW_annotations

$ cd data 
$ python3 SetPreparation.py

3. training & testing

training :

$ python3 train.py

use tensorboard, open a new terminal

$ tensorboard  --logdir=./checkpoint/tensorboard/

testing:

$ python3 test.py

4. results:

5. pytorch -> onnx -> ncnn

Pytorch -> onnx

python3 pytorch2onnx.py

onnx -> ncnn

how to build :https://github.com/Tencent/ncnn/wiki/how-to-build

cd ncnn/build/tools/onnx
./onnx2ncnn pfld-sim.onnx pfld-sim.param pfld-sim.bin

Now you can use pfld-sim.param and pfld-sim.bin in ncnn:

ncnn::Net pfld;
pfld.load_param("path/to/pfld-sim.param");
pfld.load_model("path/to/pfld-sim.bin");

cv::Mat img = cv::imread(imagepath, 1);
ncnn::Mat in = ncnn::Mat::from_pixels_resize(img.data, ncnn::Mat::PIXEL_BGR, img.cols, img.rows, 112, 112);
const float norm_vals[3] = {1/255.f, 1/255.f, 1/255.f};
in.substract_mean_normalize(0, norm_vals);

ncnn::Extractor ex = pfld.create_extractor();
ex.input("input_1", in);
ncnn::Mat out;
ex.extract("415", out);

6. reference:

PFLD: A Practical Facial Landmark Detector https://arxiv.org/pdf/1902.10859.pdf

Tensorflow Implementation: https://github.com/guoqiangqi/PFLD

pfld-pytorch's Issues

运行pytorch2onnx.py报错

您好，运行pytorch2onnx.py报错了，

错误输出一大段内容，最后是：
====> check onnx model...
段错误 (核心已转储)

不知是什么原因，求大佬help

训练模型

不好意思，打扰了，请问运行train.py文件时，是不是会保存很多个check_point_epoch_x.pth.tar文件啊？会一直保存到多少个之后会停止train，或者说训练要大概要训练多久才结束呢？谢谢您！

你好！WFLW_annotations 文件缺失，且下载不了，可以提供百度网盘下载？

PFLD 0.25X 版本的模型有吗

你好，请问有PFLD 0.25X 版本的模型？

euler angles

Hello!

PFLD-pytorch/pfld/utils.py

Line 29 in e438e2b

# X-Y-Z with X pointing forward and Y on the left and Z up.

here it is said about axes, which were used as basis for euler angle calculation, but it is not clear for me as
on

PFLD-pytorch/pfld/utils.py

Line 32 in e438e2b

# X points to the right, Y down, Z to the front

it is said about other basis, so what X, Y, Z were used in dataset for calculation of euler angle?

测试效果差（距离镜头近）。

感谢分享代码。我用自己拍摄的人脸图，通过MTCNN检测人脸，再进行pfld关键点预测，测试效果非常不好。造成这个的原因是pfld模型设计本身的原因？还是训练数据集和测试数据差别太大(训练数据集都是一些裁剪出来的低分辨率小图[112*112]，测试的时候是用手机拍摄的高分辨率大图)？通过对这种自己拍摄的高清大图进行手动的人脸裁剪再送入pfld进行关键点检测可以一定程度上提高测试效果，但是针对不同的图没有一个相同的裁剪标准。。。。。想问一下在人脸检测的基础上抠人脸图应遵循一个什么标准。

PFLD was trained using 3D Annotation and learning is not happening

I used 3D tools to annotate the dataset for 98 landmarks. Attaching the sample images annotated using 3D Tool.

But when I did Transfer Learning on the same for 350 Epochs, model is not able to learnt. Kindly guide me for the same where things went wrong

Images annotated with 98 landmarks using 3D Tool:

欧拉角检测不准， angle为检测，angle_gt为真实数据

angle = tensor([[-0.0873, 0.7688, -0.2588]], device='cuda:0')
angle_gt = tensor([[ 88.0343, -60.7853, -62.0728]], device='cuda:0')
angle = tensor([[-0.1598, 0.5773, -0.4115]], device='cuda:0')
angle_gt = tensor([[ 66.4287, -53.0196, -49.0985]], device='cuda:0')
angle = tensor([[-0.0510, 0.7076, -0.3719]], device='cuda:0')
angle_gt = tensor([[ 48.0916, -45.6873, -17.9782]], device='cuda:0')
angle = tensor([[ 0.0888, 0.7797, -0.3578]], device='cuda:0')
angle_gt = tensor([[ 54.0094, -37.7794, -10.2092]], device='cuda:0')
angle = tensor([[-0.0731, 0.8048, -0.0279]], device='cuda:0')
angle_gt = tensor([[15.3467, 3.7500, 10.6802]], device='cuda:0')
angle = tensor([[ 0.0906, 0.6620, -0.2479]], device='cuda:0')
angle_gt = tensor([[ 79.5670, -63.5313, -39.0085]], device='cuda:0')

Missing file

Hello in your test.py
you have
parser.add_argument('--test_dataset', default='./data/test_data/list.txt', type=str)
however, what is the list.txt here and could you give us a format to test the WLFWDatasets/
Thanks

tensorboard

Hello how does your tensorboard work? I have trained as required, and the corresponding files are also generated in checkpoints, but when viewing tensorboard, there is no output on the web page. This page isn’t working

Number of parameters of PFLD

The paper says that PFLD 1X and PFLD 0.25X has 12.5 Mb and 2.1 Mb, respectively.
As far as I confirmed with thop.profile and torchsummary.summary, the number of parameters of PFLD 1X's seems 1.26M, about ten times smaller than 12.5Mb.

What's `12.5Mb' meaning?

I confirmed it with the following code:

import numpy as np
import torch
from torchvision.models import MobileNetV2
from thop import profile
from torchsummary import summary

from models.pfld import PFLDInference

# models.
# MobileNetV2's are the references.
pfld_backbone=PFLDInference()
mobilenetv2 = MobileNetV2()
mobilenetv2_025 = MobileNetV2(width_mult=0.25)

# dummy input.
inputs = torch.randn([1,3,112,112])

names = ['PFLD', 'MobileNetV2', 'MobileNetV2_wm-0.25']
nets = [pfld_backbone, mobilenetv2, mobilenetv2_025]

# to Mega.
denom = np.array((1e+6,)*2)

# profiling.
for name, net in zip(names, nets):
    rlt=profile(net, inputs=(inputs,))
    print('{0}: {1[1]:.2f}M'.format(name, np.array(rlt)/denom))

#for name, net in zip(names, nets):
#    print(name)
#    summary(net, (3,112, 112))

Thanks in advance

conv2 is wrong?

class PFLDInference(nn.Module):
def init(self):
super(PFLDInference, self).init()

    self.conv1 = nn.Conv2d(
        3, 64, kernel_size=3, stride=2, padding=1, bias=False)
    self.bn1 = nn.BatchNorm2d(64)
    self.relu = nn.ReLU(inplace=True)

    self.conv2 = nn.Conv2d(
        64, 64, kernel_size=3, stride=1, padding=1, bias=False)
    self.bn2 = nn.BatchNorm2d(64)
    self.relu = nn.ReLU(inplace=True)

@polarisZhao I think conv2 is a depth-wise convolution instead of a normal convolution, right?

在train.py文件中， pic should be PIL Image or ndarray. Got <class 'NoneType>该如何解决这个问题呢

wrong loss penalize？

i.e:
#200: 姿态(pose) 0
#201: 表情(expression) 0
#202: 照度(illumination) 0
#203: 化妆(make-up) 0
#204: 遮挡(occlusion) 0
#205: 模糊(blur) 0

the cleaner sample, i.e. without attributes, the loss = 0 ？？？ why???

Testing in another data set than WFLW .

How to use SetPreparation.py to test on 300w dataset?

replace retinaface got wrong result

Hi, I simply replaced face detector, and the landmark generate is wrong:

 [ 931.92762136 2339.20769691]
 [ 510.39959192 2297.27704525]
 [  63.24823201 2284.217906  ]
 [ 494.01687384 2427.17235088]
 [ 744.55595613 1609.01716948]
 [ 909.48493481 1217.38911867]
 [ 603.90601158 1840.64791203]
 [-129.06481326 2212.34843731]
 [-133.53530467 1858.27581882]
 [ 815.31933546 1050.98218918]
 [ 681.86050057 1324.7084856 ]
 [ 932.11687803 1675.40012598]
 [ 549.24564958 1647.93741703]
 [ 837.87542582 1662.80833483]
 [ 870.4960227  1671.6031909 ]
 [ 825.64744949 2499.12979603]
 [ 105.63829914 2475.21629333]
 [ 282.52048194 1844.15559769]
 [ 818.49040389  491.52097106]
 [1190.8842802  1193.24022532]]

these coordinates generated were quit large.

and the visualized result is not very right as well:

these landmarks were at wrong place.

cuda版本过低问题，训练需要多少显存

请问cuda版本需要到多少？目前我是cuda9.0，用gtx1060 6G训练模型，所需的显存足够吗？

bad quality

网络前几层的bias为何置为False

rt

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 188 and 156 in dimension 2 at /pytorch/aten/src/TH/generic/THTensor.cpp:689

您好，非常感谢你的工作。
我在运行train.py的时候遇到了上面这个问题。具体的错误如下：Traceback (most recent call last):

File "train.py", line 232, in
main(args)
File "train.py", line 173, in main
criterion, epoch)
File "train.py", line 85, in validate
for img, landmark_gt, attribute_gt, euler_angle_gt in wlfw_val_dataloader:
File "/home//anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 819, in next
return self._process_data(data)
File "/home//anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/home//anaconda3/lib/python3.6/site-packages/torch/_utils.py", line 369, in reraise
raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/anaconda3/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/home/anaconda3/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/hom/anaconda3/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 80, in default_collate
return [default_collate(samples) for samples in transposed]
File "/home/anaconda3/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 80, in
return [default_collate(samples) for samples in transposed]
File "/home/anaconda3/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate
return torch.stack(batch, 0, out=out)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 188 and 156 in dimension 2 at /pytorch/aten/src/TH/generic/THTensor.cpp:689

找不到权重初始化的位置？

你好，请问权重初始化的位置在哪，没有找到，默认权重初始化都为0吗

提升精度问题

请问PFLD-pytorch目前可以复现论文中给出的精度吗？在WFLW上能够达到什么精度和速度？

core dump

thank you for your code ,when i download this code and try to run camera.py ,some error happend, first ,my torch version is 1.1.0, it say volatile was moved ,use torch.no_grad() instead,second ,Illegal instruction (core dumped), do you have this question, how to fix it? thank you

请问，网络输入修改为224后，会提升多少呢？

run train.py failed

@polarisZhao 非常感谢您分享的代码，在运行训练脚本的时候loss部分报错了，这个怎么解决呢？

Should be x3 = x3.view(x3.size(0), -1) or it is deliberately written like this

PFLD-pytorch/models/pfld.py

Line 128 in a98635b

x3 = x3.view(x1.size(0), -1)

pretrained model

can you please share the pretrained model?

calculate_pitch_yaw_roll

so damn hard to understand....
能分享一下资料么，这个函数好难懂...

你好，你在300W数据训练之后，在其测试集上的结果可以达到该论文的结果吗？

how to train on cpu

hi!
I have no nvidia gpu on my computer .When I run the script train.py the error shows:
"AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from"
How can I train on cpu?

调用模型

300w

How to run SetPreparation.py for 300w dataset?

Training: how much GPU memory do you need/

I try to train on NV2080 (8G), but failed with out of memory.
What's your training environment? What GPU do you use?

RuntimeError: CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 7.76 GiB total capacity; 6.88 GiB already allocated; 7.56 MiB free; 34.09 MiB cached)

跑python test.py报错

跑python test.py报错：

RuntimeError: Given groups=1, weight of size [64, 3, 3, 3], expected input[1, 112, 112, 3] to have 3 channels, but got 112 channels instead

关于训练数据不一致性的处理

您好：
想咨询下，原文章作者用wflw预训练，然后在300w上训练；而wflw有98点标注数据，且含有光照等信息，但是300w只有68点标注数据，训练数据的不一致，导致模型输出维度也不同，想咨询下作者这样在wlfw上训练的模型怎么迁移到300w数据集上呢？

模型精度问题

感谢你的分享和工作！
问题：我自己训练出来的基于ALFW98点数据集的模型相比你给的模型，错误率高了10个点，不知道是哪里有问题，万分感谢。

datasets 中的一些数据增强方法好像没有使用吗？

请问flip， channel_shuffle， random_noise， random_brightness，random_contrast，random_saturation 等 为什么没有加入训练呢？

wrong loss penalize?

original paper:

For instance,if disabling the geometry and data imbalance functionalities, our loss degenerates to a simple`2loss

your code:

        mat_ratio = torch.mean(attributes_w_n, axis=0)
        mat_ratio = torch.Tensor([
            1 / (x*256) if x > 0 else 1 for x in mat_ratio
        ]).cuda()
        weight_attribute = torch.sum(attributes_w_n.mul(mat_ratio), axis=1)

        l2_distant = torch.sum((landmark_gt - landmarks) * (landmark_gt - landmarks), axis=1)
        return torch.mean(weight_angle * weight_attribute * l2_distant), torch.mean(l2_distant)

1 / (x*256) if x > 0 else 1 for x in mat_ratio

you are decreasing weight_attribute if the geometry and data imbalance functionalities are present, but should be vice versa

文件缺失的问题

找不到‘./data/test_data/list.txt’文件

Interpretation of the output

When I use the model on a cropped face( 112*112), it return

tensor([[ 2.5031, 0.5400, 2.5610, 1.6902, 2.3800, 1.4249, 2.3219, 1.9674,
2.2124, 2.0817, 1.9422, 2.3128, 1.8053, 2.5855, 1.4570, 2.7046,
1.0818, 3.6230, 0.6297, 3.7358, 0.9336, 3.7999, 0.6649, 4.2318,
0.8000, 4.3501, 0.9931, 5.3472, 0.8499, 5.6622, 0.9833, 5.7424,
0.6650, 6.2639, 1.5096, 6.6354, 1.8387, 6.0361, 2.4721, 5.5128,
3.3405, 5.9895, 3.7352, 5.5592, 4.0576, 5.5659, 4.9748, 5.4990,
4.6798, 5.3513, 5.2311, 4.9756, 5.6050, 4.5588, 5.4349, 4.1934,
5.5895, 3.8537, 5.6141, 3.2810, 5.7145, 2.4732, 6.3659, 2.4489,
5.7713, 1.4750, 1.7614, 0.7001, 1.7819, -0.0296, 1.6259, -0.0539,
1.9667, 0.1593, 2.0979, 0.6584, 2.1205, 0.5665, 1.8736, 0.6672,
2.0189, 0.6153, 1.7041, 0.2538, 1.9706, 0.5217, 2.1157, -0.1328,
3.3149, 0.1556, 3.4920, 0.5049, 4.1756, 1.3516, 3.9482, 0.8055,
2.9670, 0.5373, 1.9500, 0.2579, 1.6205, 0.7916, 1.7664, 0.8393,
1.6058, 1.0465, 0.9469, 0.4592, 1.0506, 0.5595, 0.8714, 1.6369,
1.1845, 1.7528, 1.4024, 1.7032, 1.5809, 1.6675, 2.1324, 2.1407,
1.5833, 0.9698, 1.3089, 0.5519, 1.7143, 0.8796, 1.6684, 0.9314,
2.1470, 1.0451, 2.0533, 1.2146, 1.7021, 1.2716, 1.0598, 1.2469,
2.6481, 1.2449, 2.3563, 0.7741, 2.9258, 0.9934, 3.3385, 1.0512,
2.9415, 1.2636, 2.4886, 1.6280, 2.8286, 0.8264, 2.2652, 1.6024,
0.7494, 2.4899, 0.8105, 2.0038, 1.2357, 1.9055, 1.4521, 1.6275,
1.1433, 1.3466, 2.2189, 2.1250, 2.5340, 2.9909, 2.0008, 2.6108,
1.4500, 3.5718, 0.6595, 3.2810, 0.6810, 3.4444, 0.3239, 2.8738,
0.3464, 2.4799, 0.8839, 2.2725, 1.8056, 2.2162, 2.5020, 1.7966,
2.4657, 2.7388, 1.7206, 2.9695, 0.7993, 3.0288, 0.4674, 3.1466,
1.3650, 0.8892, 2.6473, 1.1290]], grad_fn=)

What's that mean? That's no the coordinate, right? And how can I draw dot on a picture by those results?
Thanks

模型训练时输入是BGR，使用camera时输入是RGB？

训练时读入图片使用opencv，所以训练使用bgr，这是否会对结果造成影响？（我觉得不会）

401 Unauthorized

WFLW Face Annotations 下载点击后提示需要输入用户名密码 请问怎么解决

Unclosed parenthesis in train.py

this line has unclosed parenthesis.

300W data set There is a problem with the test set and training set on the 300W data set

The training set does not contain IBUG, and IBUG belongs to the validation set.

RuntimeError: ./checkpoint/snapshot/checkpoint.pth.tar is a zip archive (did you mean to use torch.jit.load()?)

您好，请问您使用的torch的版本是多少，我用的是1.2.0+cpu
我在load模型的时候报了上述错误，请问您知道该怎么解决吗？

你好，pytorch转onnx是check失败，使用onnxsim进行简化也失败了，请问是什么原因？

numpy=1.18.3
torch=1.2.0
torchvision=0.4.0

在CPU上测试很慢

直接测试每秒大概5帧左右，有什么建议和指导吗？

PFLD 0.25X implementation

I'd like to try out PFLD 0.25X performance, but it's not provided.
I made some changes on __init__() of pfld.py as follows:

class PFLDInference(nn.Module):
    def __init__(self, width_mult=1.0):
        super(PFLDInference, self).__init__()
        assert width_mult in [0.25, 1.0]
        layer_channels=int(64*width_mult)
        layer_channels2=layer_channels*2
        self.conv1 = nn.Conv2d(
            3, layer_channels, kernel_size=3, stride=2, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(layer_channels)
        self.relu = nn.ReLU(inplace=True)

        self.conv2 = nn.Conv2d(
            layer_channels, layer_channels, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(layer_channels)
        self.relu = nn.ReLU(inplace=True)

        self.conv3_1 = InvertedResidual(layer_channels, layer_channels, 2, False, 2)

        self.block3_2 = InvertedResidual(layer_channels, layer_channels, 1, True, 2)
        self.block3_3 = InvertedResidual(layer_channels, layer_channels, 1, True, 2)
        self.block3_4 = InvertedResidual(layer_channels, layer_channels, 1, True, 2)
        self.block3_5 = InvertedResidual(layer_channels, layer_channels, 1, True, 2)

        self.conv4_1 = InvertedResidual(layer_channels, layer_channels2, 2, False, 2)

        self.conv5_1 = InvertedResidual(layer_channels2, layer_channels2, 1, False, 4)
        self.block5_2 = InvertedResidual(layer_channels2, layer_channels2, 1, True, 4)
        self.block5_3 = InvertedResidual(layer_channels2, layer_channels2, 1, True, 4)
        self.block5_4 = InvertedResidual(layer_channels2, layer_channels2, 1, True, 4)
        self.block5_5 = InvertedResidual(layer_channels2, layer_channels2, 1, True, 4)
        self.block5_6 = InvertedResidual(layer_channels2, layer_channels2, 1, True, 4)

        self.conv6_1 = InvertedResidual(layer_channels2, 16, 1, False, 2)  # [16, 14, 14]

        self.conv7 = conv_bn(16, 32, 3, 2)  # [32, 7, 7]
        self.conv8 = nn.Conv2d(32, 128, 7, 1, 0)  # [128, 1, 1]
        self.bn8 = nn.BatchNorm2d(128)

        self.avg_pool1 = nn.AvgPool2d(14)
        self.avg_pool2 = nn.AvgPool2d(7)
        self.fc = nn.Linear(176, 196)

I'm thinking that I can get PFLD 0.25X with width_mult=0.25.
Is this correct?

Thanks in advance.

annotations文件夹下的.txt文件具体作用是什么?

感谢分享，我想请问一下annotations文件夹下的三个.txt文件（list_68pt_rect_attr_test.txt，list_68pt_rect_attr_train.txt和Mirror.txt）具体作用是什么?如果使用自己的数据集该如何更改这块？

执行pytorch2onnx.py时报错

您好，我在执行pytorch2onnx.py这个文件的时候报错：
====> check onnx model...
====> Simplifying...
Traceback (most recent call last):
File "pytorch2onnx.py", line 46, in
onnx.save(model_opt, args.onnx_model_sim)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\onnx_init_.py", line 184, in save_model
proto = write_external_data_tensors(proto, basepath)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\onnx\external_data_helper.py", line 225, in write_external_data_tensors
for tensor in _get_all_tensors(model):
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\onnx\external_data_helper.py", line 170, in _get_initializer_tensors
for initializer in onnx_model_proto.graph.initializer:
AttributeError: 'tuple' object has no attribute 'graph'
问题应该是Simplifying的时候出错了，应该怎么解决呢？