yuvalnirkin / face_segmentation Goto Github PK

Hello Yuval,

Thank you for your project. I am also working on face swap projects. Now I am trying to tackle the problem of face occlusions.

I find the performance on the README page really amazing. However, after installation I find that the performance on my face-swap training images not as good as your demo image. The model is unable to segment eye-glasses.

Here are some examples:

I ran the model by this command:

face_seg_image my_image.jpg -o . -m ../data/face_seg_fcn8s.caffemodel -d ../data/face_seg_fcn8s_deploy.prototxt

where my_image.jpg is a 256x256 image in which the face is aligned via 5 landmarks (eyes, nose and mouth).

Do you have any ideas on this? Or is there anything wrong with my input format?

Thank you in advance.

Hi,
I met this error when running the demo
I0921 13:13:41.511903 1711 net.cpp:242] This network produces output score
I0921 13:13:41.511961 1711 net.cpp:255] Network initialization done.
I0921 13:13:43.045377 1711 net.cpp:744] Ignoring source layer data
I0921 13:13:43.045660 1711 net.cpp:744] Ignoring source layer data_data_0_split
I0921 13:13:43.154325 1711 net.cpp:744] Ignoring source layer loss
Segmentation fault (core dumped)

I run in CPU mode, using a computer with 32 GB RAM. I think that is related to Caffe.
Did someone meet this error.
Thank you!

How to organize data

hi,in your face_seg.py ,u write :in_ -= np.array((104.00698793,116.66876762,122.67891434))
can u explain the meaning of three numbers in the brackets
thanks a lot~

When compile this the cmake file found caffe folder properly, but shows below error.
Does it only supports caffe without gpu?

device_alternate.hpp:34:23: fatal error: cublas_v2.h: No such file or directory

Great work. I've enjoyed the results and experimenting with your concept. Appreciate the effort you made towards sharing your code and pipeline.

I saw a reference to a new and improved Unet version in PyTorch? How goes the experiments with the better version?

Hi,

When trying to run tour code, I got the following error:

F0613 23:16:31.623750 22135 common.cpp:66] Cannot use GPU in CPU-only Caffe: check mode.

Is it possible to use this code without the GPU?

Best,
Michal

I tried the python caffe inference. It took 4 seconds to detect an image using CPU.
Will the C++ inference be much faster than caffe?

Thanks

Installation is not clear. Could you update the README with more thorough installation instructions?

Hello, I am interested in your work and want to train my own face segmentation network.
Could you release your database for face segmentation?
Thank you very mach!

Greetings,

This may be off topic, but I'm just trying to find some help.

I'm trying to get 300 FCN model to run in my xCode project. I'm converting .coffemodel to .mlmodel with coremltools:

coreml_model = coremltools.converters.caffe.convert(caffe_model, image_input_names='data', is_bgr = True, red_bias = -104, blue_bias = -123, green_bias = -117, image_scale = 1)

As far as I understand the input image is in BGR color space, with above mentioned biases.
After conversions when I read the model description with coremltools:

input { name: "data" type { imageType { width: 300 height: 300 colorSpace: BGR } } } output { name: "score" type { multiArrayType { dataType: DOUBLE } } } metadata { userDefined { key: "coremltoolsVersion" value: "3.3" } }

The output has no shapes.

When I add the model to Xcode project, I run the model by passing CVPixelBuffer as input

let input = buffer(from: userSelectedImage_UI) guard let prediction = try? model.prediction(data: input!) else { return }

the output of the model is MultyArray.

let output = prediction.score

How can I convert it to CVPixelBuffer, if there're no shapes.

I've tried using MultiArray converters to no avail, the output is just black image

I've tried this and this methods.

If anybody knows how to get this working in CoreML I'd really appreciate it

I coded the model in lasagne and changed the weights of the model accordingly but I am not getting good results. There is too much of noise in the results. Below is an example showing the original image and the corresponding segmentation mask.

Any idea why is this happening and where the issue might be?

I'm trying to understand the output of the caffe network and where argmax prediction is coming from in the python interface: out = net.blobs['score'].data[0].argmax(axis=0)

The output: net.blobs['score'].data has 21 channels. What are the additional 20 channels and how do we know that if a face pixel is present, the maximum value will be in the second channel?

Thanks

I am getting the following error while trying to install face-segmentation.

face_segmentation/build$ make -j8
Scanning dependencies of target face_seg
[ 14%] Building CXX object face_seg/CMakeFiles/face_seg.dir/face_seg.cpp.o
[ 28%] Building CXX object face_seg/CMakeFiles/face_seg.dir/utilities.cpp.o
In file included from /usr/include/cublas_v2.h:65:0,
                 from /usr/include/caffe/util/device_alternate.hpp:34,
                 from /usr/include/caffe/common.hpp:22,
                 from /usr/include/caffe/blob.hpp:8,
                 from /usr/include/caffe/caffe.hpp:7,
                 from /home/deyrahul/Research/3DMM/face_segmentation/face_seg/face_seg/face_seg.h:11,
                 from /home/deyrahul/Research/3DMM/face_segmentation/face_seg/face_seg.cpp:1:
/usr/include/cublas_api.h:72:10: fatal error: driver_types.h: No such file or directory
 #include "driver_types.h"
          ^~~~~~~~~~~~~~~~
compilation terminated.
face_seg/CMakeFiles/face_seg.dir/build.make:82: recipe for target 'face_seg/CMakeFiles/face_seg.dir/face_seg.cpp.o' failed
make[2]: *** [face_seg/CMakeFiles/face_seg.dir/face_seg.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
CMakeFiles/Makefile2:197: recipe for target 'face_seg/CMakeFiles/face_seg.dir/all' failed
make[1]: *** [face_seg/CMakeFiles/face_seg.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2

Ubuntu: 18.04
CUDA: 10.1
Caffe: 1.0
Boost: 1.72
OpenCV: 4.2

My CMake-GUI 3.17.1 produces the following configuration for it:

Any help would be appreciated! Thanks.

Hi,

Thanks for the great work, many years later it is in fact still difficult to find this particular functionality and quality elsewhere.
Unfortunately, Caffe was never a tame dependency and this is not improving, to the point that it is harder to get it to work nicely with other dependencies as it is to rewrite full setups from scratch.

Fast forward to this moment, I've ported the model to PyTorch. Hopefully it can be helpful for others:
https://github.com/andres-fr/face_segmentation_pytorch

I'm happy to receive any feedback. Cheers!
Andres

where is your U-Net models? and the train and inference code of Pytorch?

I had a hard time getting Caffe installed, so I figured I'd try out your model using openCV's dnn.readNetFromCaffe() along with your .caffemodel and .prototxt.

The output of net.forward() when using this method is a 1x21x300x300 matrix which can be squeezed to 21x300x300. Each of the 21 300x300 arrays, when normalized, seem to constitute a type of heat map. Some of these can be seen below.

My question is, how would I combine these to get the actual face segmentation? I tried to parse your code to see if I could figure it out but fell short of understanding. Thanks!

Hi, Thanks for this project. It works good for the front faces. However, for some scenarios results are not so good.

• if person in the image is far from camera(no segmentation at all)
• Side faces (bad segmentation)

Few suggestions or update to this paper will be great help. Thanks!

Hello. I want to use your model to segment my own face. But i have no idea about the pricinple of face bbox cropping. I failed to segment face when i used face bbox which is to tight. So i expand the face by some ratios. However, the result is not good enough as well.

Can u tell me how to choose the most suitable face to achieve your performance? Thank you.

could this output an alpha mask?

Can u release Unet model，thanks。

Hi
Is it possible to have your training dataset?
I would like to train from scratch for a new model.

Best regards,

Hello, thanks for your wonderfull work! What is the license for the pre-trained model? And could we use it for a commercial application?

Hi,

Thanks for the software. When running "face_seg_image", here is an error:

WARNING: Logging before InitGoogleLogging() is written to STDERR
F0212 19:18:55.035207 21461 common.cpp:66] Cannot use GPU in CPU-only Caffe: check mode.
*** Check failure stack trace: ***

Is there an option in the compiling stage to specify to use Caffe CPU?

Thanks!

Hi, I tried your system - and maybe i am doing something wrong, but this system seems to give rather poor results. How were you able to get the results you describe?
I am using your implementation directly, so it should be same code i am running as you.

See example of poor result here:

Maybe you used some special model which you have not shared?

Hi, I got a error when I try to compile.

/home/migudm/caffe/build/include/caffe/proto/caffe.pb.h:17:2: error: #error This file was generated by an older version of protoc which is incompatible with your Protocol Buffer headers. Please regenerate this file with a newer version of protoc.

There are two versions of protobuf(2.5 and 3.5.1) in my Ubuntu14. caffe is complied by protoc 2.5
I try to change the default protoc to version2.5. but it failed again.

shall I recomplie the Caffe with protoc 3.5.1 ?
I have looked for many similar problems, but still not solved

I come across the following question. Anyone can help me.
//usr/lib/libblas.so.3: undefined reference to `gotoblas'
collect2: error: ld returned 1 exit status
face_seg_image/CMakeFiles/face_seg_image.dir/build.make:151: recipe for target 'face_seg_image/face_seg_image' failed
make[2]: *** [face_seg_image/face_seg_image] Error 1
CMakeFiles/Makefile2:140: recipe for target 'face_seg_image/CMakeFiles/face_seg_image.dir/all' failed
make[1]: *** [face_seg_image/CMakeFiles/face_seg_image.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

when I run

python face_seg.py

I get an error as No module surgery

line number 4 in face_seg.py is

 import surgery

But there is no module as surgery any where on my system
from where can i get the surgery module

I am interested in your work and want to train my own face segmentation network. Could you release your database for face segmentation? Thanks a lot!