shaoanlu / fewshot-face-translation-gan

Generative adversarial networks integrating modules from FUNIT and SPADE for face-swapping.

👋Hello, I'm Shao-An!

• 🔭 Working as a Control Software Engineer.
• 🌏 Grew up in Taiwan, residing in the vibrant city of Tokyo, Japan.
• ⭐ Interested in Control Systems, Optimization, and Deep Generative Models.
• 🌱 Currently engaged in the exploration of Software Architecture within Robotic Systems.

Recent Activity

1. 🎉 Merged PR #1 in shaoanlu/CBF_QP_safety_filter
2. 💪 Opened PR #1 in shaoanlu/CBF_QP_safety_filter
3. 💪 Opened PR #234 in qpsolvers/qpsolvers
4. 🎉 Merged PR #6 in shaoanlu/qpsolvers
5. 💪 Opened PR #6 in shaoanlu/qpsolvers

Projects

• Control and Optimization
  • Diffusion policy for Quadrotor Control: Showcases the application of cutting-edge diffusion models for imitation learning in quadrotor control.
  • Safety Filter for Collision Avoidance: Demonstrates the use of Control Barrier Functions (CBF) as a safety filter for collision avoidance.
  • Motion Planning with MPPI and CBF: Offers combinations of sampling based MPC with CBF for motion planning, showcased through jupyter notebooks.
• Deep Learning and Computer Vision
  • Few-Shot Face Translation: Features Generative Adversarial Networks trained on the VGGFace2 dataset for face-translation with few-shot examples.
  • Face Swapping GAN: Utilizes Generative Adversarial Networks for face-swapping in images.
  • Facial Analysis Toolbox: Provides a suite of deep learning models for facial analysis, adapted to the Keras framework.

fewshot-face-translation-gan's Issues

InvalidArgumentError: Shape must be rank 1 but is rank 0 for 'bn10.20053864458101422_5/cond/Reshape_4' (op: 'Reshape') with input shapes: [1,64,1,1], [].

manual load data:
code:
fn_src = 'source.jpg'
fns_tar = ['dest.jpg']

temp = []

for name in os.listdir('data_cnn/'):

name = 'data_cnn/' + name

temp.append(name)

fns_tar = temp

print(fn_src)
print(fns_tar)
out:
source.jpg
['dest.jpg']
then while using :
src, mask, aligned_im, (x0, y0, x1, y1), landmarks = utils.get_src_inputs(fn_src, fd, fp, idet)
tar, emb_tar = utils.get_tar_inputs(fns_tar, fd, fv)
occur error:
Building FAN for landmarks detection...

InvalidArgumentError Traceback (most recent call last)
~/anaconda2/envs/kjj/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs)
1627 try:
-> 1628 c_op = c_api.TF_FinishOperation(op_desc)
1629 except errors.InvalidArgumentError as e:

InvalidArgumentError: Shape must be rank 1 but is rank 0 for 'bn10.20053864458101422_5/cond/Reshape_4' (op: 'Reshape') with input shapes: [1,64,1,1], [].

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
in
----> 1 src, mask, aligned_im, (x0, y0, x1, y1), landmarks = utils.get_src_inputs(fn_src, fd, fp, idet)
2 tar, emb_tar = utils.get_tar_inputs(fns_tar, fd, fv)

~/kjj/deepfakes/fewshot-face-translation-GAN-master/utils/utils.py in get_src_inputs(fn, fd, fp, idet)
177 im = cv2.imread(fn)[..., ::-1]
178 im = auto_resize(im)
--> 179 (x0, y0, x1, y1), landmarks = detect_face(im, fd)
180 aligned_im = align_image(im, x0, y0, x1, y1, landmarks)
181

~/kjj/deepfakes/fewshot-face-translation-GAN-master/utils/utils.py in detect_face(im, fd, with_landmarks)
94 landmarks = None
95 if with_landmarks:
---> 96 bboxes, landmarks = fd.detect_face(im, with_landmarks=with_landmarks)
97 else:
98 bboxes = fd.detect_face(im, with_landmarks=with_landmarks)

~/kjj/deepfakes/fewshot-face-translation-GAN-master/face_toolbox_keras/models/detector/face_detector.py in detect_face(self, image, with_landmarks)
64 if self.lmd == None:
65 print("Building FAN for landmarks detection...")
---> 66 self.build_FAN()
67 print("Done.")
68 landmarks_list = []

~/kjj/deepfakes/fewshot-face-translation-GAN-master/face_toolbox_keras/models/detector/face_detector.py in build_FAN(self)
45
46 def build_FAN(self):
---> 47 self.lmd = FANLandmarksDetector(self.lmd_weights_path)
48
49 def detect_face(self, image, with_landmarks=True):

~/kjj/deepfakes/fewshot-face-translation-GAN-master/face_toolbox_keras/models/detector/landmarks_detector.py in init(self, path_to_weights_file)
20 def init(self, path_to_weights_file=FILE_PATH+"/FAN/2DFAN-4_keras.h5"):
21 if not tf.version >= '1.13':
---> 22 self.net = load_model(path_to_weights_file)
23 else:
24 self.net = tf.keras.models.load_model(path_to_weights_file)

~/anaconda2/envs/kjj/lib/python3.6/site-packages/keras/engine/saving.py in load_model(filepath, custom_objects, compile)

~/anaconda2/envs/kjj/lib/python3.6/site-packages/keras/engine/saving.py in _deserialize_model(f, custom_objects, compile)

~/anaconda2/envs/kjj/lib/python3.6/site-packages/keras/engine/saving.py in model_from_config(config, custom_objects)

~/anaconda2/envs/kjj/lib/python3.6/site-packages/keras/layers/init.py in deserialize(config, custom_objects)
53 module_objects=globs,
54 custom_objects=custom_objects,
---> 55 printable_module_name='layer')

~/anaconda2/envs/kjj/lib/python3.6/site-packages/keras/utils/generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
143 custom_objects=dict(list(_GLOBAL_CUSTOM_OBJECTS.items()) +
144 list(custom_objects.items())))
--> 145 with CustomObjectScope(custom_objects):
146 return cls.from_config(config['config'])
147 else:

~/anaconda2/envs/kjj/lib/python3.6/site-packages/keras/engine/network.py in from_config(cls, config, custom_objects)

InvalidArgumentError: You must feed a value for placeholder tensor 'input_14' with dtype float and shape [?,512] [[{{node input_14}}]]

InvalidArgumentError: You must feed a value for placeholder tensor 'input_14' with dtype float and shape [?,512]
[[{{node input_14}}]]
In colab notebook I get the following error after this cell out = model.inference(src, mask, tar, emb_tar)

Is the error related to image size or some thing else

I am using intel-tensorflow==1.13.2 because of the tensorflow.contrib error followed by the this error

How/where do I get the exe to run this on a windows pc?

May I know when will the training codes be released?

Hi,

Thank you very much for your fantastic work! I am wondering if you have any plan to release the training codes? so I can train it on my own datasets:)

No longer works on newest Colab version

Python version = 3.6
Tensorflow version = 2.5.0
Tensorflow 1 version = 1.15.2

ImportError: cannot import name 'keras_export'

I am trying to run this notebook, but I am getting below error at this line from models import FaceTranslationGANInferenceModel

Using TensorFlow backend.
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-13-e0f766e65a9c> in <module>()
----> 1 from models import FaceTranslationGANInferenceModel

13 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/initializers/initializers_v1.py in <module>()
     21 from tensorflow.python.framework import dtypes
     22 from tensorflow.python.ops import init_ops
---> 23 from tensorflow.python.util.tf_export import keras_export
     24 
     25 keras_export(v1=['keras.initializers.Zeros', 'keras.initializers.zeros'])(

ImportError: cannot import name 'keras_export'

Libraries: tensorflow-gpu==1.13.1, keras==2.2.4

weird results

i tried these two images but the result was like this

ipynb colab error

inputs are 256x256 already aligned faces

Building FAN for landmarks detection...
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

WARNING:tensorflow:No training configuration found in save file: the model was *not* compiled. Compile it manually.

WARNING:tensorflow:No training configuration found in save file: the model was *not* compiled. Compile it manually.

Done.

---------------------------------------------------------------------------

InvalidArgumentError                      Traceback (most recent call last)

<ipython-input-17-a57db805edd7> in <module>()
      1 src, mask, aligned_im, (x0, y0, x1, y1), landmarks = utils.get_src_inputs(fn_src, fd, fp, idet)
----> 2 tar, emb_tar = utils.get_tar_inputs(fns_tar, fd, fv)

6 frames

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in __call__(self, *args, **kwargs)
   1470         ret = tf_session.TF_SessionRunCallable(self._session._session,
   1471                                                self._handle, args,
-> 1472                                                run_metadata_ptr)
   1473         if run_metadata:
   1474           proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: input image must be of non-zero size
	 [[{{node model_5/lambda_39/ResizeBilinear}}]]
  (1) Invalid argument: input image must be of non-zero size
	 [[{{node model_5/lambda_39/ResizeBilinear}}]]
	 [[model_6/lambda_41/l2_normalize/_12935]]
0 successful operations.
0 derived errors ignored.

Training process

can you provide training pipeline, and how can you process on vgg2 data?

AttributeError: module 'tensorflow' has no attribute 'get_default_graph'

I tried but it gives above error

Bug in the function get_square_bbox

In the function utils/utils.py->def get_square_bbox(x0, x1, y0, y1, input_img)
The parameters should be in the order of x0, y0, x1, y1, such that it can match the call in x0, y0, x1, y1 = map(np.int32, get_square_bbox(x0, y0, x1, y1, im)).
Otherwise the cropped face will often lose part of the jaw.

Train

Greetings. I'd love to see the code for training. Is there any hope that it will be posted? :-)

donations?

How can we donate money to help fund development of this project?

thanks

tensorflow.python.framework.errors_impl.InvalidArgumentError: Default AvgPoolingOp only supports NHWC on device type CPU [[{{node 0.3562700616991372/AvgPool}}]]

tensorflow.python.framework.errors_impl.InvalidArgumentError: Default AvgPoolingOp only supports NHWC on device type CPU
[[{{node 0.3562700616991372/AvgPool}}]]

Minimum GPU memory requirement for running the notebook

What's the minimum GPU memory requirement to run this notebook (locally)

what's the purpose of the second discriminator?

hi, the first discriminator is for real/fake, what's the purpose of the second discriminator?

The trained model can only transform the special pair people?

i study the face swap recently. most papaer's model can only swap special peopel face. i want to konw it is the same?

Does this have a license?

Just wondering if I can use this legally

AttributeError: module 'tensorflow' has no attribute 'get_default_graph'

Preprocess for build_inceptionresnetv1

Just a reminder:
The current implementation of build_inceptionresnetv1 in dev branch seems problematic, althrough not finished yet, since the input of the FaceVerifier(extractor="facenet", classes=512) is assumed to be an image of 0255 uint8 values, while all the input images within models.py are preprocessed to be within -1+1.
The images should be rescaled back to 0~255 similar to the function build_hybrid_ir50s.

eyes jitter

eyes jitter was removed by simply using colored_parsing_map instead of parsing_map_with_iris returned from src_inputs function. was there any reason iris was being drawn on the segmentation map as i couldnt see any improvements?

ValueError: Dimension 0 in both shapes must be equal, but are 1024 and 512. Shapes are [1024,256] and [512,256]. for 'Assign_12' (op: 'Assign') with input shapes: [1024,256], [512,256].

This error still occurs at Keras==2.2.4.

output image resolution doesn't match input

It seems like the output image that's inferred is sometimes of lower resolution than the input.
For example, my input was 1280x720, but the output is 768x432.

It looks like the resizing is happening in utils.py/post_process_result, but I'm not sure why it's resizing - is it to keep the output face from being upscaled too much?

thank you!

Face size not not matching the photo

In the output image, you can see a square of new face overlapping the original image.
It seems that the size of the generated grid is not correct.

it is obvious if the input image resolution is high.

When would the training code be released?

What's A and B in training phase?

hi,since during training phase the input face A is a heavily blurred face，Is B the exact same picture as original A or just a face belonging to the same identity with A?

Shape must be rank 1 but is rank 0 for 'bn10.20053864458101422_1/cond/Reshape_4' (op: 'Reshape') with input shapes: [1,64,1,1], [].

Occurs when loading models for the first infernce.

fsgan

thoughts on fsgan and there method for face swapping, results look impressive https://nirkin.com/fsgan/

Dev colab notebook broken

Hello,

It seems that the notebook on the dev branch is broken, I got that error:

InvalidArgumentError: You must feed a value for placeholder tensor 'input_14' with dtype float and shape [?,512]
	 [[{{node input_14}}]]

in the cell with

out = model.inference(src, mask, tar, emb_tar)

Note that the notebook works with this commit:

!git checkout b379194340938a91a60ff0a43643b85aa8895da2

where can download this weight

older submodule of facetoolbox keras

I noticed the older version of facetoolbox keras is cloned into fewshot-face-translation-GAN
and not the updated version with mtcnn etc