lesliezhoa / headswap

This is a HeadSwap project not only face

Hi there 👋

I am Leslie from China, nice to meet you all!

• 😄 Interst: computer vision especially Image generation
  
  
• 📫 How to reach me: send email to [email protected]

headswap's Issues

Improve color and style transfer from the source

I tried to swap between a source image with blonde hair and pale skin, and a target with brown hair and tanned skin. The outcome has the shape of the source image but the color of the underlying target image (brown hair instead of blonde hair).

This is similar to one of your examples: https://github.com/LeslieZhoa/HeadSwap/blob/main/assets/1-e6de879a-f3c2-47f8-9588-feab95df7b9e.png
The source image has blonde hair but the output has black hair like the target.

I'm hoping for improvements that allows me to choose from keeping the color of the target or using the color from source. Would this require further model training? If this does, you might release improved code and models since I don't have the hardware to train anything yet.

public again!!!

Welcome

public again again!!!!

Pretrained models missing file needed for BFM

After I set up /pretrained_models/BFM folder and run inference.py I got this error:

Traceback (most recent call last):
  File "C:\Users\User\anaconda3\envs\simswap\lib\site-packages\scipy\io\matlab\mio.py", line 39, in _open_file
    return open(file_like, mode), True
FileNotFoundError: [Errno 2] No such file or directory: 'pretrained_models/BFM\\similarity_Lm3D_all.mat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "inference.py", line 172, in <module>
    'pretrained_models/BFM')
  File "inference.py", line 20, in __init__
    Process.__init__(self,params_path,bfm_folder)
  File "D:\HeadSwap\process\process_func.py", line 24, in __init__
    self.lm3d_std = load_lm3d(bfm_folder)
  File "D:\HeadSwap\model\third\Deep3dRec\preprocess.py", line 8, in load_lm3d
    Lm3D = loadmat(osp.join(bfm_folder, 'similarity_Lm3D_all.mat'))
  File "C:\Users\User\anaconda3\envs\simswap\lib\site-packages\scipy\io\matlab\mio.py", line 222, in loadmat
    with _open_file_context(file_name, appendmat) as f:
  File "C:\Users\User\anaconda3\envs\simswap\lib\contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "C:\Users\User\anaconda3\envs\simswap\lib\site-packages\scipy\io\matlab\mio.py", line 17, in _open_file_context
    f, opened = _open_file(file_like, appendmat, mode)
  File "C:\Users\User\anaconda3\envs\simswap\lib\site-packages\scipy\io\matlab\mio.py", line 45, in _open_file
    return open(file_like, mode), True
FileNotFoundError: [Errno 2] No such file or directory: 'pretrained_models/BFM\\similarity_Lm3D_all.mat'

In the distribution of Basel Face Model (241,347 kB) there is no similarity_Lm3D_all.mat. Please specify which files are needed in BFM folder other than 01_MorphableModel.mat and Exp_Pca.bin from Deep3DFaceRecon_pytorch

Face light adjustment?

I find that in those samples the result face light is affected by the source face but not the target one, any way to fix that?

What alignment/cropping/format?

Hello,

I tried your repo, but the results didn't work. What preprocessing do I need to do on the images before running them through inference.py?

Thanks!

If you get errors in Colab

Try this version :
https://colab.research.google.com/drive/1zoRLaQvpqEBLpxk-Vp4MtNnkaku5Lb5t?usp=sharing

I think the face-alignment version seems to be particularly prone to errors.

Have you tried using it for video?

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (5,) + inhomogeneous part.

Does anyone meet this problem? I've done everything before the final step and finally met this.

ValueError Traceback (most recent call last)
Cell In[5], line 1
----> 1 oup = model.run_single(src,tgt,crop_align=True,cat=True)
2 cv2.imshow("",oup)

Cell In[2], line 50
47 src_inp = self.preprocess(src_align)
48 tgt_inp = self.preprocess(tgt_align)
---> 50 tgt_params = self.get_params(cv2.resize(tgt_align,(256,256)),
51 info['rotated_lmk']/2.0).unsqueeze(0)
53 gen = self.forward(src_inp,tgt_inp,tgt_params)
55 gen = self.postprocess(gen[0])

File d:\HeadSwap\process\process_func.py:45, in Process.get_params(self, img, lmk)
43 def get_params(self,img,lmk):
---> 45 img,_,crop_param = self.preprocess_params(img,lmk)
46 with torch.no_grad():
47 coeff_3dmm = self.ParamsModel(img)

File d:\HeadSwap\process\process_func.py:73, in Process.preprocess_params(self, img, lm)
69 W,H = images.size
71 lm[:, -1] = H - 1 - lm[:, -1]
---> 73 trans_params, img, lm, _ = align_img(images, lm, self.lm3d_std)
74 img = torch.tensor(np.array(img)/255., dtype=torch.float32).permute(2, 0, 1)
...
76 img_new, lm_new, mask_new = resize_n_crop_img(img, lm, t, s, target_size=target_size, mask=mask)
---> 77 trans_params = np.array([w0, h0, s, t[0], t[1]],dtype=np.float32)
79 return trans_params, img_new, lm_new, mask_new

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (5,) + inhomogeneous part.

Google Colab

Hello, can you create a google colab version of this?