Comments (24)
Quick and dirty code to help you get started. I've excluded the lighting bit, but its detailed above if you want to include it (its optional).
Step 1 - Align your image
Use Align_face_image in utils.py. This will change your input image and align the detected face. It's best to crop the photo manually to avoid GPU memory issues.
Step 2 - Project your aligned image
Project your custom image via https://github.com/rolux/stylegan2encoder.
I used the cmd "python project_images.py aligned_images/ generated_images/" after dropping my aligned image into aligned_images. This will product a npy file and a preview of the achieved image.
Step 3 - Get the attributes
Go to https://azure.microsoft.com/en-us/services/cognitive-services/face/ upload your face and make a note of the values using detection model 1:
0-Gender (girl is 0)
1-Glasses (no glasses is 0)
2-Yaw
3-Pitch
4-Baldness
5-Beard
6-Age
7-Expresion (Smile I think)
Step 4 - Insert into StyleFlow
This is up to you. However, I hijacked the existing code to bypass the requirement for lighting file.
I changed the sample indexes to the first 100.
self.keep_indexes = [i for i in range(0,100)]
I then overwrote the first 3 latent with my custom images (3 as an example):
self.raw_w = pickle.load(open("data/sg2latents.pickle", "rb"))
self.raw_w['Latent'][0][0] = np.load('data/DW.npy')
self.raw_w['Latent'][1][0] = np.load('data/JS.npy')
self.raw_w['Latent'][2][0] = np.load('data/RC.npy')
I then overwrote the first 3 attributes with my custom data:
self.all_attr[0][0][0] = 0 #Gender
self.all_attr[0][1][0] = 0 #Glasses
self.all_attr[0][2][0] = -1.5 #Yaw
self.all_attr[0][3][0] = -9.2 #Pitch
self.all_attr[0][4][0] = 0.02 #Baldness
self.all_attr[0][5][0] = 0 #Beard
self.all_attr[0][6][0] = 25 #Age
self.all_attr[0][7][0] = 3 #Expresion
#1
self.all_attr[1][0][0] = 0 #Gender
self.all_attr[1][1][0] = 0 #Glasses
self.all_attr[1][2][0] = -9.9 #Yaw
self.all_attr[1][3][0] = -0.2 #Pitch
self.all_attr[1][4][0] = 0.24 #Baldness
self.all_attr[1][5][0] = 0 #Beard
self.all_attr[1][6][0] = 30 #Age
self.all_attr[1][7][0] = 1 #Expresion
#3
self.all_attr[2][0][0] = 0 #Gender
self.all_attr[2][1][0] = 0 #Glasses
self.all_attr[2][2][0] = -5.1 #Yaw
self.all_attr[2][3][0] = -5.1 #Pitch
self.all_attr[2][4][0] = 0.5 #Baldness
self.all_attr[2][5][0] = 0 #Beard
self.all_attr[2][6][0] = 29 #Age
self.all_attr[2][7][0] = 1 #Expresion
I also modified the blue circles to help find the first 3 indexes:
count = 0
for point in self.X_samples:
######### don't use np.uint8 in tuple((point*1024).astype(int))
if count < 3:
cv2.circle(self.map, tuple((point * 1024).astype(int)), 7, (0, 0, 100), -1)
else:
cv2.circle(self.map, tuple((point * 1024).astype(int)), 6, (0, 0, 255), -1)
count = count + 1
from styleflow.
Will try to include real image projection and setup in the next update.
from styleflow.
For projected faces, since there are many methods to project the images (see https://arxiv.org/abs/2012.09036 ) with variable quality and editing capabilities, it becomes difficult to handle all these frameworks. Currently, https://github.com/eladrich/pixel2style2pixel also does a good job striking a balance between this trade-off. Also, we add a discussion regarding the editability in the final version that will be online soon. After projecting the images, also make sure to modify (increase for the real images) the degree (here) and ranges (here) for the edits.
from styleflow.
Are you suggesting that the transformation results are not as expected due to bad image projection? If this is the case do you have any settings regarding the projection used throughout the project that led to the best results?
Whenever I run the UI and play with the sliders, the results are nowhere near as striking as shown in the demos. The switch in gender or glasses is not convincing (even after adjusting the slider ranges and degree). This makes it look like the model doesn't work as well on any unseen data.
from styleflow.
Hello @PaulCzaban,
I think I manage to do what you said. I can open my pictures but when I move the cursors the face does not get modify by a lot.
For exemple gender does not switch to female for a male and vice-versa.
I see small modifications of the face but not a gender switch.
Do you know what I can do to improve the results or maybe what I missed?
from styleflow.
Hello @PaulCzaban,
I think I manage to do what you said. I can open my pictures but when I move the cursors the face does not get modify by a lot.
For exemple gender does not switch to female for a male and vice-versa.
I see small modifications of the face but not a gender switch.
Do you know what I can do to improve the results or maybe what I missed?
I also observed this behaviour. Some pictures work better than others so do try another picture. I suggest you also try increasing the max/min boundaries of the attribute sliders. I posted an issue regarding this below.
from styleflow.
The best bet is to try a selection of photos, go for passport worthy, decent definition photos.
Its my belief the model just doesn't respond perfectly well to all faces... Maybe they sit too far away from the training data.
It's also possible that the alignment and projection process didn't run perfectly. If the features of the face aren't found then it might result in this unresponsive behaviour.
Either way I don't think there is an easy fix, other than trying a different photo.
from styleflow.
The framework depends on Microsoft API for the annotations. The lighting parameters can be derived from here. After getting the projections of the images, These files can be replaced.
from styleflow.
Quick and dirty code to help you get started. I've excluded the lighting bit, but its detailed above if you want to include it (its optional).
Step 1 - Align your image
Use Align_face_image in utils.py. This will change your input image and align the detected face. It's best to crop the photo manually to avoid GPU memory issues.Step 2 - Project your aligned image
Project your custom image via https://github.com/rolux/stylegan2encoder.
I used the cmd "python project_images.py aligned_images/ generated_images/" after dropping my aligned image into aligned_images. This will product a npy file and a preview of the achieved image.Step 3 - Get the attributes
Go to https://azure.microsoft.com/en-us/services/cognitive-services/face/ upload your face and make a note of the values using detection model 1:
0-Gender (girl is 0)
1-Glasses (no glasses is 0)
2-Yaw
3-Pitch
4-Baldness
5-Beard
6-Age
7-Expresion (Smile I think)Step 4 - Insert into StyleFlow
This is up to you. However, I hijacked the existing code to bypass the requirement for lighting file.I changed the sample indexes to the first 100.
self.keep_indexes = [i for i in range(0,100)]
I then overwrote the first 3 latent with my custom images (3 as an example):
self.raw_w = pickle.load(open("data/sg2latents.pickle", "rb")) self.raw_w['Latent'][0][0] = np.load('data/DW.npy') self.raw_w['Latent'][1][0] = np.load('data/JS.npy') self.raw_w['Latent'][2][0] = np.load('data/RC.npy')
I then overwrote the first 3 attributes with my custom data:
self.all_attr[0][0][0] = 0 #Gender self.all_attr[0][1][0] = 0 #Glasses self.all_attr[0][2][0] = -1.5 #Yaw self.all_attr[0][3][0] = -9.2 #Pitch self.all_attr[0][4][0] = 0.02 #Baldness self.all_attr[0][5][0] = 0 #Beard self.all_attr[0][6][0] = 25 #Age self.all_attr[0][7][0] = 3 #Expresion #1 self.all_attr[1][0][0] = 0 #Gender self.all_attr[1][1][0] = 0 #Glasses self.all_attr[1][2][0] = -9.9 #Yaw self.all_attr[1][3][0] = -0.2 #Pitch self.all_attr[1][4][0] = 0.24 #Baldness self.all_attr[1][5][0] = 0 #Beard self.all_attr[1][6][0] = 30 #Age self.all_attr[1][7][0] = 1 #Expresion #3 self.all_attr[2][0][0] = 0 #Gender self.all_attr[2][1][0] = 0 #Glasses self.all_attr[2][2][0] = -5.1 #Yaw self.all_attr[2][3][0] = -5.1 #Pitch self.all_attr[2][4][0] = 0.5 #Baldness self.all_attr[2][5][0] = 0 #Beard self.all_attr[2][6][0] = 29 #Age self.all_attr[2][7][0] = 1 #Expresion
I also modified the blue circles to help find the first 3 indexes:
count = 0 for point in self.X_samples: ######### don't use np.uint8 in tuple((point*1024).astype(int)) if count < 3: cv2.circle(self.map, tuple((point * 1024).astype(int)), 7, (0, 0, 100), -1) else: cv2.circle(self.map, tuple((point * 1024).astype(int)), 6, (0, 0, 255), -1) count = count + 1
I cannot proceed with Step 3 (getting face attributes from Azure) as there's no option to run face detection demo online (I'm wondering if I'm alone with this problem). Any ideas how can I resolve this? I guess another option would be to use the API from code, but then I would have to write unreasonably high amount of code just to run this simple step.
from styleflow.
@RameenAbdal thanks, will you consider adding a tutorial for this ?
from styleflow.
This issue should be re-opened and closed only when the feature is available 👍
from styleflow.
Thanks @RameenAbdal !
from styleflow.
Has anyone succeeded in editing their own images?
from styleflow.
I was hoping to hijack the current image to be edited and try to find a similar sg2latents image that matches lighting, attributes etc. Unfortunately its not that simple as images are encoded via stylegan-encoder.
I don't have days to burn on this so hopefully the real image projection etc will be as simple as we all hope.
from styleflow.
here
So if I want to upload and 'modify' the image from elsewhere, the image should processed via Microsoft API and DRP for lighting?
from styleflow.
here
So if I want to upload and 'modify' the image from elsewhere, the image should processed via Microsoft API and DRP for lighting?
Yes.
I believe you'll need to change 4 files.
TSNE.npy (These look like the circles you click on to select an index. Maybe just keep the circle for the first index)
light.npy (https://github.com/zhhoper/DPR)
sg2latents.pickle (https://github.com/spiorf/stylegan-encoder)
attributes.npy (Microsoft Face Detection)
The outputs from those repos will need further formatting to match the files above. Some backward engineering is needed to get it just right I think.
from styleflow.
Thanks. If I understood correctly,
- I have to change 4 files to process images from elsewhere, right?
- Or process images using Microsoft API and DPR(lighting), and change codes for 'load' it?
Is there any plan for support custom image processing in near future?
from styleflow.
Thanks I edited the min and max and retry. I will try with another picture.
Do you think if I increase the number of iterations in step 2 it could be better? I tried 2000 but the execution crashed (not enough memory in my GPU, it's only a GTX 1050.
from styleflow.
It seems the training set from this repo suffers greatly from bias making it perform terrible on any custom portrait I've tried.
from styleflow.
Has anyone achieved good results when projecting custom images? I used encoders suggested by @PaulCzaban and results are bad. But I noticed that if I set --num-steps
to a small value (https://github.com/rolux/stylegan2encoder) it produces another face and this face works great. I can transform gender and beard and other, but if I run more steps projection will fit the given image and stop working.
from styleflow.
For projected faces, since there are many methods to project the images (see https://arxiv.org/abs/2012.09036 ) with variable quality and editing capabilities, it becomes difficult to handle all these frameworks. Currently, https://github.com/eladrich/pixel2style2pixel also does a good job striking a balance between this trade-off. Also, we add a discussion regarding the editability in the final version that will be online soon. After projecting the images, also make sure to modify (increase for the real images) the degree (here) and ranges (here) for the edits.
.I would like to know if you plan to add attribute editing of custom images. I edited the actual images I took by myself and processed them, and the photos I got seemed to be made up(cosmetics),Not real . I can't find the reason。
from styleflow.
Quick and dirty code to help you get started. I've excluded the lighting bit, but its detailed above if you want to include it (its optional).
Step 1 - Align your image
Use Align_face_image in utils.py. This will change your input image and align the detected face. It's best to crop the photo manually to avoid GPU memory issues.Step 2 - Project your aligned image
Project your custom image via https://github.com/rolux/stylegan2encoder.
I used the cmd "python project_images.py aligned_images/ generated_images/" after dropping my aligned image into aligned_images. This will product a npy file and a preview of the achieved image.Step 3 - Get the attributes
Go to https://azure.microsoft.com/en-us/services/cognitive-services/face/ upload your face and make a note of the values using detection model 1:
0-Gender (girl is 0)
1-Glasses (no glasses is 0)
2-Yaw
3-Pitch
4-Baldness
5-Beard
6-Age
7-Expresion (Smile I think)Step 4 - Insert into StyleFlow
This is up to you. However, I hijacked the existing code to bypass the requirement for lighting file.I changed the sample indexes to the first 100.
self.keep_indexes = [i for i in range(0,100)]
I then overwrote the first 3 latent with my custom images (3 as an example):
self.raw_w = pickle.load(open("data/sg2latents.pickle", "rb")) self.raw_w['Latent'][0][0] = np.load('data/DW.npy') self.raw_w['Latent'][1][0] = np.load('data/JS.npy') self.raw_w['Latent'][2][0] = np.load('data/RC.npy')
I then overwrote the first 3 attributes with my custom data:
self.all_attr[0][0][0] = 0 #Gender self.all_attr[0][1][0] = 0 #Glasses self.all_attr[0][2][0] = -1.5 #Yaw self.all_attr[0][3][0] = -9.2 #Pitch self.all_attr[0][4][0] = 0.02 #Baldness self.all_attr[0][5][0] = 0 #Beard self.all_attr[0][6][0] = 25 #Age self.all_attr[0][7][0] = 3 #Expresion #1 self.all_attr[1][0][0] = 0 #Gender self.all_attr[1][1][0] = 0 #Glasses self.all_attr[1][2][0] = -9.9 #Yaw self.all_attr[1][3][0] = -0.2 #Pitch self.all_attr[1][4][0] = 0.24 #Baldness self.all_attr[1][5][0] = 0 #Beard self.all_attr[1][6][0] = 30 #Age self.all_attr[1][7][0] = 1 #Expresion #3 self.all_attr[2][0][0] = 0 #Gender self.all_attr[2][1][0] = 0 #Glasses self.all_attr[2][2][0] = -5.1 #Yaw self.all_attr[2][3][0] = -5.1 #Pitch self.all_attr[2][4][0] = 0.5 #Baldness self.all_attr[2][5][0] = 0 #Beard self.all_attr[2][6][0] = 29 #Age self.all_attr[2][7][0] = 1 #Expresion
I also modified the blue circles to help find the first 3 indexes:
count = 0 for point in self.X_samples: ######### don't use np.uint8 in tuple((point*1024).astype(int)) if count < 3: cv2.circle(self.map, tuple((point * 1024).astype(int)), 7, (0, 0, 100), -1) else: cv2.circle(self.map, tuple((point * 1024).astype(int)), 6, (0, 0, 255), -1) count = count + 1
When trying to align an image I get a Name Error: name 'image' is not defined
. Any ideas?
I'm calling the function from powershell using python -c 'import utils; utils.Align_face_image(image.jpg)'
from styleflow.
Has anyone achieved good results when projecting custom images? I used encoders suggested by @PaulCzaban and results are bad. But I noticed that if I set
--num-steps
to a small value (https://github.com/rolux/stylegan2encoder) it produces another face and this face works great. I can transform gender and beard and other, but if I run more steps projection will fit the given image and stop working.
You can try to use https://github.com/omertov/encoder4editing to obtain the inverted latent code instead of https://github.com/rolux/stylegan2encoder. In my experiment, e4e achieves better editing results.
from styleflow.
Hi, does anyone successfully use microsoft azure api now? The microsoft seems to limit the usage of face api unitl a form is applied for access
from styleflow.
Related Issues (20)
- web-ui question
- How to get a series of continuous poses HOT 1
- Thanks for you work, And I want to ask for advises about styleflow used for image-to-image translation. HOT 1
- Error when running on Windows 10 HOT 2
- Overfitting HOT 2
- [Q]. How to only change face pose? HOT 4
- Upgrade to stylegan3
- latents, lighting and attribute parameters
- AssertionError: If using tupled rtol it must have the same length as the tuple y0 HOT 2
- The training loss is extramely large (close to 3.368e6)?
- qt.qpa.screen: QXcbConnection: Could not connect to display HOT 1
- web ui issue: "UnpicklingError: pickle data was truncated"
- The Microsoft Face API can not be used
- How to change .pkl ?
- Execution freezes after some steps HOT 1
- Attribute adjustment function
- Attribute adjustment
- About the generation of ".npy" files
- training problem
- cant use ui upload image
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from styleflow.