bioinf-jku / ttur Goto Github PK
View Code? Open in Web Editor NEWTwo time-scale update rule for training GANs
License: Apache License 2.0
Two time-scale update rule for training GANs
License: Apache License 2.0
Hi,
I'm training a DC GAN on MNIST dataset. I want to compute FID for my model. In this case should I use the Inception network itself or should I use a different classifier trained on MNIST dataset?
Hello,
I was experimenting with the --lowprofile flag but an error is thrown saying 'n_images' from line 226 of fid.py is not defined. Likely a typo since the author probably meant to reference n_imgs.
Additionally, line 234 pred_arr[start:end] = pred.reshape(batch_size,-1)
will crash if the number of images is not divisible by batch_size. Which is the case with 2048 images and the default batch_size of 50.
I have trained "GoodGenerator" (in the provided code) with TTUR (using the given default hyperparameters) for a couple of times. My trained model achieves about 29 FID score for the CIFAR-10 dataset. To achieve the score in Table 1, should I pick the best score among multiple training and multiple generations? Or, should I change the hyperparameters? Thank you.
I use the fid.py to measure fid score of my images datasets, I generated 10000 images in tImages directory and used the command 'python fid.py ./tImages fid_stats_celeba.npz' or 'python fid.py ./tImages ./sImages ' (sImages directory is another images datasets), but waiting a while, I always get a ValueError. e.g. 'ValueError: Imaginary component 89928690.58258057' or
'ValueError: Imaginary component 1.376687186290827e+24'. I don't know which step I did wrong. Could anyone tell me what the problem is, thanks!
error information:
Traceback (most recent call last):
File "fid.py", line 334, in
fid_value = calculate_fid_given_paths(args.path, args.inception, low_profile=args.lowprofile)
File "fid.py", line 317, in calculate_fid_given_paths
fid_value = calculate_frechet_distance(m1, s1, m2, s2)
File "fid.py", line 155, in calculate_frechet_distance
raise ValueError("Imaginary component {}".format(m))
ValueError: Imaginary component 1.376687186290827e+24
Hi
the output of function "get_activations_from_files" is "A numpy array of dimension (num images, 2048) ..." .
That is to say,the "act" is "A numpy array of dimension (num images, 2048) ".
In the formula "sigma = np.cov(act, rowvar=False)",the rowvar is False.
But I think the rowvar should be True because the first dimension of "act" is "num images",so the dimension of "sigma" should be (num images,num images) instead of (2048,2048).
Hi, I found that you implemented the final FID calculation result in this line
return diff.dot(diff) + np.trace(sigma1) + np.trace(sigma2) - 2 * tr_covmean
But according to your formula, the trace are applied after both sigmas and covmean
being calculated, so it should be in
return diff.dot(diff) + np.trace(sigma1 + sigma2 - 2 * covmean)
I have experimented using both, and it seems that the calculation are both similar (until 13 numbers behind decimal point). Any explanation on reason of using the former?
Hello martin!
The work is very great.But i want to consult whether FID can be used in medical image field? After all,Iception mode is trained on natural images.Thank you very much!
It seems that there is no any create_incpetion_graph()
definition in fid.py
.
Also, the error message from fid_example.py runs as:
AttributeError: module 'fid' has no attribute 'create_incpetion_graph'
Have you met the same problem?
I calculated the state file of cifar-10(32x32) with precalc_stats_example.py, and calculate FID with the obtained state file and the one in your website, the FID is 12. Could you kindly help to point out how this could happen?
Why is the FID score calculated using the CelebA real image 64*64 always around 27? It should be less than 5. . .The images I used are images cropped with list_bbox_celeba.txt for a total of 50,000.
the number of real images is 6120, and the number of generated images is 6120. but fid.py is always killed on Linux? The available memory is 27GB,batch_size= 8,is it because the memory is too small?
I was trying to run fid scores for some CelebA images and kept bugging out at 20K-30K samples. After some examination, I found that the error lay in fid.py typecasting to np.float32, and this was resulting in NaN and inf values etc.
Should this be mentioned in fid.py or simply typecast to float64 ? I am not aware of any downside to float64.
I met a error for 'tf.Tensor._shape cannot be assigned', so I change the
o._shape = tf.TensorShape(new_shape)
to
o.set_shape(tf.TensorShape(new_shape))
mentioned by another project's commit
Unfortunately, this work fine for above project, but this time I still get a
ValueError: Cannot feed value of shape (50, 128, 128, 3) for Tensor 'FID_Inception_Net/ExpandDims:0', which has shape '(1, ?, ?, 3)
.
I find if I print the tensor shape use get_shape()
before and after the set_shape
call in function _get_inception_layer
, the Tensor shape don't change. It seems the set_shape
don't work.
Does anyone have any ideas? Thanks
It appears that the code never resizes images to be the correct 299x299 for the inception model. Is it the case that all of the results on 64x64 images are obtained by feeding smaller images into the convolutional network and simply assuming that the outputs are meaningful? Or is there a resize somewhere I'm not seeing?
I also observed that resolution mattered immensely when comparing to the precomputed npz matrices in this repository. In particular, if the images were not 64x64, the FID was extremely high, so I'm assuming those npz matrices were computed by feeding 64x64 images directly into the inception graph.
The InceptionV3 is trained with image size as 299x299.
But I don't find the code that resize image as 299x299. Dose calculating FID not require this
process? Are the images resized offline?
Original stack trace for 'FID_Inception_Net/mixed_5/tower_1/conv_1/CheckNumerics':
I use the version tensorflow 1.14.
This problem appears randomly, maybe in the early stage of training, maybe in the later stage, and it always appears during a certain FID evaluation.
I don't know the reason, but it doesn't seem to be because of the mode collapse, at least the generated sample looks good to the naked eye.Can you help me?
you use the following formula to calculate fid:
d^2 = ||mu_1 - mu_2||^2 + Tr(C_1 + C_2 - 2sqrt(C_1C_2)).
The code uses dot product to calculate C_1*C_2, but the original formula seems to be based on matrix multiply. Could you clarify on this?
Hi,
I'm trying FID.
I get "nan" or "complex number", which stems from the "sp.linalg.sqrtm".
Have you ever faced similar issues? How should I solve this problem?
Thanks a lot.
Hi, thanks for your wonderful work. I have met a value error recently. However, sometimes it happens, but if I rerun the code, it disappears. I wonder how can I solve it? But I am not sure why it happens.
is there a way to calculate the Frechet Inception Distance between a dataset and just one sample (instead of a set of them), producing consistent results?
Hi,
I'm trying to get FID score on two dataset of Images (one real world data and other is GAN generated).
Sizes: Real - 15780, Synthetic - 16000
Whenever I'm executing python fid.py
or python precalc_stats_example.py
I'm getting above mentioned error. I've attached the SS below:
Any inputs would be appreciated!
Hi,
I have a few questions about FID score:
I have a dataset smaller than 2048 images, but I still want to compute FID score. I understand that 2048 images are required as to get a full rank covariance matrix. Computing the FID on my dataset still gives me sensible values, and no complex numbers or NaNs (or warnings). Can I still trust the FID score computed this way as a measure of visual quality?
In this paper, section 5.1, it is noted:
We observe that FID has rather high bias, but small variance, From this perspective, estimating the full covariance matrix might be unnecessary and counter-productive, and a constrained version might suffice.
What is your take on this? Does this hint to FID being usable with smaller datasets?
I also ported your implementation to PyTorch, for people who do not want to have the Tensorflow dependency (see here). I hope that is okay with you.
Thanks!
Hello, thanks for your wonderful work. I run in colab and I have met an error recently. but I don't know how to fix it.
Here is the whole message:
KeyError Traceback (most recent call last)
in ()
6 with tf.compat.v1.Session() as Sess:
7 Sess.run(tf.compat.v1.global_variables_initializer())
----> 8 mu_gen, sigma_gen = fid.calculate_activation_statistics(images, Sess, batch_size=100)
9
10 fid.fid_value = calculate_frechet_distance(mu_gen, sigma_gen, mu_real, sigma_real)
5 frames
/content/TTUR/fid.py in calculate_activation_statistics(images, sess, batch_size, verbose)
178 the incption model.
179 """
--> 180 act = get_activations(images, sess, batch_size, verbose)
181 mu = np.mean(act, axis=0)
182 sigma = np.cov(act, rowvar=False)
/content/TTUR/fid.py in get_activations(images, sess, batch_size, verbose)
81 activations of the given tensor when feeding inception with the query tensor.
82 """
---> 83 inception_layer = _get_inception_layer(sess)
84 n_images = images.shape[0]
85 if batch_size > n_images:
/content/TTUR/fid.py in _get_inception_layer(sess)
47 """Prepares inception net for batched usage and returns pool_3 layer. """
48 layername = 'FID_Inception_Net/pool_3:0'
---> 49 pool3 = sess.graph.get_tensor_by_name(layername)
50 ops = pool3.graph.get_operations()
51 for op_idx, op in enumerate(ops):
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in get_tensor_by_name(self, name)
3900 raise TypeError("Tensor names are strings (or similar), not %s." %
3901 type(name).name)
-> 3902 return self.as_graph_element(name, allow_tensor=True, allow_operation=False)
3903
3904 def _get_tensor_by_tf_output(self, tf_output):
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in as_graph_element(self, obj, allow_tensor, allow_operation)
3724
3725 with self._lock:
-> 3726 return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
3727
3728 def _as_graph_element_locked(self, obj, allow_tensor, allow_operation):
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in _as_graph_element_locked(self, obj, allow_tensor, allow_operation)
3766 raise KeyError("The name %s refers to a Tensor which does not "
3767 "exist. The operation, %s, does not exist in the "
-> 3768 "graph." % (repr(name), repr(op_name)))
3769 try:
3770 return op.outputs[out_n]
KeyError: "The name 'FID_Inception_Net/pool_3:0' refers to a Tensor which does not exist. The operation, 'FID_Inception_Net/pool_3', does not exist in the graph."
I've been reading a lot of papers regarding FID in terms of "number of samples", without really clarifying if that applies to the fake samples from the generator, the real samples, the sum, the same number for both, etc.
When you say "number of samples" or sample size, which one is it? Because I am currently generating fake samples from a very small dataset (300 images) and I want to calculate the FID between the 300 real images and fake images, but I am not sure how many samples to use. The logic tells me that I should use the same number of samples for both real and fake images but that may cause calculation problems.
Thank you in advance.
Hi, when using your script to compute the FID between two folders of 10k images (as is recommended) the system memory requirement of the script is gigantic.
I have looked into the script and seen that you load all of the images into RAM before starting to evaluate the statistics. Is there a way to handle this in batched form or even create a running version of the statistics?
That way it might take a little longer to compute, but would definitely relax the memory requirements by a large amount, which in will speed up the script by reducing swap space needed.
Thanks in advanced.
P.S: If you say this can be done, but don't have the time to do it, I can try implementing it myself and issue a pull request, but I'd like to know from you wether that would affect the correctness of the algorithm.
I tried to calculated FID on 2 images that are the same, and it returns a negative number that's really close to 0
when I use the version tensorflow 1.1and1.2 ,the problem still exist.
ValueError: Cannot feed value of shape (50, 256, 256, 3) for Tensor 'FID_Inception_Net/ExpandDims:0', which has shape '(1, ?, ?, 3)'
I don't know the reason? Can you help me?
Hello, I'm interested in using your code to evaluate several GAN models that I used as of now. Although, I have few question regarding your implementation of FID score
batch_size
parameter. How does this affect the calculation? And which size is better for a given set of data?Hi,
I changed dim to 32 and ran gan_64x64.py for cifar10 dataset, but I got 'IS_mean: 7.0972', 'IS_std: 0.0815', FID: 32.4961. What's the problem?
Hi,
In Table 1, the two FID scores for CelebA are completely identical to that of SVHN. From my own experiments, they are quite different. I would like to cross-check with you if this is an editing typo. Thanks.
Hello, I have a problem in using your code on my custom dataset.
My dataset is a collection of images already in npz format, and I'm able to extract it directly without had to do imread first. But I did an initial resize since my image is in 1-channel format
dataset = np.load('dataset_file.npz')
x_test_target = dataset['img_set'].astype('float32')
x_test_target = np.reshape(x_test_target, (len(x_test_target), row, col, chann))
x_test_target.resize((row, col, 3))
After resizing, I tested it using the calculate_activation_statistics
method to see if Inception able to receive it as input
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
mu_real, sigma_real = fid.calculate_activation_statistics(x_test_target, sess, batch_size=100)
But I receive this error,
ValueError: Cannot feed value of shape (100, 128, 3) for Tensor 'FID_Inception_Net/ExpandDims:0', which has shape '(?, ?, ?, 3)'
Is there something wrong with my data setup? Or the batch_size
setup not suitable with my data?
I am using celebA datase where for training a GAN i preprocess the images to 64x64.
Further there are two methods mentioned to compute fid score which are :
Now as my generated samples are of 64x64 while statistics file available for celebA dataset here is corresponding to original image size. Therefore I suspect that using M2 method will give me a wrong FID score.
Also to use M1 method, I thought of a solution to sample 10k images from celebA & store them by preprocessing the images to 64x64. Now I will have original samples (10k in numbers) which have same size as generated one i.e 64x64.
Now my confusion is what do people do in practice i.e in papers when they report FID scores which method do they use M1 or M2 ?
I trained my GAN on cifar-10 (height,weight=3232), and got FID=51. This score is much lower than the one reported in other paper. Should I resize the image to 6464?
Lines 140 to 142 in dc4e5b5
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.