iusaspb / deep_learning_nikolenko_and_co Goto Github PK

Thanks for offering this great repository! We deployed a tool that automatically detected this possible numerical bug from your repository.

To fix this, maybe we can replace

z_log_sigma_sq = tf.add( tf.matmul(enc_layer_2, w["w_recog"]['out_log_sigma']), w["b_recog"]['out_log_sigma'])

at Line 46 by

z_log_sigma_sq = tf.add( tf.matmul(enc_layer_2, w["w_recog"]['out_log_sigma']), w["b_recog"]['out_log_sigma'])
z_log_sigma_sq = tf.clip_by_value(z_log_sigma_sq, -87, 87)

Similar possible issues were also found at ch10_04_04_Pic_10_06.py:

at ch10_04_05_Pic_10_07.py:

and at ch10_04_06_Pic_10_08.py:

If they are valid issues, the fixing method could be the same as above.

Thanks!

Hello~

Thank you very much for sharing your code.

I ran ch10_04_01.py on my computer. Unfortunately, I got NAN loss.

After preliminary inspection of the code, I couldn't find the root cause of NaN loss. 1e-10 was added to the log to prevent log(0). This confused me for a long time. After I checked the program carefully with tfdebug, I found the problem!

In the code, 1e-10 was added to log to prevent log(0)

reconstr_loss = -tf.reduce_sum(x * tf.log(1e-10 + x_reconstr_mean) +(1-x) * tf.log(1e-10 + 1-x_reconstr_mean), 1)

However, 1e-10 may be too small to avoid 1e-10 + 1-x_reconstr_mean becoming 0, because I found that x_reconstr_mean may sometimes become around 1.0000000001 due to floating-point accuracy issue, which may lead to log(0)

Increasing 1e-10 to 1e-7 can solve this problem.