Job failed: 0a1c51a1-2ce1-423e-92ca-0f04db8cbf3e with err=array must not contain infs or NaNs, Traceback: Traceback (most recent call last):
File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/vader_hyperparameters_optimizer.py", line 198, in run_cv_full_job
result = job.run()
File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/job/abstract_optimization_job.py", line 88, in run
cv_fold_result = self._cv_fold_step(X_train, X_val, W_train, W_val)
File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/job/full_optimization_job.py", line 42, in _cv_fold_step
test_latent_loss) = clustering_func(X_train, X_val, W_train, W_val)
File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/job/full_optimization_job.py", line 126, in _single_clustering
effective_k = len(Counter(vader.cluster(X_train, W_train)))
File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 562, in cluster
return self._cluster(mu_tilde, mu_c, sigma2_c, phi_c)
File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 425, in _cluster
p = np.array([f(mu_t, mu[i], sigma2[i], phi[i]) for i in np.arange(mu.shape[0])])
File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 425, in <listcomp>
p = np.array([f(mu_t, mu[i], sigma2[i], phi[i]) for i in np.arange(mu.shape[0])])
File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 424, in f
return np.log(self.eps + phi) + np.log(self.eps + multivariate_normal.pdf(mu_t, mean=mu, cov=np.diag(sigma2)))
File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/stats/_multivariate.py", line 525, in pdf
psd = _PSD(cov, allow_singular=allow_singular)
File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/stats/_multivariate.py", line 158, in __init__
s, u = scipy.linalg.eigh(M, lower=lower, check_finite=check_finite)
File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/linalg/decomp.py", line 445, in eigh
a1 = _asarray_validated(a, check_finite=check_finite)
File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/_lib/_util.py", line 263, in _asarray_validated
a = toarray(a)
File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/numpy/lib/function_base.py", line 499, in asarray_chkfinite
"array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs
This error might be related to the interim calculations, since the input data set does not contain ins or NaNs.