yalchik / vader

Deep learning for clustering of multivariate short time series with potentially many missing values

Hi there 👋

• 🔭 I’m currently working on a deep learning model called "VaDER" (https://github.com/yalchik/VaDER) that can cluster patients data.
• 👯 I’m looking to collaborate on AI projects related to bio-, med-, pharma-, chemo- industries.

vader's Issues

Add option to use bayesian optimization instead of the grid search

Modify the output report (or add a new type of it) to recommend certain 'k'

The current report looks like this:

It does not tell us which 'k' we should use (or which set of hyperparameters with 'k'). One idea is to sort the pages not by the median of difference, but rather by maximum difference of prediction strength among all 'k'-s.

vader.cluster method fails with ValueError: array must not contain infs or NaNs

Some jobs fail with the following error:

Job failed: 0a1c51a1-2ce1-423e-92ca-0f04db8cbf3e with err=array must not contain infs or NaNs, Traceback: Traceback (most recent call last):
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/vader_hyperparameters_optimizer.py", line 198, in run_cv_full_job
    result = job.run()
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/job/abstract_optimization_job.py", line 88, in run
    cv_fold_result = self._cv_fold_step(X_train, X_val, W_train, W_val)
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/job/full_optimization_job.py", line 42, in _cv_fold_step
    test_latent_loss) = clustering_func(X_train, X_val, W_train, W_val)
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/job/full_optimization_job.py", line 126, in _single_clustering
    effective_k = len(Counter(vader.cluster(X_train, W_train)))
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 562, in cluster
    return self._cluster(mu_tilde, mu_c, sigma2_c, phi_c)
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 425, in _cluster
    p = np.array([f(mu_t, mu[i], sigma2[i], phi[i]) for i in np.arange(mu.shape[0])])
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 425, in <listcomp>
    p = np.array([f(mu_t, mu[i], sigma2[i], phi[i]) for i in np.arange(mu.shape[0])])
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 424, in f
    return np.log(self.eps + phi) + np.log(self.eps + multivariate_normal.pdf(mu_t, mean=mu, cov=np.diag(sigma2)))
  File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/stats/_multivariate.py", line 525, in pdf
    psd = _PSD(cov, allow_singular=allow_singular)
  File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/stats/_multivariate.py", line 158, in __init__
    s, u = scipy.linalg.eigh(M, lower=lower, check_finite=check_finite)
  File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/linalg/decomp.py", line 445, in eigh
    a1 = _asarray_validated(a, check_finite=check_finite)
  File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/_lib/_util.py", line 263, in _asarray_validated
    a = toarray(a)
  File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/numpy/lib/function_base.py", line 499, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs

This error might be related to the interim calculations, since the input data set does not contain ins or NaNs.

Add a function to generate normalized cluster mean trajectories relative to baseline (x-axis in months).

Reference: Figure 4 form the paper.

Improve vader-sklearn-adapter scoring function to use prediction strength

Currently, VaDERSklearnClustering (from vader_sklearn.py) uses sum of reconstruction_loss and latent_loss as the score function. We may need to change it to support prediction_strength (and possibly adjusted rand score too).

Investigate why the significance of difference sometimes stucks at the value of ~2.2 for all k-s for certain sets of hyperparameters

Sometimes, we get the following report (attention to the "significance of difference"):

We need to investigate why it happens and what is the meaning of this particular p-value.

vader.cluster method fails with numpy.linalg.LinAlgError: singular matrix

Sometimes the optimization jobs fail with the following error:

Job failed: ca88fd3f-f30c-4fcc-9580-a1943282c5ee and job_params_dict={'k': 2, 'n_hidden': [128, 8], 'learning_rate': 0.01, 'batch_size': 64, 'alpha': 1.0}
Traceback (most recent call last):
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/vader_hyperparameters_optimizer.py", line 199, in run_cv_full_job
    result = job.run()
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/job/abstract_optimization_job.py", line 88, in run
    cv_fold_result = self._cv_fold_step(X_train, X_val, W_train, W_val)
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/hp_opt/job/full_optimization_job.py", line 52, in _cv_fold_step
    y_true = vader.cluster(X_val, W_val)
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 616, in cluster
    return self._cluster(mu_tilde, mu_c, sigma2_c, phi_c)
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 475, in _cluster
    p = np.array([f(mu_t, mu[i], sigma2[i], phi[i]) for i in np.arange(mu.shape[0])])
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 475, in <listcomp>
    p = np.array([f(mu_t, mu[i], sigma2[i], phi[i]) for i in np.arange(mu.shape[0])])
  File "/home/iyalchyk/my_repo/VaDER/tensorflow2/vader/vader.py", line 474, in f
    return np.log(self.eps + phi) + np.log(self.eps + multivariate_normal.pdf(mu_t, mean=mu, cov=np.diag(sigma2)))
  File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/stats/_multivariate.py", line 525, in pdf
    psd = _PSD(cov, allow_singular=allow_singular)
  File "/home/iyalchyk/vader_venv/lib/python3.7/site-packages/scipy/stats/_multivariate.py", line 165, in __init__
    raise np.linalg.LinAlgError('singular matrix')
numpy.linalg.LinAlgError: singular matrix

Probably, the root cause is the same as in #4, and the Johann's commit 00d2f84 hasn't fixed the problem.

Johann mentioned that the problem starts by getting very high variance in some columns in "z" calculated during the "pre_fit" stage in the "z = self.map_to_latent(self.X, self.W, n_samp=10)" statement.