ucl-pond / pysustain Goto Github PK

I am in the process of applying the mixture_KDE version of Sustain to an external dataset that contains cross-sectional cognitive test score data for several thousand patients. I have been trying to modify the simrun.py function, but I'm running into a few conceptual and technical roadblocks. For one, we don't have control group data, which to my understanding is acceptable for the mixture_KDE model. Without controls, I'm wondering if the random assignment approach from simrun.py for generating ground_truth_sequences, ground_truth_subtypes, ground_truth_stages_control, and ground_truth_stages_other is the appropriate first step? In the pySustain white paper it says,

Within simrun.py, simulated subjects assigned earliest stages are used as controls and those in latest stages as cases.

I don't quite understand how this transfers onto applying Sustain on real data?

In my script, after generating the random ground truth sequences etc., I comment out this line
data, data_denoised = generate_data_mixture_sustain(ground_truth_subtypes, ground_truth_stages, ground_truth_sequences, sustainType)
and use the numpy array of my own data, which is in the exact same shape as what would be generated by the above line of code. However, I am receiving a LinAlgError: Singular matrix error when running this line:
mixtures = fit_all_kde_models(true_data, labels).

I think having a clearer example script of a mixture_KDE implementation with real data would be very useful in helping me answer some of my questions. Please let me know if there are any resources that you could share with me that might be helpful, or if you could address some of my issues directly.

I can also share my current working script if it would be of any help. Thanks!

Hi,
Is there a notebook for reproducing the ordinal sustain simulations/implementation (as mentioned in ordinal sustain article)?
The current notebooks doesn't support ordinal Sustain and I am not sure how to apply it on external data set.

Hello!

I'm trying to run PySustain with my data (1129 observations and 38 biomarkers), but maybe for the high number of biomarkers, the algorithm does not move forward (even after 10 hours) on the print "Finding ML solution to 1 cluster problem". I found, inserting some print into the code to debug it, that the "heavy" code is in AbstractSustain.py into the _find_ml(): in particular for these lines of code:

partial_iter = partial(self._find_ml_iteration, sustainData)
pool_output_list = self.pool.map(partial_iter, range(self.N_startpoints))
if ~isinstance(pool_output_list, list):
             pool_output_list = list(pool_output_list)

I think that the map is very slow: the execution hangs on "list(pool_output_list)".
Do you have any idea how to resolve this problem ? I tried also generating simulated data (with 1129 observations and 38 biomarkes) but nothing happened.

Thank you in advance.

I get divide by zero errors relating to model likelihood, which I tracked back to missing data causing problems with max() and min(), etc. Couldn't fix it with numpy.nanmax(), so we probably need to devise a robust method for handling missing data.

Hi SuStaIn team!

I am trying to use SuStaIn with a train / test like approach, in which I have two dataset:

• the training one, that I want to use as to infer the summary subtypes (that is to say: subtype 1 is this sequence of abnormalities, subtype 2 is this sequence, etc), and I do not really worry about the actual subtyping of the individual subjects in this dataset. This is where I would run the run_sustain_algorithm method, if i'm correct.
• the test one, that I would like to use as follows: given the subtypes discovered on the training set, I want to subtype (and maybe stage, even though I'm less interest in this) these new subjects with respect to the subtypes discovered on the training set. Intuitively, if I go back to the following notations from the Young et al. (2018) paper, if (f_c, S_c)_c are the subtypes (and their prevalence) discovered at the training step, given some new X (the test data), I would want to evaluate P(X | S_c) for each c, and infer the best subtype for each of my new subjects from this mixture of subtypes.
  

So it seems to me that this makes sense from a methodological point of view (but I could be mistaken 😅).

Now I don't seem to find exactly how I would proceed to perform this last step, given the output from the first step. I went back to the notebook from the workshop (that I had followed some time ago) and it looks to me that the presented cross_validate_sustain_model mainly focuses on cross validation metrics, rather than outputting the subtypes corresponding to the "test" subtypes.

I am sorry if this is treated somewhere that I have missed, and don't hesitate if the question is somewhat unclear, I'm happy to rephrase or go more into details 🙂

Cheers,
Nemo

Can we model SuStaIn in probabilistic modelling library like pymc3, also can we use Variational Inference instead of MCMC?

Hi, pySuStaIn is a great work, thanks for your effort!

I hope to use SuStaIn to subtype patients from our private Alzheimer's Disease structure MRI dataset.

However, I find it hard to implement the data preparation part (i.e. how to obtain z-scores from the raw MRI images).

Could you share your AD MRI data preparation code? or maybe provide some more detailed pipeline instructions of it?

(I checked the instruction provided in your Nat.Comm. paper, but found it too coarse to reproduce.)

Grateful for your help.

Background

At present, there are some variables (e.g. ml_f_EM) that are not returned by run_sustain_algorithm but are saved in the pickle files. It doesn't make sense to me that running the model gives you different outputs from loading the pickle file that you get from running it. The run_sustain_algorithm should just output what gets saved (and should probably output it as a dict to avoid having to refer to the order in which things are output).

Further to this, when trying to load previous results from a pickle file, a lot of the same setup is still required as when running the model in the first place, which can result in a lot of unnecessary boilerplate. There is some code that indicates there was once a desire for this. If anybody knows why this wasn't pursued or if there is some obstacle I haven't noticed please let me know.

Solutions

1. Expand what's pickled and reconstruct model instance

A @classmethod to recreate the model instance from a pickle file, resulting in the same thing as if you were to run the method (with Z_vals etc. bundled in), would simplify the average workflow significantly, and would be a fairly simple change. The main issue with this is that it would probably break people's existing code, and would require the notebook(s) to be updated. It would, however, keep to the current concept (pickle the results/arrays).

2. Modify how pySuStaIn pickles and save/load model in its entirety

We could also just pickle/unpickle the model instance itself, following a few changes to enable this. The process should also fix #27 and #41, on top of addressing the above (and simplifying things a lot). I reckon it should be doable without too much bother. Further details/considerations can be found here. This would also be best-served by turning the arrays that are usually pickled into attributes, and so would lead to a lot of small changes through the code.

Practicalities

• Either approach will very likely break previous code (if people update, that is!). So, this may want to form part of a proper, versioned release.
• I am happy to work on this, but it will be at weekends etc. (though shouldn't take long).

If core maintainers agree, I'll go ahead with it, but if not then I will leave it.

Hi Neil, Leon, Others

I am trying to use Mixture SuStaIn with fixed controls in GMM (i.e. without optimizing for the controls Gaussian) and I would like to get your opinion if what I am doing is okay.

In mixture_model/utils/fit_all_gmm_models, instead of calling the fit function, I am trying to call the fit_constrained function in gmm.py

Is this okay ? I see that all the necessary functions are in place for making use of this functionality. Is there a reason for you to not provide access to this function through user controllable options (for e.g. if there is an unfixed bug in the fit_constrained function etc.) ?

Do let me know.

Cheers,
Vikram

Just putting this here for reference. Not sure there's anything we can do.

use_parallel_startpoints fails on my Mac because pickle cannot pickle a numpy object.

I was using numpy 1.21 and managed to fix the issue by downgrading to v1.20

Hi pySustain team,

I am running into the following error when setting parallelization = True in ZscoreSustain: "TypeError: cannot pickle '_abc._abc_data' object". This error is picked up ~60 times, always tracing to the pickle.py or the _dill.py files within the pysustain package. This has happened using both Jupyter Notebook and Spyder, however I can run Sustain fine when parallelization is set to False.

File "[pysustain package location]/lib/python3.10/site-packages/spyder_kernels/py3compat.py", line 356, in compat_exec
  exec(code, globals, locals)

File [notebook], line 231, in <module>
  prob_subtype_stage  = sustain_input.run_sustain_algorithm()

File "[pysustain package location]/lib/python3.10/site-packages/pySuStaIn/AbstractSustain.py", line 164, in run_sustain_algorithm
  ml_likelihood_mat_EM        = self._estimate_ml_sustain_model_nplus1_clusters(self.__sustainData, ml_sequence_prev_EM, ml_f_prev_EM) #self.__estimate_ml_sustain_model_nplus1_clusters(self.__data, ml_sequence_prev_EM, ml_f_prev_EM)

File "[pysustain package location]/lib/python3.10/site-packages/pySuStaIn/AbstractSustain.py", line 615, in _estimate_ml_sustain_model_nplus1_clusters
  ml_likelihood_mat               = self._find_ml(sustainData)

File "[pysustain package location]/lib/python3.10/site-packages/pySuStaIn/AbstractSustain.py", line 704, in _find_ml
  pool_output_list                    = self.pool.map(partial_iter, seed_sequences.spawn(self.N_startpoints))

File "[pysustain package location]/lib/python3.10/site-packages/pathos/multiprocessing.py", line 135, in map
  return _pool.map(star(f), zip(*args)) # chunksize

File "[pysustain package location]/lib/python3.10/site-packages/multiprocess/pool.py", line 367, in map
  return self._map_async(func, iterable, mapstar, chunksize).get()

File "[pysustain package location]/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get
  raise self._value

File "[pysustain package location]/lib/python3.10/site-packages/multiprocess/pool.py", line 540, in _handle_tasks
  put(task)

File "[pysustain package location]/lib/python3.10/site-packages/multiprocess/connection.py", line 214, in send
  self._send_bytes(_ForkingPickler.dumps(obj))

File "[pysustain package location]/lib/python3.10/site-packages/multiprocess/reduction.py", line 54, in dumps
  cls(buf, protocol, *args, **kwds).dump(obj)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 394, in dump
  StockPickler.dump(self, obj)

File "[pysustain package location]/lib/python3.10/pickle.py", line 487, in dump
  self.save(obj)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 560, in save
  f(self, obj)  # Call unbound method with explicit self

File "[pysustain package location]/lib/python3.10/pickle.py", line 902, in save_tuple
  save(element)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 560, in save
  f(self, obj)  # Call unbound method with explicit self

File "[pysustain package location]/lib/python3.10/pickle.py", line 887, in save_tuple
  save(element)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 560, in save
  f(self, obj)  # Call unbound method with explicit self

File "[pysustain package location]/lib/python3.10/pickle.py", line 887, in save_tuple
  save(element)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 560, in save
  f(self, obj)  # Call unbound method with explicit self

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 1824, in save_function
  _save_with_postproc(pickler, (_create_function, (

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 1089, in _save_with_postproc
  pickler.save_reduce(*reduction)

File "[pysustain package location]/lib/python3.10/pickle.py", line 692, in save_reduce
  save(args)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 560, in save
  f(self, obj)  # Call unbound method with explicit self

File "[pysustain package location]/lib/python3.10/pickle.py", line 887, in save_tuple
  save(element)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 603, in save
  self.save_reduce(obj=obj, *rv)

File "[pysustain package location]/lib/python3.10/pickle.py", line 692, in save_reduce
  save(args)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 560, in save
  f(self, obj)  # Call unbound method with explicit self

File "[pysustain package location]/lib/python3.10/pickle.py", line 887, in save_tuple
  save(element)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 560, in save
  f(self, obj)  # Call unbound method with explicit self

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 1427, in save_instancemethod0
  pickler.save_reduce(MethodType, (obj.__func__, obj.__self__), obj=obj)

File "[pysustain package location]/lib/python3.10/pickle.py", line 692, in save_reduce
  save(args)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 560, in save
  f(self, obj)  # Call unbound method with explicit self

File "[pysustain package location]/lib/python3.10/pickle.py", line 887, in save_tuple
  save(element)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 560, in save
  f(self, obj)  # Call unbound method with explicit self

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 1824, in save_function
  _save_with_postproc(pickler, (_create_function, (

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 1084, in _save_with_postproc
  pickler._batch_setitems(iter(source.items()))

File "[pysustain package location]/lib/python3.10/pickle.py", line 998, in _batch_setitems
  save(v)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 560, in save
  f(self, obj)  # Call unbound method with explicit self

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 1698, in save_type
  _save_with_postproc(pickler, (_create_type, (

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 1070, in _save_with_postproc
  pickler.save_reduce(*reduction, obj=obj)

File "[pysustain package location]/lib/python3.10/pickle.py", line 692, in save_reduce
  save(args)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 560, in save
  f(self, obj)  # Call unbound method with explicit self

File "[pysustain package location]/lib/python3.10/pickle.py", line 902, in save_tuple
  save(element)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 560, in save
  f(self, obj)  # Call unbound method with explicit self

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 1186, in save_module_dict
  StockPickler.save_dict(pickler, obj)

File "[pysustain package location]/lib/python3.10/pickle.py", line 972, in save_dict
  self._batch_setitems(obj.items())

File "[pysustain package location]/lib/python3.10/pickle.py", line 998, in _batch_setitems
  save(v)

File "[pysustain package location]/lib/python3.10/site-packages/dill/_dill.py", line 388, in save
  StockPickler.save(self, obj, save_persistent_id)

File "[pysustain package location]/lib/python3.10/pickle.py", line 578, in save
  rv = reduce(self.proto)

TypeError: cannot pickle '_abc._abc_data' object

Thank you for your help!
Katrina

Hello and thank you very much for sharing your work!
I am sorry for the trouble, but I am not being able to find out how to run your model for multiple sclerosis outcome prediction on the pretrained pkl you provide.
Would it be possible for you to provide some explanation on how to do so, or a reference to some instructions you have already provided and I am not seeing. From the input required, up to the inference process and the expected output.
I thank you enormously in advance.
Lucia

Hey team,

Just a tiny one here. I did a fresh install of pySuStaIn recently and ran into a minor issue. The requirements.txt file lists "sklearn" but it should be "scikit-learn". This caused some issues with my install, which were resolved by making that small change to the requirements.txt file.

--Jake

Hello,

Is it possible to apply the optimal model (after CV) to external independent data? If so, can you please add functionality to this tutorial: https://github.com/ucl-pond/pySuStaIn/blob/master/notebooks/SuStaInWorkshop.ipynb

I assume we only need to z-score the new data relative to the CN data used for the training?

Thank you very much!

I have noticed an edge case where subtypes are labelled differently by the positional variance diagrams, and the model output.

I'm not 100% certain, but I think this is due to the subtype numbering in the PVDs being assigned according to maximum likelihood of the positional variance, whilst I think the ml_subtype number is assigned according to number of individuals per subtype. In rare cases a smaller subtype can have a higher maximum likelihood PVD. 😕

SusStaIn is a cool idea for extracting models from cross-sectional data, but one idea I have is, if my data is longitudinal, I could constrain the possible progression models that are possible?

Can such a feature go into SuStaIn? Does it conceptually make sense?

We have lots of animal longitudinal developmental and interventional data which could be used to test this.

We should code up a script that implements 10-fold CV for selecting the optimal number of subtype (see the SuStaIn paper).
For each fold:

1. Calculate model likelihood for test and training sets;
2. Output/Save this, and optionally also plot.

Hi, Thanks for your great work!

I am working on your SuStaInWorkshop notebook tutorial. It plots positional variance diagrams to interpret the subtype progressions.

However, I am a little confused about how to read these diagrams.

1. What's different colors means?
2. What's the learned progression order for each subtype?
3. Why in demo subtype-1, Biomarker 3 becomes severe quickly and in demo subtype-2, Biomarker 3 becomes abnormal far later?

Grateful for your help again.

Hi, Thanks for releasing pySuStaIn !

I am interested in AD pathology and currently trying to reproduce your subtyping results on ADNI dataset (i.e. the resulst reported in your Nat.Comm. 2018 paper).

I am confused of two problems:

1. Some subjects in ADNI 3T and 1.5T dataset have longitudinal data (datapoints collected at different time). So, how did you decide which datapoint to use?
2. Can you provide the positional variance diagrams (containing all brain regions) learned by SuStaIn on ADNI 3T/1.5T dataset? Because the qualitative results of disease progression in your paper are rather rough to understand detailed pathology.

Grateful for your help 😁

I was running cross-validation in parallel on a cluster using cross_validate_sustain_model() with argument select_fold set to the CV fold desired for each compute job.

I noticed that all 10 folds were returning results for only fold0.

The culprit is line 276, where the loop is through range(Nfolds) (where Nfolds=len(select_fold)) rather than explicitly through the select_fold array itself.

Will send a PR to fix shortly, but wanted to raise this in case others have the same problem

Hi Leon, Peter, others,

I tried running Sustain on a sporadic AD dataset and it ran well for around 8 hours or so until it crashed with the following error. It seems like some corner case scenario which doesn't occur quite often. Would really appreciate your help in addressing this issue.

Splitting cluster 1 of 3
 + Resolving 2 cluster problem
 + Finding ML solution from hierarchical initialisation
- ML likelihood is [-4233.34529467]
Splitting cluster 2 of 3
 + Resolving 2 cluster problem
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-bce0b47cba4f> in <module>
     65                               dataset_name,False)
     66 
---> 67 sustain_input.run_sustain_algorithm()

~/anaconda3/lib/python3.8/site-packages/pySuStaIn/AbstractSustain.py in run_sustain_algorithm(self)
    142                 ml_sequence_mat_EM, \
    143                 ml_f_mat_EM,        \
--> 144                 ml_likelihood_mat_EM        = self._estimate_ml_sustain_model_nplus1_clusters(self.__sustainData, ml_sequence_prev_EM, ml_f_prev_EM) #self.__estimate_ml_sustain_model_nplus1_clusters(self.__data, ml_sequence_prev_EM, ml_f_prev_EM)
    145 
    146                 seq_init                    = ml_sequence_EM

~/anaconda3/lib/python3.8/site-packages/pySuStaIn/AbstractSustain.py in _estimate_ml_sustain_model_nplus1_clusters(self, sustainData, ml_sequence_prev, ml_f_prev)
    584 
    585                     print(' + Resolving 2 cluster problem')
--> 586                     this_ml_sequence_split, _, _, _, _, _ = self._find_ml_split(sustainData_i)
    587 
    588                     # Use the two subtype model combined with the other subtypes to

~/anaconda3/lib/python3.8/site-packages/pySuStaIn/AbstractSustain.py in _find_ml_split(self, sustainData)
    695 
    696         if ~isinstance(pool_output_list, list):
--> 697             pool_output_list                = list(pool_output_list)
    698 
    699         ml_sequence_mat                     = np.zeros((N_S, sustainData.getNumStages(), self.N_startpoints))

~/anaconda3/lib/python3.8/site-packages/pySuStaIn/AbstractSustain.py in _find_ml_split_iteration(self, sustainData, seed_num)
    740 
    741             temp_seq_init                   = self._initialise_sequence(sustainData)
--> 742             seq_init[s, :], _, _, _, _, _   = self._perform_em(temp_sustainData, temp_seq_init, [1])
    743 
    744         f_init                              = np.array([1.] * N_S) / float(N_S)

~/anaconda3/lib/python3.8/site-packages/pySuStaIn/AbstractSustain.py in _perform_em(self, sustainData, current_sequence, current_f)
    826             candidate_sequence,     \
    827             candidate_f,            \
--> 828             candidate_likelihood            = self._optimise_parameters(sustainData, current_sequence, current_f)
    829 
    830             HAS_converged                   = np.fabs((candidate_likelihood - current_likelihood) / max(candidate_likelihood, current_likelihood)) < 1e-6

~/anaconda3/lib/python3.8/site-packages/pySuStaIn/ZscoreSustain.py in _optimise_parameters(self, sustainData, S_init, f_init)
    237         p_perm_k_weighted                   = p_perm_k * f_val_mat
    238         p_perm_k_norm                       = p_perm_k_weighted / np.sum(p_perm_k_weighted, axis=(1,2), keepdims=True)
--> 239         f_opt                               = (np.squeeze(sum(sum(p_perm_k_norm))) / sum(sum(sum(p_perm_k_norm)))).reshape(N_S, 1, 1)
    240         f_val_mat                           = np.tile(f_opt, (1, N + 1, M))
    241         f_val_mat                           = np.transpose(f_val_mat, (2, 1, 0))

TypeError: 'int' object is not iterable

Thanks in advance.

Vikram

Hi all,

I just wanted to share some code here in case it may help anyone else: I've added a legend to the PVDs which shows the Z_val associated with each colour. It assumes that the Z_vals are the same for all biomarkers when creating the legend labels. I'm aware that there's an open issue on PVD colourbars and am not offering a solution to that problem, but simply a fix in the meantime to show clearly which value maps to which colour (I myself found this quite confusing to parse out when using more than 3 Z_vals as there was no documentation showing the mappings). It would be well paired with explanations I've seen used in papers about what the intensity, etc. of each colour indicates.

This requires an extra import from Matplotlib in ZscoreSustain: import matplotlib.patches as mpatches.

To implement it, add the following to plot_positional_var in ZscoreSustain between lines 660 (ax.set_title(title_i, fontsize=title_font_size)) and 661 (# Tighten up the figure). Further customization (i.e. legend size, location, etc) is possible through use of the ax.legend arguments.

                ax.set_xlabel(stage_label, fontsize=stage_font_size+2)
                ax.set_title(title_i, fontsize=title_font_size)
                
                # Add a legend 
                # adding an extra dim to colour mat for RGB reasons
                legend_colour_mat = np.array([[[1, 0, 0], [1, 0, 1], [0, 0, 1], [0.5, 0, 1], [0, 1, 1], [0, 1, 0.5]]])[:N_z]

                patches = [ mpatches.Patch(color=legend_colour_mat[0][i], label=f"Z_val = {zvalues[i]}") for i in range(len(zvalues)) ]

                # put those patches as legend-handles into the legend
                ax.legend(handles=patches, loc = "best" )
            
            # Tighten up the figure
            #plt.tight_layout()
            fig.tight_layout()

which I generated using:

import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import numpy as np
legend_colour_mat = np.array([[[1, 0, 0], [1, 0, 1], [0, 0, 1], [0.5, 0, 1], [0, 1, 1], [0, 1, 0.5]]])
ax = plt.subplot()
ax.imshow(legend_colour_mat)
ax.set_yticks([])
ax.set_xticks(range(6), ["1st Z_val", "2nd Z_val", "3rd Z_val", "4th Z_val", "5th Z_val", "6th Z_val"])
plt.show()

Before running SuStaIn using my own data, I wanted to run SuStaIn using the workshop file. Everything went smoothly except for when I want to plot the positional variance diagrams (under "Evaluate subtypes" and "choosing the optimal number of subtypes"). Python then throws the following error at me:

IndexError Traceback (most recent call last)
in
1 N_S_selected = 2
2
----> 3 pySuStaIn.ZscoreSustain._plot_sustain_model(sustain_input,samples_sequence,samples_f,M,subtype_order=(0,1))
4 _ = plt.suptitle('SuStaIn output')
5

~\xxx\pySuStaIn\ZscoreSustain.py in _plot_sustain_model(self, *args, **kwargs)
450
451 def _plot_sustain_model(self, *args, **kwargs):
--> 452 return ZscoreSustain.plot_positional_var(*args, Z_vals=self.Z_vals, **kwargs)
453
454 def subtype_and_stage_individuals_newData(self, data_new, samples_sequence, samples_f, N_samples):

~\xxx\pySuStaIn\notebooks\pySuStaIn\ZscoreSustain.py in plot_positional_var(samples_sequence, samples_f, n_samples, Z_vals, biomarker_labels, ml_f_EM, cval, subtype_order, biomarker_order, title_font_size, stage_font_size, stage_label, stage_rot, stage_interval, label_font_size, label_rot, cmap, biomarker_colours, figsize, separate_subtypes, save_path, save_kwargs)
622 # Shuffle vals according to subtype_order
623 # This defaults to previous method if custom order not given
--> 624 vals = temp_mean_f[subtype_order]
625
626 if n_samples != np.inf:

IndexError: too many indices for array

Could it be that there is something wrong with the dimensions of the array? When I get rid of ",subtype_order=(0,1))" (line 3), I get at least part of the output. Same applies to when I want to plot the positional variance diagrams before crossvalidation.

Any hint would be very welcome.

Kind regards

Hiya Leon et al.,

I ran into an interesting issue when the sys admin of my cluster reached out about some problematic processes that were instantiated when running a parallelized version of the SuStaIn cross-validation. The wrapper script looked something like this:

from pySuStaIn import Zscore SuStaIn
import multiprocessing as mp

sustain_input = ZscoreSuStaIn(args)
test_idxs = <a list of lists>

jobs = []
NFolds = 10
for fold in range(NFolds):
    p = mp.Process(target = target = sustain_input.cross_validate_sustain_model,
                             args = (test_idxs,fold))
    jobs.append(p)
    p.start()

This script was then submitted to the cluster with an .sh script specifying some parameters, such as the number of nodes and cores (in this case I asked for 1 node and 32 cores). However, it seems that individual jobs were themselves starting several other threads/processes. In this sense, they were overriding the specifications on my .sh script. The result was me asking for 32 cores, but having 32^2 threads running on the node. This results in many context switches and inefficient use of the processors on the node.

I admit this is kind of a niche issue and maybe folks don't care so much about how efficient the code is. But I think this issue might be surmounted quite easily by allowing an argument where the user can control the internal parallelization to some degree, a la the n_jobs framework in sklearn. As is, the parallel qualities do not seem to be controllable by the user.

Forgive me if this isn't clear. Would be happy to provide greater detail!

As always, thanks for such making this amazing library!

Hi all,

Thanks for your help with my past issue!

I'm now encountering a new error within the AbstractSuStaIn package that seems to relate to the staging portion of the algorithm:

This error is happening both locally and on a remote computing cluster. I've already added in an assert that none of the data going into SuStaIn contains NaNs, and ensured that my Z_vals are all integers (I am using Zscore SuStaIn). Do you have any ideas of what may be causing this issue or how to solve it?

As a related question, my research group and I are wondering why negative z-scores are not allowed in SuStaIn, and best practices handle them. We are currently shifting the z-score distribution to the right to ensure all values are > 0, but this means that we are losing the interpretability of z = 0, etc. Do you have any advice?

Thank you!

Hi all,
After I initialize the ZSuStaIn model, I run into the following Index out of bounds exception:

Finding ML solution to 1 cluster problem
/camhpc/mspaths/data/dev/pySuStaIn/pySuStaIn/ZscoreSustain.py:238: RuntimeWarning: invalid value encountered in true_divide
  p_perm_k_norm                       = p_perm_k_weighted / np.sum(p_perm_k_weighted, axis=(1,2), keepdims=True)
Traceback (most recent call last):
  File "try0_mspaths_sustain.py", line 151, in <module>
    sustain_input.run_sustain_algorithm()
  File "/camhpc/mspaths/data/dev/pySuStaIn/pySuStaIn/AbstractSustain.py", line 144, in run_sustain_algorithm
    ml_likelihood_mat_EM        = self._estimate_ml_sustain_model_nplus1_clusters(self.__sustainData, ml_sequence_prev_EM, ml_f_prev_EM) #self.__estimate_ml_sustain_model_nplus1_clusters(self.__data, ml_sequence_prev_EM, ml_f_prev_EM)
  File "/camhpc/mspaths/data/dev/pySuStaIn/pySuStaIn/AbstractSustain.py", line 552, in _estimate_ml_sustain_model_nplus1_clusters
    ml_likelihood_mat               = self._find_ml(sustainData)
  File "/camhpc/mspaths/data/dev/pySuStaIn/pySuStaIn/AbstractSustain.py", line 643, in _find_ml
    pool_output_list                = list(pool_output_list)
  File "/camhpc/mspaths/data/dev/pySuStaIn/pySuStaIn/AbstractSustain.py", line 676, in _find_ml_iteration
    _                               = self._perform_em(sustainData, seq_init, f_init)
  File "/camhpc/mspaths/data/dev/pySuStaIn/pySuStaIn/AbstractSustain.py", line 828, in _perform_em
    candidate_likelihood            = self._optimise_parameters(sustainData, current_sequence, current_f)
  File "/camhpc/mspaths/data/dev/pySuStaIn/pySuStaIn/ZscoreSustain.py", line 304, in _optimise_parameters
    this_S                      = this_S[0, :]
IndexError: index 0 is out of bounds for axis 0 with size 0

My current ZSustaIn class initialization is as followed:

# Start pySuStaIn
N = 4 # number of biomarkers
SuStaInLabels = ['Bio1', 'Bio2', 'Bio3', 'Bio4'] # biomarker labels

unt_data = np.vstack((biom_1, biom_2, biom_3, biom_4)) # each biomarker z-scored
data = np.transpose(unt_data) # data.shape --> (5123, 4) # data.shape returns (5123, 4)
Z_vals = np.array([[1,2,3]]*N)  # Z-score stage threshold # Z_vals.shape return (4,3)
Z_max = np.array([np.max(biom_bpf), np.max(biom_t2les),
                  np.max(biom_cgmf), np.max(biom_dgmf)]) # Z_max.shape returns (4,)


# Snipper of my prepared data for SuStaIn:
N_S_gt = 3 # Number of ground truth subtypes
N_startpoints = 10
N_S_max = N_S_gt+1
N_iterations_MCMC = int(1e4)
output_folder = os.path.join(os.path.dirname(__file__), 'rp_mspaths_sim')
if os.path.isdir(output_folder) is False:
    os.mkdir(output_folder)
dataset_name = 'sim'
sustain_input = ZscoreSustain(data,
                              Z_vals,
                              Z_max,
                              SuStaInLabels,
                              N_startpoints,
                              N_S_max,
                              N_iterations_MCMC,
                              output_folder,
                              dataset_name,
                              False)
sustain_input.run_sustain_algorithm() ## 

After spending some time attempting to debug, I believe the issue is related to the following functions returning a nan array type:
f_opt = (np.squeeze(sum(sum(p_perm_k_norm))) / sum(sum(sum(p_perm_k_norm)))).reshape(N_S, 1, 1)

    def _optimise_parameters(self, sustainData, S_init, f_init):
        # Optimise the parameters of the SuStaIn model

        M                                   = sustainData.getNumSamples()   #data_local.shape[0]
        N_S                                 = S_init.shape[0]
        N                                   = self.stage_zscore.shape[1]

        S_opt                               = S_init.copy()  # have to copy or changes will be passed to S_init
        f_opt                               = np.array(f_init).reshape(N_S, 1, 1)
        f_val_mat                           = np.tile(f_opt, (1, N + 1, M))
        f_val_mat                           = np.transpose(f_val_mat, (2, 1, 0))
        p_perm_k                            = np.zeros((M, N + 1, N_S))

        for s in range(N_S):
            p_perm_k[:, :, s]               = self._calculate_likelihood_stage(sustainData, S_opt[s])

        p_perm_k_weighted                   = p_perm_k * f_val_mat
        p_perm_k_norm                       = p_perm_k_weighted / np.sum(p_perm_k_weighted, axis=(1,2), keepdims=True)
        f_opt                               = (np.squeeze(sum(sum(p_perm_k_norm))) / sum(sum(sum(p_perm_k_norm)))).reshape(N_S, 1, 1)
        f_val_mat                           = np.tile(f_opt, (1, N + 1, M))
        f_val_mat                           = np.transpose(f_val_mat, (2, 1, 0))
        order_seq                           = np.random.permutation(N_S)  # this will produce different random numbers to Matlab
.
.
.

Now, this issue occurs randomly on the nth iteration, as replicated when running my script multiple times.
Any insights on how to fix/troubleshoot this problem?
Thanks in advance

Hello!

Parallelizing the cross-validation requires use of the "select_fold" argument. One might, for example, launch 10 instances of cross-validation, one for each of (say) 10 folds, in which case the given fold (say 3) would be passed to the select_fold argument. However, I've run into a few issues with this function.

First, line 218
if select_fold:
Many users will pass 0 to get the first fold. However, 0 will not fulfill the conditional statement if select_fold. There are many ways to fix this, but since the default setting for select_fold is [], my janky fix was just:
if select_fold != []:

The next issue comes in the following lines, 218-220

if select_fold:
    test_idxs  = test_idxs[select_fold]
Nfolds  = len(test_idxs)

I'm not sure what the intention was here, but the result is that Nfolds actually becomes the number of subjects in the test set. So, instead of having the desired 1 fold, you end up with N folds, where N is the number of subjects in the test set.

My solution here requires a few changes. First, lines 218-220 are changed to this:

if select_fold != []:
    Nfolds = 1
else:
    Nfolds = len(test_idxs)

Then, in order to disrupt the code as little as possible, I added the following lines under line 226. I include 226 below for reference:

for fold in range(Nfolds): 
    if select_fold != []:  # or whatever you change line 218 to
        fold = select_fold

Adding these three small changes resulted in the script working without issue for me, though maybe there are more elegant solutions. Thanks for bringing SuStaIn to Python!!

Dear SuStaIn friends,

I had encountered an error on a number of occasions when using SuStaIn on different datasets. The error itself looked something like this:

Traceback (most recent call last):
  File "/Users/jacobv/SuStaIn_workshop/lib/python3.7/site-packages/multiprocess/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/Users/jacobv/SuStaIn_workshop/lib/python3.7/site-packages/multiprocess/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/Users/jacobv/SuStaIn_workshop/lib/python3.7/site-packages/pathos/helpers/mp_helper.py", line 15, in <lambda>
    func = lambda args: f(*args)
  File "/Users/jacobv/SuStaIn_workshop/lib/python3.7/site-packages/pySuStaIn/AbstractSustain.py", line 709, in _find_ml_iteration
    _                               = self._perform_em(sustainData, seq_init, f_init, rng)
  File "/Users/jacobv/SuStaIn_workshop/lib/python3.7/site-packages/pySuStaIn/AbstractSustain.py", line 864, in _perform_em
    candidate_likelihood            = self._optimise_parameters(sustainData, current_sequence, current_f, rng)
  File "/Users/jacobv/SuStaIn_workshop/lib/python3.7/site-packages/pySuStaIn/ZscoreSustain.py", line 324, in _optimise_parameters
    this_S                      = this_S[0, :]
IndexError: index 0 is out of bounds for axis 0 with size 0.

As it turns out, this is caused by a divide by zero problem during the normalization of p_perm_k. This NaN then propagates forward a bit and doesn't turn up as an error until line 324, as shown. This is not itself necessarily caused by any outlying "bad values" (e.g. NaNs) in the original dataset, so it's quite hard (impossible?) to detect before running SuStaIn and getting the error.

This is apparently a known issue, as the following comment exists on line 333 of ZScoreSustain:
#adding 1e-250 fixes divide by zero problem that happens rarely

A few lines later at 335, the "corrected" line occurs:
p_perm_k_norm = p_perm_k_weighted / np.sum(p_perm_k_weighted + 1e-250, axis=(1, 2), keepdims=True)

However, at least in my case, the offending divide by zero problem occurred earlier. Note that the fix (ln 335) occurs before the error in my traceback (ln 324). Instead, the divide by zero problem occurs for me at line 238, which is incidentally the same calculation:
p_perm_k_norm = p_perm_k_weighted / np.sum(p_perm_k_weighted, axis=(1,2), keepdims=True)

By once again adding the "corrected" line, the problem is surmounted and I no longer get the error. I'm not sure how rare this issue really is, because this is maybe the third time I've encountered it (on different datasets). Requesting a patch to fix it, pretty please!

Thanks as always for this incredible software!! <3 <3 <3

This is minor in the grand scheme of things, but it might be important when creating clear figures for e.g. publication. I've toyed with a few ideas so thought I'd raise an issue on this before unilaterally merging one option for others to use and save them some time.

The Problem

Adding a colourbar to the PVD for the mixture version is straightforward, as colour intensity equates to the certainty of that position. For the z-score version, however, it has two dimensions. While certainty of the colour for a single z-score event equates to certainty (e.g. from pure white to pure red), the colours also mix when different z-score events overlap (and they mix proportionally to their certainty). For example, if a single stage (for a single biomarker) has 50% certainty for z=1 and z=2, this square will be a 50:50 mix of red and magenta (both of which are themselves at 50% intensity). A single colourbar cannot (to me, at least) capture this.

The question is, if adding a colourbar is to be useful, which information is it best that it captures.

Proposals

Here's a few variants I made for this. Other suggestions are welcome.

Simple Colourbar

Gets the point across, but doesn't integrate intensity/certainty or z-score mixing.

Intensity Colourbar

Highlights the difference in intensity, but not z-score mixing.

Mixing Colourbar

Highlights z-score mixing, but not intensity.

The point of this is to add something so others don't need to do this themselves. If no-one feels strongly, I'll just pick one after a week or so to integrate.