eminserin / nbs-predict

A prediction-based extension of network-based statistics.

Hi 👋, I'm Emin

• 🧠 PhD candidate in Predictive Clinical Neuroscience working on applying machine learning in psychiatry at Charité, Berlin.
• 🏙️ Lives in Berlin, Germany.
• 👨‍💻 Interested in: developing user-friendly toolboxes to facilitate ML in psychiatry, graph theory, brain imaging
• 🌱 Currently learning: Federated Learning, Advanced Bayesian Methods.
• 💬 Love to talk about: computer science, psychiatry, technology, blockchain.

🔨 Stack

Languages:

• 🐍 Python
• MATLAB

Packages:

• 🤖 Machine Learning: scikit-learn, PyTorch, Keras
• 📊 Visualization: Matplotlib, Seaborn

nbs-predict's Issues

FC Matrix

Hi

Emin, I was trying to test your software but when I tried to import the FC matrix (333x333x57), it doesn't allow me to select it from the selection bar! (no option for selecting CSV or mat file). Is this normal?

Starting up troubles:

I got these errors when starting Matlab (fresh) and following the start-up instructions on your help wiki:

In matlab 2022

In matlab 2018a

Interpretation of the results generated by NBS-predict

Dear Dr. Serin,
I am a psychiatrist form China. Recently, I am using functional conenctome to predict HAMD depression scores. I am seting the parameter to 10 fold, 10-repetad times, and the p value to 0.0005. The results showed the HAMD scores can be predicted with µScore: 0.155, σScore: 0.012 by using SVM method and with µScore: 0.147, σScore: 0.017 by using linear regression. However, I am confused about the results, cause when I choose the SVM regression and correlation method, I found the R squared value is 0.003, and r corelation is 0.155 for SVM method. While, when I choose the linear regression and correlation method, I found the R squared value is -0.122, and r correlation is 0.147 for linear correlation method. The brain regions presentd for the two different method are also quite different. Can you tell me which method shoul I choose, or I can choose randomly?

Second, the R squred for linear regression is -0.122 is unbelieveble. What's more, I can't tell it is a positive prediction or negative prediction of the HAMD scores. Can you tell me how to see if it is a positive corelation or negative corelation?

Looking forward for your reply. All the best,
Hua

CV length error

Hello,
Thank you for this wonderful resource.

I am running into this error: Error using tabular/length
Undefined function 'LENGTH' for input arguments of type 'table'. Use the height, width, or size
functions instead.

Error in gen_cvpartition (line 20)
leny = length(y);

I was wondering where I should start for troubleshooting?

Thanks,
Alex

Inconsistent Learning Performance with Default Tutorial Data and Parameters

Issue Description:

We are encountering an issue where the NBS-Predict algorithm is not learning effectively when using the default tutorial data and parameters. The output consistently shows average AUC scores around 0.5, which suggests that the model is not performing better than random chance.

Environment:

• NBS-Predict Version: 1.0.0-beta.10
• MATLAB Version: R2021b
• Operating System: SUSE 15.3

Steps to Reproduce our issue:

• Cloned the NBS-Predict repository from GitHub.
• Used the sample data provided in the NITRC link in the documentation.
• Followed the standard procedure as per the Tutorial_HCP.pdf file.
  • Ran the analysis with the following initial parameters:
  • Searching Algorithm: bayesOpt
  • METRIC: auc
  • Number of Folds: 10
  • Number of Repetitions: 10

Results: The algorithm consistently returned AUC scores around 0.5. With the default tutorial data and parameters, I expected the algorithm to learn effectively and provide AUC scores significantly different from 0.5, but the following is the MATLAB command window output:

ESTIMATOR: LogReg
Searching Algorithm: bayesOpt
METRIC: auc
Number of Folds: 10
Number of Repetitions: 10
-------------
|   Score   |
-------------
|   0.490   |
|   0.496   |
|   0.475   |
|   0.498   |
|   0.507   |
|   0.490   |
|   0.506   |
|   0.507   |
|   0.498   |
|   0.500   |
-------------
10x10 repeated-CV: µScore: 0.497, σScore: 0.010
The elapsed time is 61.308250 seconds.
Permutation testing is running! Permutations: 1000 

The confusion matrix also showed that the predictions were not reflect correct true positive and true negative values

To note: We have not modified the default settings or data in any significant way. The issue persists even after repeating the analysis on different computers with different OS, Matlab versions with different parameters.

Given our results we have the following questions:

Is there a known issue with the current version of NBS-Predict that might be causing this behaviour?
Could there be an issue with the data, the choice of model/hyperparameters, or a potential bug in the software?
Any assistance or guidance you could provide would be greatly appreciated.

Best regards,
Javier

help with error message

Using pre-computed F/p matrices.

Hi Emin,
Thanks for making this toolbox. I was wondering if pre-computed NBS t or P maps can be used for the ML/model selection part for the toolbox. Due to the requirements of the dataset I am using, Ive run an NBS using a gamma regression, so have pre-computed F and p matrices. I'm now trying to assess whether the significant component from the NBS is associated with a set of behavioural variables and thought the ML models and selection procedure proposed in the toolbox might be useful for this.
Thanks,
Sid

Unrecognized function or variable 'adjMat'.

Hello!

I have ran into a new error with your toolbox. It seems that the code has run into an error when reaching the visualization step.
Here was the error message:

Unrecognized function or variable 'adjMat'.

My parameters are slightly different than my last issue so here they are again:

Contrast: [1,-1,0,0]
ML Model: Auto
K-Fold: 10
Repeat CV: 10
P-value: 0.01
Seed: 42
CPU Cores: 10
Method: Grid Search

Please let me know if this is trouble on my end!

Estimate weights for confound regression in one group

Hi,

I am using NBS-predict to predict disease status (patients vs healthy controls) based on functional connectome data, with age and sex as nuisance variables.
Regarding confound regression, given the strong correlation between age and disease duration, I was afraid that estimating confound weights on the whole population might lead to regressing-out some disease-related effects from patients' data. Is this concern justified?

Would it be possible to estimate the weights only in the HCs and then regress-out the effect of physiological aging from all subjects?

Thank you in advance for your help and for this great resource!

Best,
Giuseppe

No graph produced when contrast [1 1]

Dear Emin,

If I try and run the toolbox by setting the contrast as [1 1] (which should be an F-test, or ANOVA, right?) I get some accuracy score but no meaningful graph, just a matrix where all connections are weighted 1. Is this an expected behavior?

Thank you,
Ramtin

One-class classification error

hello to everyone, unfortunately I encounter an error that I can't explain by simply comparing two groups.
I can only see the accuracy while for all the other parameters NBSPredict gives an error: "Warning: Only one class present! sensitivity does support one-class classification, and being set to NaN Please use Accuracy instead!"
thanks for any replies

Confusion matrix plotted incorrectly

The elements on the leading diagonal of confmat need to be switched for the confusion matrix to plot correctly.

confmat = fliplr(fliplr(confmat)');

in function confMatPush_Callback() works as a hack.

Error using classreg.learning.Linear.prepareDataCR: Empty X or Y not supported.

Hello! I have been recently trying to run your repository utilizing some data of my own. To prepare my data, I made sure that the connectivity matrices were in .mat format, downloaded the respective schaefer brain regions, and created a design matrix. I made sure that none of the files had missing values, and I am certain that none of them do. However, I have been running into the following error:

Error using classreg.learning.Linear.prepareDataCR
Empty X or Y not supported.

This simply appears when running the program and leaving it to run for approximately 5 minutes. Here are the parameters I used:
Contrast: [1,1,0,0]
ML Model: Auto
K-Fold: 10
Repeat CV: 10
P-value: 0.01
Seed: 42
CPU Cores: 10
Method: Grid Search

I have also attached the history.mat file and the screen recording of the error in the following google drive link:
https://drive.google.com/file/d/1D8b2xyoL1UzV01rP58n4XVGZtCJ2mhFQ/view?usp=sharing

Thank you!

Problem with compute_modelMetrics and NaN values in Matlab 2021b

Dear all,

I get a quite peculiar warning which is not reproducible in all machines. When using "balanced_accuracy", it returns warnings in several test models/designs:

"y_pred contains classes not in y_true."

It originates at line 156 of this function and only happens in Matlab versions newer than 2020a. I know why the warning occurs (the division some lines above) but I can not figure out why, even when tracing it back. You have any ideas?
Best,
David