arielletambini / denoiser

Nuisance removal tool for fMRI data

Denoiser: A nuisance regression tool for fMRI BOLD data

Denoiser is a tool for removing sources of noise from and performing temporal filtering of functional MRI data. It also provides visualization of the content of nuisance signals (including motion information, if provided), allowing the user to get a quick sense of data quality before and after noise removal.

Denoiser acts on 4D fMRI data (takes either a nifti file path or an already loaded nibabel object as input). Nuisance signal removal and temporal filtering are performed on a voxel-wise level, and a 'cleaned' 4D Nifti file (_NR file)/ nibabel object are created as outputs.

The specific noise signals to be removed are specified by the user (contained in a tsv file). This tool should be used only after BOLD data are minimally preprocessed (for example, after preprocessing the data using fmriprep, which creates a .tsv file containing nuisance signals).

For instructions on how to run, type: python run_denoise.py -h:

usage: run_denoise.py [-h] [--col_names COL_NAMES [COL_NAMES ...]] [--hp_filter HP_FILTER] [--lp_filter LP_FILTER] [--out_figure_path OUT_FIGURE_PATH] img_file tsv_file out_path

Function for performing nuisance regression. Saves resulting output nifti file, information about nuisance
regressors and motion (html report), and outputs nibabel object containing clean data

positional arguments:
  img_file              4d nifti img: file path or nibabel object loaded into memory
  tsv_file              tsv file containing nuisance regressors to be removed
  out_path              output directory for saving new data file

optional arguments:
  -h, --help            show this help message and exit
  --col_names COL_NAMES [COL_NAMES ...]
                        which columns of TSV file to include as nuisance regressors. defaults to ALL columns.
  --hp_filter HP_FILTER
                        frequency cut-off for high pass filter (removing low frequencies). Recommend .009 Hz
  --lp_filter LP_FILTER
                        frequency cut-off for low pass filter (removing high frequencies). Recommend .1 Hz for non-task data
  --out_figure_path OUT_FIGURE_PATH
                        output directory for saving figures. Defaults to location of out_path + _figures

If you run into troubles using the tool or have any questions please post them here.

denoiser's Issues

add high pass filtering

Via linear regression with cosine basis functions

Decimate the signal after low-pass filetering

Since low-pass filtering changes the effective TR it might be worth decimating (downsampling) the output to save space. This can be performed (in one step together with low-pass filtering) by one of the following functions:

• https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.resample_poly.html (FIR based)
• https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.resample.html (FFT based)

Important: the output file header information needs to be updated with the new effective TR

RuntimeWarning: Divide by zero

Hi , I run the denoiser and get the output file but not the html report due to the following error:

Computing dvars...
/home/amir/.local/lib/python3.8/site-packages/numpy/lib/shape_base.py:379: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
res = asanyarray(func1d(inarr_view[ind0], *args, **kwargs))
/home/amir/.local/lib/python3.8/site-packages/numpy/lib/shape_base.py:402: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
buff[ind] = asanyarray(func1d(inarr_view[ind], *args, **kwargs))
Traceback (most recent call last):
File "run_denoise.py", line 462, in
denoise(img_file, tsv_file, out_path, col_names, hp_filter, lp_filter, out_figure_path)
File "run_denoise.py", line 316, in denoise
temp = nac.compute_dvars(in_file=in_file, in_mask=mask_file)[1]
File "/home/amir/.local/lib/python3.8/site-packages/nipype/algorithms/confounds.py", line 1079, in compute_dvars
func_diff / np.array([diff_sdhat] * func_diff.shape[-1]).T
RuntimeWarning: divide by zero encountered in true_divide

Heavy Memory Usage?

Hi there,
I just found this package and was wondering about average memory usage?
I am trying to denoise a single subject and 32 GB RAM seem not enough, i.e., python throws me a MemoryError.
I just wanted to know if memory usage is simply very high or whehter there might be a leakage?
Thanks,
Freddy

Plot total R^2 map

Similar to #7.
Q: Should linear trend and bandpass regressors be included in this?

Problem running denoise

Hi,
I am trying to run denoise on my data but getting the following error:
Command:
'python run_denoise.py /home/data/OUT/derivatives/fmriprep/sub-101/ses-01/func/sub-101_ses-01_task-touch_run-1_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz /home/data/OUT/derivatives/fmriprep/sub-101/ses-01/func/sub-101_ses-01_task-touch_run-1_desc-confounds_timeseries.tsv /home/data/OUT/derivatives/fmriprep/sub-101/ses-01/func/ --col_names csf white_matter framewise_displacement --hp_filter 0.01'

Output:
/home/amiman/anaconda3/lib/python3.7/site-packages/nilearn/glm/init.py:56: FutureWarning: The nilearn.glm module is experimental. It may change in any future release of Nilearn.
'It may change in any future release of Nilearn.', FutureWarning)
/home/amiman/anaconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:17: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Mapping, defaultdict

of TRs: 296

Filling in 1 NaN value for framewise_displacement

of Confound Regressors: 3 [SELECTED regressors in CSV]

of High-pass Filter Regressors: 9

Saving output file...
Output directory: /home/data/OUT/derivatives/fmriprep/sub-101/ses-01/func/sub-101_ses-01_task-touch_run-1_space-MNI152NLin2009cAsym_desc-preproc_bold_NR_figures
FD not found: FD timeseries not plotted
Computing dvars...
Traceback (most recent call last):
File "run_denoise.py", line 463, in
denoise(img_file, tsv_file, out_path, col_names, hp_filter, lp_filter, out_figure_path)
File "run_denoise.py", line 376, in denoise
del fig, ax0, ax_curr, ax_d, dvars
UnboundLocalError: local variable 'ax0' referenced before assignment

Selected subset of input regressors

The input TSV file will most likely include more regressors than user wants. We should allow user to select which columns should be used.

Add command line interface

Use argparse module.

add low pass filtering

As an option

Add in full compcor functionality

Provide anatomical file also as input
Estimate CSF/WM nuisance signals using specified options (selection of number of principal components, task design option)

IndexError: index 240 is out of bounds for axis 0 with size 240

Hi!
Just wondering what I'm doing wrong here.

I run:
run_denoise.py sub-01_ses-01_task-rest_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz sub-01_ses-01_task-rest_desc-confounds_timeseries.tsv denoise --col_names a_comp_cor_00 a_comp_cor_01 a_comp_cor_02 a_comp_cor_03 a_comp_cor_04 a_comp_cor_05 framewise_displacement trans_x trans_y trans_z rot_x rot_y rot_z

I get this output:

/home/weberam2/.local/lib/python3.8/site-packages/nilearn/glm/init.py:55: FutureWarning: The nilearn.glm module is experimental. It may change in any future release of Nilearn.
warn('The nilearn.glm module is experimental. '
'# of TRs: 240'
Filling in 1 NaN value for framewise_displacement
'# of Confound Regressors: 13 [SELECTED regressors in CSV]'
No High-pass Filter Applied
Saving output file...
Output directory: denoise/sub-01_ses-01_task-rest_space-MNI152NLin2009cAsym_desc-preproc_bold_NR_figures
Traceback (most recent call last):
File "/home/weberam2/Scripts/denoiser/run_denoise.py", line 462, in
denoise(img_file, tsv_file, out_path, col_names, hp_filter, lp_filter, out_figure_path)
File "/home/weberam2/Scripts/denoiser/run_denoise.py", line 244, in denoise
sc_idx[thr_idx][curr_idx] = True
IndexError: index 240 is out of bounds for axis 0 with size 240

The number of timepoints is indeed 240 for the .nii.gz file.
There are 241 rows in the .tsv file (as the first row is the column titles)
There are some n/a values in the first row... I wonder if this is where I'm getting the error...

Gather plots in HTML reports

Templating could be done via http://jinja.pocoo.org/docs/2.10/

Set up continuous integration

Preferably CircleCI with smoke tests.

Integrate all filtering and confound regression steps into one command

Re: Linquist 2019 and the thread below, may be best to do filtering and nuisance regression all at once with 3dTproject, which is implemented in nipype.

https://neurostars.org/t/testing-different-noise-models-output-from-fmriprep-unclear-results-with-ica-aroma/2352

https://onlinelibrary.wiley.com/doi/full/10.1002/hbm.24528

Plot a statistical map for unique variance from each regressor

plot_glass_brain from nilearn

Config files

Instead of a set of command line (see #5) switches user should be able to provide one reusable config file specifying:

• which regressors to use #10
• highpass cutoff #2
• lowpass cutoff #3

Citing the denoiser and contributors?

I have recently used this great tool to perform denoising on some functional data. I was wondering if you have a preferred or recommended method of acknowledging your contributions and citing the denoiser?

Thanks,
Natasha

ValueError: Digital filter critical frequencies must be 0 < Wn < 1

Hey!

I am having trouble running denoiser.. I get the following error: ValueError: Digital filter critical frequencies must be 0 < Wn < 1
My code is as follows:
run_denoise.py
$raw/$sub/$ses/func/sub-006_ses-01_task-rest_space-MNI152NLin6Asym_desc-smoothAROMAnonaggr_bold.nii.gz $raw/$sub/$ses/func/sub-006_ses-01_task-rest_desc-confounds_timeseries.tsv $raw/denoised
--col_names csf white_matter
--hp_filter 0.009
--lp_filter 0.8
--out_figure_path $raw/denoised/$sub/$ses

I specified the high pass and low pass filter options as suggested. I have a feeling that something goes wrong while selecting the high pass filter as of the following output that I get:

of TRs: 160

of Confound Regressors: 2 [SELECTED regressors in CSV]

of High-pass Filter Regressors: 0

/data/jcpottkamper/miniconda3/envs/fmriprep/bin/run_denoise.py:114: RuntimeWarning: divide by zero encountered in true_divide
Fs = 1. / TR

Could you help me please?

Cheers, Julia

Issue with DVARS calculation for AROMA, high resolution data

I've run Denoiser successfully on a number of datasets, and while the actual denoising step always works, the calculation of DVARS for the QA plot crashes when I denoise fMRIPrep data that was AROMA cleaned (I'm using the "MNI152NLin6Asym_desc-smoothAROMAnonaggr_bold" data).

If I resample the AROMA data from 2x2x2 mm (the required output) to 3x3x3 mm, DVARS calculation works fine.

Here is the error message

_Computing dvars...
Traceback (most recent call last):
File "/onrc/home/pipeline/data/pipelineb/home/Chyatt/onrc/data/Apps/denoiser-master/run_denoise.py", line 469, in
denoise(img_file, tsv_file, out_path, col_names, hp_filter, lp_filter, out_figure_path)
File "/onrc/home/pipeline/data/pipelineb/home/Chyatt/onrc/data/Apps/denoiser-master/run_denoise.py", line 317, in denoise
temp = nac.compute_dvars(in_file=in_file, in_mask=mask_file)[1]
File "/home/Chyatt/onrc/data/Apps/miniconda3/lib/python3.9/site-packages/nipype/algorithms/confounds.py", line 1057, in compute_dvars
ar1 = np.apply_along_axis(
File "<array_function internals>", line 5, in apply_along_axis
File "/home/Chyatt/onrc/data/Apps/miniconda3/lib/python3.9/site-packages/numpy/lib/shape_base.py", line 402, in apply_along_axis
buff[ind] = asanyarray(func1d(inarr_view[ind], *args, **kwargs))
File "/home/Chyatt/onrc/data/Apps/miniconda3/lib/python3.9/site-packages/nitime/algorithms/autoregressive.py", line 96, in AR_est_YW
ak = linalg.solve(Tm, y)
File "/home/Chyatt/onrc/data/Apps/miniconda3/lib/python3.9/site-packages/scipy/linalg/basic.py", line 219, in solve
_solve_check(n, info)
File "/home/Chyatt/onrc/data/Apps/miniconda3/lib/python3.9/site-packages/scipy/linalg/basic.py", line 29, in _solve_check
raise LinAlgError('Matrix is singular.')
numpy.linalg.LinAlgError: Matrix is singular.
_
The Matrix is singular issue only occurs with 2x2x2 data, not 3x3x3 resampled data (so far). I tried inputting a single fairly short fMRI data (only 310 images) that was 2x2x2 mm, to see if it was a memory issue (performed on a computer with 768 GB ram), but this error remained.

Great application! Thank you!

Add scrubbing option(s)

Add in optional inputs for scrubbing data

Take 4D nifti and TSV files, fit GLM, save residuals

Basic functionality.

TypeError: array() takes 1 positional argument but 2 were given

HI everyone,

When i'm using the "denoiser", I got the following error message: TypeError: array() takes 1 positional argument but 2 were given.
However, I still get output files including _NR file saved in my output folder. Can I still use the _NR files as cleaned data?

Add carpet plots

Before and after denoising. Reuse code from https://github.com/poldracklab/mriqc. Might be worth extracting the carpet plot code into nilearn and reuse in both projects.

Carpet plot

Hi,

Here is my script.

python3 run_denoise.py
--col_name a_comp_cor_00 a_comp_cor_01 a_comp_cor_02 a_comp_cor_03 a_comp_cor_04 a_comp_cor_05
t_comp_cor_00 t_comp_cor_01 t_comp_cor_02 t_comp_cor_03 t_comp_cor_04 t_comp_cor_05
dvars framewise_displacement
cosine00 cosine01 cosine02 cosine03 cosine04 cosine05
trans_x trans_y trans_z rot_x rot_y rot_z
[aroma_motion_components]
non_steady_state_outlier00 non_steady_state_outlier01 non_steady_state_outlier02
--hp_filter 0.015
--lp_filter 0.1
--out_figure_path $HOME/Desktop/resting_denoise_fig/sub-${subj}/denoised
$preproc_BOLD $confounds $HOME/Desktop/resting_denoise/sub-${subj}/denoised

Here is a sample of carpetplot

Is this what the clean_LP should look like? I thought it would look smoother (smooth transitions) than the input.

Thanks.

What's "_temp.nii" file?

Dear all,

I'm a beginner of Denoiser, and I find that sometimes Denoiser produces files named "_temp.nii" file (and of course, a "_NR.nii" file), and the number of figures is less (only 2 figures, Corr_matrix_regressors and Design_matrix) than usual.

My data have 3 runs. Usually, Denoiser produces "_NR.nii" file and 8 figures. (All data only were preprocessed by fMRIprep, no further processing)

In addition, sometimes I get an error, but not for all data/runs
jinja2.exceptions.TemplateNotFound:

My script is the following for selecting 3 confounds (Global Signals, CSF, and White_matter from fMRIprep) and Band_Pass (0.01 - 0.1Hz)

python run_denoise.py --col_names global_signal csf white_matter --hp_filter 0.01 --lp_filter 0.1 --out_figure_path $FIG $Nii $TSV $OUT

Does anyone know what makes the results different and how to fix them (if needed)?

Thanks!

Make it pip installable

Add setup.py

detailed guidelines

Can you please add more detailed instructions on how to run denoiser, what arguments it takes, and outputs?