Hello,

First of all thank you for this amazing pipeline! I am having an issue with the output when I run the COMMIT2 pipeline on my data. I am attaching two photos. The first is the dictionary that is created. As you can see the white matter boundaries are well aligned together. But when I look at the RMSE output, this is not the case. Can you please help me troubleshoot this problem?

Thanks in advance!

Ravi

Hi,

In the COMMIT 2 wiki when one is getting the tractogram that only has connecting fibers in the connectome, the tckedit command string is very long when using an atlas with a large amount of nodes (number of inputs is scaled by n^2). Many terminals will not be able to handle a string that large.

However, tckedit accepts wildcard operators, so the code block that looks like:

CMD = 'tckedit -force -nthreads 0'
for i in range(C.shape[0]):
    for j in range(i,C.shape[0]):
        if C[i,j] > 0 :
            CMD += ' bundles/bundle_%d-%d.tck' %(i+1,j+1)
os.system( CMD + ' demo01_fibers_connecting.tck' )

can simply be changed to:

os.system('tckedit -force -nthreads 0 bundles/bundle*.tck demo01_fibers_connecting.tck' )

Hopefully this helps,
Steven

For the case: normIC == group_sparsity and group_is_ordered == True.
The streamlines (in the .trk file) are sorted by group and the input structureIC contains the number of elements per group. If all groups have the same number of streamlines, the following error occurs when the function fit is executed:

commit/proximals.pyx in commit.proximals.prox_group_sparsity()
ValueError: Buffer has wrong number of dimensions (expected 1, got 2)

The Proximal operator of the infinity norm is:
prox_InfinityNorm(x) = x − Proj{NormL1 ≤ 1}(x)

I think the implementation is not complete.
@matteofrigo, can we check it??

At the moment, the estimates weights for each streamline are stored in file streamline_weights.txt with a 5-digit precision (5.13625E-2). This should be increased.

Hi there ,

I tried with global fiber tracking and deterministic fiber tracking as input for COMMIT and It worked well.
I am wondering whether it is possible to use probabilistic fiber tracking of FSL (fdt_paths.nii) as an input for COMMIT?

Best
Behnam

Hello.

Before processing all the data using COMMIT2, I tried to compare several values to determine which regularization lambda and fiber shift to use.

However, even if the value of Lambda is changed (applying anatomic priors), the results of the binary connectome of COMMIT1 and COMMIT2 seemed to be almost no difference, so I am inquiring about it.

The code What I used is below.
test_lambda.txt

The first two pictures below are the binary connectome of COMMIT1 and COMMIT2 with fiber shift of 0.5 and lambda of 0.5 using one of the data I will process. The upper is COMMIT1 and the lower is COMMIT2.

The next two pictures are duplications that I followed the COMMIT2 tutorial that you wrote. The upper is COMMIT1 and the lower is COMMIT2.

From what I see, the results of your tutorial have difference (COMMIT2 is sparser) but the results of my data don't.

Could you give me some advices? It will be helpful!
And I'm sorry to bother you again 😭

Best,
Joseph.

Hi,

I've been trying to run this and I get a Segmentation fault during the "Exporting EC compartments" step while creating the dictionary from the tractogram. Do you have any idea how to debug this?

I saw that you had a COMMIT debugger. In order to compile it I had to add the flag -std=c++11 to the CMakeLists, but now when I run it I instantly get a seg fault... I am on Ubuntu 14.04.

If you have time, I can email you my data.

Thanks a lot,
Martin

Hi COMMIT group,

I am going to understand the idea of A matrix in voxels containing crossing fibers. My question is that:
for a given fiber, in a voxel containing crossing fibers, how COMMIT framework selects between different peaks?
As I show in the attached image, for the 1st fiber in 1st voxel, how COMMIT choose the 1st column of left A matrix instead of the 1st column of the right A matrix.(I think, this is the example which explained by Daducci during his presentation in ESMRMB conference).

in another word, I am wondering what the criteria is for selecting between 2 peaks in a voxel for a given fiber.

Best
Behnam

Hi,
To get to know the steps of running COMMIT2 I wanted to do the tutorial but it is giving me the following error when trying to run on the provided test dataset:

Traceback (most recent call last):
File "/Users/bkalkhov/Tractography/Tractography_performance_testing/Tests/Commit2test/Demo_data/test.py", line 84, in
regnorms = [commit.solvers.group_sparsity, commit.solvers.non_negative, commit.solvers.non_negative],
AttributeError: module 'commit.solvers' has no attribute 'group_sparsity'

If I look into the commit.solvers it indeed does not have attribute group_sparsity but also the commit.solvers.non_negative is not there... So I changed it to commit.solvers.omega_sparse_group_lasso and .....non_negativity as they seem to be in the proximal but then I get the following error:

Traceback (most recent call last):
File "/Users/bkalkhov/Tractography/Tractography_performance_testing/Tests/Commit2test/Demo_data/test.py", line 82, in
prior_on_bundles = commit.solvers.init_regularisation(
TypeError: init_regularisation() got an unexpected keyword argument 'regnorms'

Can you please help me figure this out?

Kind regards, Boaz

Following the last improvement of the code and due to the rotation in the affine of the DWI.nii the demo should use peaks_use_affine = True.

from commit import trk2dictionary
trk2dictionary.run(
    filename_trk   = 'LausanneTwoShell/fibers.trk',
    path_out       = 'LausanneTwoShell/CommitOutput',
    filename_peaks = 'LausanneTwoShell/peaks.nii.gz',
    filename_mask  = 'LausanneTwoShell/WM.nii.gz',
    fiber_shift    = 0.5,
    peaks_use_affine = True
)

Furthermore will be good to double check how the peaks and the tractography was generated.
In that case of mrtrix, in order to avoid problems in the orientation, the best option is to import the gradient table with the option -fslgrad.

Dear Dr Daducci,
I am applying COMMIT2 to my research. When my tractogram contains 1 million streamlines and my parcellation of the Gray Matter contains over 100 ROIs, I can't get a file like demo01_fibers_connection.tck using your method on my computer (Threadripper 2990WX and 32GB of memory). Are there too many tck files (streamlines in bundles) from the previous step, or are these tck files too large in general, or some other reason? What's the solution? If I want to organize these tck files step by step directly in MRTRIX3, how could I re-organize them as contiguous groups. Thank you very much.

Hello, experts.

When I ran Group the streamlines in bundles of your COMMIT2 wiki, unlike your tutorial tck files, my tck files(original and connecting tck file) had differences of the number of tracks (from 5M to about 3.3M).

Is it a natural result which you intend?
Is there any problem if I ignore it and proceed?

Any response will be helpful!

Best,
Joseph.

Hi COMMIT group,

I used COMMIT for ex-vivo optic chiasm dataset.
I just wondering why middle part, region containing crossing fibers, has higher value that other parts ?
In fact, I expect to get the same result that I got from NODDI for intracellular compartment map.

Thank you
Behnam

Hello.

I'm trying to use COMMIT2 to filter false positives in tck files based on MRtrix3.

The problem is that it took too long to fit model. (About 6h 21m each COMMIT 1 & 2)

The cause of the problem I expect is

1. Are 5 million fibers in tck too many?
2. Thread I used was only one.

The code I used is below.

import os
import commit
from commit import trk2dictionary


os.system( 'tck2connectome -force -nthreads 0 -assignment_radial_search 2 \
        -out_assignments test2_fibers_assignment.txt \
        dhollander_tractogram_SD_STREAM_100408.tck \
        Schaefer400_7Net_2mm_100408.nii.gz test2_commit2_connectome.csv' )


if not os.path.isdir( 'bundles' ) :
    os.mkdir( 'bundles' )

os.system( 'connectome2tck -force -nthreads 0 -exclusive -files per_edge \
    -keep_self dhollander_tractogram_SD_STREAM_100408.tck test2_fibers_assignment.txt \
        bundles/bundle_' )

C = np.loadtxt( 'test2_commit2_connectome.csv', delimiter=',' ) # NB: change 'delimiter' to suits your needs
CMD = 'tckedit -force -nthreads 0'
for i in range(C.shape[0]):
    for j in range(i,C.shape[0]):
        if C[i,j] > 0 :
            CMD += ' bundles/bundle_%d-%d.tck' %(i+1,j+1)
os.system( CMD + ' dhollander_tractogram_SD_STREAM_100408.tck' )


trk2dictionary.run(
    filename_tractogram = 'dhollander_tractogram_SD_STREAM_100408.tck',
    filename_mask       = 'wm_axis3_bin_125.nii.gz',
    fiber_shift         = 0.5
)


import amico
amico.util.fsl2scheme( 'bvals', 'bvecs', 'DWI.scheme' )


mit = commit.Evaluation('/local_raid1/03_user/sunghyoung/01_project/04_COMMIT', 'test2') # commit.Evaluation(STUDY_PATH, SUBJECT_PATH )

mit.load_data( 'data.nii.gz', 'DWI.scheme' )


mit.set_model( 'StickZeppelinBall' )
d_par       = 1.7E-3             # Parallel diffusivity [mm^2/s]
d_perps_zep = []                 # Perpendicular diffusivity(s) [mm^2/s]
d_isos      = [ 1.7E-3, 3.0E-3 ] # Isotropic diffusivity(s) [mm^2/s]
mit.model.set( d_par, d_perps_zep, d_isos )

mit.generate_kernels( regenerate=True )
mit.load_kernels()


mit.load_dictionary( 'COMMIT' ) #  should match a mask with your DWI data as the same geometry

mit.set_threads(1) # mit.set_threads()
mit.build_operator()


mit.fit( tol_fun=1e-3, max_iter=1000, verbose=False )
mit.save_results( path_suffix="_COMMIT1" )

x_nnls, _, _ = mit.get_coeffs()

C = np.loadtxt( 'test2_commit2_connectome.csv', delimiter=',' )
C = np.triu( C ) # be sure to get only the upper-triangular part of the matrix
group_size = C[C>0].astype(np.int32)

tmp = np.insert( np.cumsum(group_size), 0 , 0)
group_idx = np.array( [np.arange(tmp[i],tmp[i+1]) for i in range(len(tmp)-1)] )

group_w = np.empty_like( group_size, dtype=np.float64 )
for k in range(group_size.size) :
    group_w[k] = np.sqrt(group_size[k]) / ( np.linalg.norm(x_nnls[group_idx[k]]) + 1e-12 )

reg_lambda = 5e-4 # change to suit your needs

prior_on_bundles = commit.solvers.init_regularisation(
    mit,
    regnorms    = [commit.solvers.group_sparsity, commit.solvers.non_negative, commit.solvers.non_negative],
    structureIC = group_idx,
    weightsIC   = group_w,
    lambdas     = [reg_lambda, 0.0, 0.0]
)

mit.fit( tol_fun=1e-3, max_iter=1000, regularisation=prior_on_bundles, verbose=False )
mit.save_results( path_suffix="_COMMIT2" )

In conclusion, I have two questions.

1. Are there ways to reduce fitting time?

2. Because I don't want to compare COMMIT1 and COMMIT2, how can I only produce results for COMMIT2? Please comment my code.

I'm sorry if they are stupid questions. However, if you help me, it will be very helpful!

Best,
Joseph.

Hi,

We are applying commit2 to some data and noticed that the peaks.nii file is not explicitly used in the code provided in
the commit2 wiki (https://github.com/daducci/COMMIT/wiki/COMMIT2), unlike the standard commit code where it is specified in trk2dictionary.run - we just wanted to confirm that the peaks file is still required to run commit2.

Thanks,
Sid

Hello! I've been trying to go through the tutorial, and I'm at the point where I'm inputting trk2dictionary.run( filename_tractogram = 'demo01_fibers.tck', filename_peaks = 'peaks.nii.gz', filename_mask = 'WM.nii.gz', fiber_shift = 0.5, peaks_use_affine = True )
But it's not moving past this particular line:

- flipping axes : [ x=False, y=False, z=False ]

I was wondering if there was anything I could do to make this work?

Dear COMMIT group ,

I used COMMIT for whole human brain and the results were great.
Now, I am going to use COMMIT framework for ex-vivo optic chiasm data. But, the result does not look reasonable and Intracellular compartment map does match to Mask or DWI as can be seen in the attached image. I adjusted the parallel and perpendicular diffusivity for ex-vivo condition and checked the inputs for COMMIT (fibers, peaks, mask) but I could not find the reason for it.

I appreciate for any idea or suggestion about this problem.
Is it possible to send my data for more accurate evaluation ?

Best
Behanm

Dear Developer,
When I run step by step codes based on wiki page, I see an error:

trk2dictionary.run(
... filename_tractogram = 'demo01_fibers.tck',
... filename_peaks = 'peaks.nii.gz',
... filename_mask = 'WM.nii.gz',
... fiber_shift = 0.5,
... peaks_use_affine = True
... )
Traceback (most recent call last):
File "", line 6, in
File "commit/trk2dictionary/trk2dictionary.pyx", line 27, in commit.trk2dictionary.run
File "commit/trk2dictionary/trk2dictionary.pyx", line 108, in commit.trk2dictionary.run
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

I would be appreciated if you help me to fix this error.
Best,
Mohammad

Now that we increased the information related to solvers we should think about updating both the showed and returned informations.

Add the possibility to extract the estimated coefficients corresponding to either the original problem or the preconditioned one. At the moment, the choice is driven by the parameter doNormalizeKernels but, for some applications, it could be useful to decide which version to use. Perhaps, we could add an optional parameter to the get_coeffs() function to control this behavior. The default should be as it is right now.

Hi
In COMMIT, it seems like the peak orientation is expected to be in voxel coordinates (similar to the current interpretation of the scheme file), however if the nifti header contains an affine transformation with a rotational component (e.g. a rotated FOV in the scanner), peak-files generated with MRtrix will not be interpreted correctly. Conversely, if e.g. dipy is used, the peak orientation is assumed to be in voxel-coordinates (instead of world coordinates) and are therefore interpreted correctly in trk2dictionary and in the COMMIT_debugger.
It might be beneficial to clarify the used conventions and/or add an additional flag to enable users to apply the affine transformation in trk2dictionary. Furthermore, it might be helpful to also state in the documentation that the scheme-file is expected to contain gradient directions in the voxel-coordinate system.

I am trying to install Commit on a Debian 9 machine with anaconda python. I followed the installation instruction (AMICO python code ) and then just did pip install . on COMMIT directory. I have no error but when I try to import commit from my python shell I have
ImportError : No module named core

If you have any idea of what I did wrong or a hint for the error. I precise that import amico worked perfectly.

I am going to use COMMIT framework for fiber tracking, but I've faced with the following error
_importError:Buildingnmodule commit.operator.operator failed:permission denied: '/common/apps/.../operator/operator_withLUT.c
when I want to build an operator with command mit.build.operator()
I am grateful for any guidance to solve it.

Thank you
Behnam

It would be nice to save to the results.pickle file also the information concerning the optimization run, e.g. cost function, relative error etc at each iteration. This could help analyzing the performance of the optimizer offline. At the moment, these values are reported to screen only.

Hello,

I am trying to run COMMIT2 on preprocessed DWI and tractograms from QSIPrep, using the AAL atlas registered to native space. I get the following error when adapting the tutorial.

Code input

# import packages
import os
import numpy as np
import commit
from commit import trk2dictionary
import scipy.io
import amico
#commit.setup()

# Get names of relevant directories
bids = '/om4/group/gablab/data/HBN/'
derivatives = os.path.join(bids,'derivatives')
qsiprep_dir = os.path.join(derivatives,'qsiprep')
qsirecon_dir = os.path.join(derivatives,'qsirecon')
commit_outdir = os.path.join(derivatives,'commit_testing')
if not os.path.exists(commit_outdir):
    os.makedirs(commit_outdir)
    


# Get qsiprep/qsirecon output
sub = 'sub-NDARAA306NT2'
dwi_path = os.path.join(qsiprep_dir,sub,'dwi',sub+'_acq-64dir_space-T1w_desc-preproc_dwi.nii.gz')
bval_path = os.path.join(qsiprep_dir,sub,'dwi',sub+'_acq-64dir_space-T1w_desc-preproc_dwi.bval')
bvec_path = os.path.join(qsiprep_dir,sub,'dwi',sub+'_acq-64dir_space-T1w_desc-preproc_dwi.bvec')
wm_mask_path =  os.path.join(qsiprep_dir,sub,'anat',sub+'_label-WM_mask.nii.gz')
tractogram_path = os.path.join(qsirecon_dir,sub,'dwi',
                               sub+'_acq-64dir_space-T1w_desc-preproc_space-T1w_desc-tracks_ifod2.tck')
atlas_path = os.path.join(qsirecon_dir,sub,'dwi',
                               sub+'_acq-64dir_space-T1w_desc-preproc_space-T1w_desc-aal116_atlas.mif.gz') # AAL atlas

# Get only connecting fibers
os.system(('/om2/user/smeisler/anaconda3/envs/commit/bin/tck2connectome -force -nthreads 0 -assignment_radial_search 2 -out_assignments fibers_assignment.txt ' + tractogram_path + ' ' + atlas_path + ' connectome.csv'))
os.system(( '/om2/user/smeisler/anaconda3/envs/commit/bin/connectome2tck -force -nthreads 0 -exclusive -files per_edge -keep_self ' + tractogram_path + ' fibers_assignment.txt bundles/bundle_' ))
os.system('/om2/user/smeisler/anaconda3/envs/commit/bin/tckedit -force -nthreads 0 bundles/bundle*.tck fibers_connecting.tck' )

# Define dictionary
trk2dictionary.run(
    filename_tractogram = 'fibers_connecting.tck',
    filename_mask       = wm_mask_path)

amico.util.fsl2scheme( bval_path, bvec_path, 'DWI.scheme' )

# load the data
mit = commit.Evaluation( '.', '.' )
mit.load_data( dwi_path, 'DWI.scheme' )

# use a forward-model with 1 Stick for the streamlines and 2 Balls for all the rest
mit.set_model( 'StickZeppelinBall' )
d_par       = 1.7E-3             # Parallel diffusivity [mm^2/s]
d_perps_zep = []                 # Perpendicular diffusivity(s) [mm^2/s]
d_isos      = [ 1.7E-3, 3.0E-3 ] # Isotropic diffusivity(s) [mm^2/s]
mit.model.set( d_par, d_perps_zep, d_isos )

mit.generate_kernels( regenerate=True )
mit.load_kernels()

# create the sparse data structures to handle the matrix A
mit.load_dictionary( 'COMMIT' )
##### CODE DOES NOT GET PAST HERE #####

mit.set_threads()
mit.build_operator()

# perform the fit
mit.fit( tol_fun=1e-3, max_iter=1000, verbose=False )
mit.save_results( path_suffix="_COMMIT1" )

# Retrieve the streamline contributions estimated by COMMIT for later use in COMMIT2:
x_nnls, _, _ = mit.get_coeffs( get_normalized=False )

Terminal Output (beginning from amico.util.fsl2scheme)

amico.util.fsl2scheme( bval_path, bvec_path, 'DWI.scheme' )
-> Writing scheme file to [ DWI.scheme ]
'DWI.scheme'
>>> 
>>> # load the data
>>> mit = commit.Evaluation( '.', '.' )
>>> mit.load_data( dwi_path, 'DWI.scheme' )

-> Loading data:
	* Acquisition scheme:
		- diffusion-weighted signal
		- 129 samples, 2 shells
		- 1 @ b=0, 64 @ b=1000.0, 64 @ b=2000.0
	* Signal dataset:
		- dim    : 162 x 193 x 164 x 129
		- pixdim : 1.200 x 1.200 x 1.200
		- values : min=0.00, max=1144.45, mean=11.99
   [ 20.4 seconds ]

-> Preprocessing:
	* Normalizing to b0... [ min=0.00, max=308603.06, mean=0.49 ]
   [ 1.4 seconds ]
>>> 
>>> # use a forward-model with 1 Stick for the streamlines and 2 Balls for all the rest
>>> mit.set_model( 'StickZeppelinBall' )
>>> d_par       = 1.7E-3             # Parallel diffusivity [mm^2/s]
>>> d_perps_zep = []                 # Perpendicular diffusivity(s) [mm^2/s]
>>> d_isos      = [ 1.7E-3, 3.0E-3 ] # Isotropic diffusivity(s) [mm^2/s]
>>> mit.model.set( d_par, d_perps_zep, d_isos )
>>> 
>>> mit.generate_kernels( regenerate=True )

-> Simulating with "Stick-Zeppelin-Ball" model:
   |������������������������������������������������������������                                       | 3   |������������������������������������������������������������������������������������������������������������������������������������������������������������������������������| 100.0%
   [ 2.0 seconds ]
>>> mit.load_kernels()

-> Resampling LUT for subject ".":
	* Keeping all b0 volume(s)...
   |������������������������������������������������������������                                       | 3   |������������������������������������������������������������������������������������������������������   |������������������������������������������������������������������������������������������������������   |������������������������������������������������������������������������������������������������������������������������������������������������������������������������������| 100.0%
	  [ OK ]
	* Normalizing... [ OK ]
   [ 3.7 seconds ]
>>> 
>>> # create the sparse data structures to handle the matrix A
>>> mit.load_dictionary( 'COMMIT' )

-> Loading the dictionary:
[ WARNING ] Dictionary does not have the same geometry as the dataset
	* Segments from the tracts... [ 8328748 fibers and 1495561589 segments ]
	* Segments from the peaks...  [ 0 segments ]
	* Isotropic contributions...  [ 446120 voxels ]
	* Post-processing...          Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "commit/core.pyx", line 359, in commit.core.Evaluation.load_dictionary
  File "commit/core.pyx", line 497, in commit.core.Evaluation.load_dictionary
IndexError: index 5138709 is out of bounds for axis 0 with size 5127624

Please let me know what other information I can provide to help.

Thanks,
Steven

The current progress bar doesn't display properly in Jupyter notebook/labs. We should consider using the tqdm progress bar.

When testing the code by running, in the same terminal session, COMMIT multiple times and changing each time some parameters such as the number of threads, calling build_operator() does not force recompilation which, in turnm, can raise a segmentation fault error.

Hi there ,

I am using COMMIT.It works amazing.
I am wondering whether it is possible to get the weight of fibers after implementation of COMMIT in order to see which fibers have higher and which fibers have lower weight.

Best
Maarkez

I report in the following an excerpt from the email I received from a user (Stefan Sommer) about an issue (and possible solution) with files larger than 2GB:

Hi Ale,

I figured my trk2dictionary on windows (the python compiled c-code) has troubles with track-files >2 GB. The problem is buried in the fseek function and some internally used offset variable (I think 32bit int signed, that would make sense for the 2 GB). Maybe on a different system with another compiler this might not be an issue (32/64bit, even though I am on a 64bit system but maybe it's compiled 32bit? I think I am using the visual studio compiler on windows) but it might be worth a check on other systems if that is an issue.

Anyway here is my quick fix using a mostly undocumented modified fseek:

• include this line in the top part of trk2dictionary_c.cpp:
  extern "C" int __cdecl _fseeki64(FILE *, __int64, int);
• substitute the two fseek calls in the read_fiber() function at the end of the file with _fseeki64() (there are two calls like that):
  _fseeki64(fp,4*ns,SEEK_CUR);

Just wanted to inform you if somebody ran into similar troubles or will in the future.

trk2dctionary just hang for "no reason" but I couldn't identify a malicious fiber. There is a while loop in fiberForwardModel() where the segments are checked and splitted for voxel intersection and after 2GB the file-offset variable probably wrapped around to -2^16 reading any non-sense memory and interpret those values as segment coordinates, typically almost +/- inf or 0. So in the extreme cases, the segments were just split forever and the while loop never terminated.
There's a break condition for the segment length if it's <1e-3, maybe it's a good idea to also limit the upper segment length in case of some very wrong fiber coordinate. In the >2GB case this leads to a segmentation fault and "at least" python crashes (caused by a segmentation fault I guess). Not sure what solution is "better" in that case though ;-)

Hi ,

I want to apply COMMIT framework on a diffusion MRI data set. I need help for generation of peak file.
Could you give me information about this file and how I can create it.

Best
Zarandi

Hello, COMMIT2 experts!
I'm sorry to bother you again. 😅

Thanks to your help in the past, I could apply COMMIT2 to my HCP data.

However, although I could run COMMIT2 in FACT and SD STREAM algorithms, the following error occurs for exactly the same data except for the iFOD2 algorithm in the tckgen command.

I tried the solutions you suggested in other people's same issues in the past, but it didn't work.

1. https://github.com/daducci/COMMIT/issues/110

   • Matching the geometry of white matter mask and DWI data
     

2. https://github.com/daducci/COMMIT/issues/92

   • Using bStep parameter (bStep = 10)
   • When I called commit.Evaluation I specified the STUDY_path and the SUBJECT_path
     

Is it because iFOD2 is probabilistic algorithm?

Do you have any idea to resolve this problem?
Any comments will help me.

Thanks!
Joseph.

The PR #13 introduced a bug in the reordering of the coefficients.

In the fit() function (and all functions inside solvers.py called from there) the parameter verbose is an integer whose value is 0 or 1, and its value is checked as if verbose>=1. I think it would be more intuitive to set verbose to be a boolean.

I tried to run the example, but it assumes the nibabel 2.0 command for the affine (without expliciting the version requirements in the install). The old 1.0 way of doing it is get_affine, but that is deprecated according to http://nipy.org/nibabel/reference/nibabel.spatialimages.html

So, as a workaround, I suggest either

• putting nibabel 2.0 as a dependency
• checking for nibabel version with nibabel.version and use affine for 2.0 (like it is currently doing) and use get_affine for nibabel 1.

Second option won't force people to upgrade, but requires some code edit.

Hello,

I get the following error when trying to import commit:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [6], in <module>
      1 import os
      2 import numpy
----> 3 import commit
      4 from commit import trk2dictionary

File /om2/user/smeisler/anaconda3/envs/commit/lib/python3.9/site-packages/commit/__init__.py:3, in <module>
      1 __all__ = ['core','models','solvers','trk2dictionary']
      2 from .info import __version__
----> 3 from .core import Evaluation, setup

File commit/core.pyx:1, in init commit.core()

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Steps to replicate

1. Create new conda environment with python 3.9, ipykernel, and some compiler libraries needed to install pip packages
   conda create -n commit -c conda-forge python=3.9 gxx_linux-64 gcc gcc_linux-64 isl libblas ipykernel liblapack
2. install dmri-commit: pip install dmri-commit
3. error occurs upon importing commit with import commit

Troubleshooting steps

I have tried using pip to uninstall and reinstall numpy, with different versions (one 1.21 installation, and one 1.22 installation)

I am on a Linux HPC running CentOs 7.6. Please let me know if any other information would help.

Thank you,
Steven

IT would be nice to add integration with the Travis IC platform.

The nibabel.trackvis interface is deprecated and will be removed in version 4 of Nibabel.

In its place, it is recommended to use nibabel.streamlines.

You could also use the new dipy.io.streamlines including StatefulTractogram, which provides more space checks.

This is the same issue as in AMICO (daducci/AMICO#89). Dipy 1.0 drops support for python 2, and therefore removes the dipy.utils.six module and functions.

The same fixes should be applied as in AMICO.

Hi,

I am trying to run the COMMIT program using my data. But I have some questions with the input tractograms.

I found the description of main features of COMMIT: "Accepts and works with any input tractogram (i.e. set of fiber tracts)." And the example program and data showed the use of Trackvis ('.trk'). However, my tracking results were obtained by in-house code using MATLAB (.mat). Do i need to transfer .mat to .trk? Or, Could COMMIT load .mat file directly? It would be appreciated if you could offer some examples.

Thanks,
.
Zhipeng Yang

Hi there ,

I am using COMMIT framework and have a question about the outputs of CylinderZeppelinBall model.

I used CylinderZeppelinBall model for my dataset and only got IC, EC and ISO maps but no axon diameter maps with mit.save_results(save_coeff = True) command. I am grateful if you could inform me how it is possible to have the axon diameter maps or diameter index for each fiber like this abstract : https://www.researchgate.net/publication/305996486_Assessment_of_bundle-specific_axon_diameter_distributions_using_diffusion_MRI_tractography

Best
Behnam

With PR #36 I introduced a bug regarding the backward compatibility of group sparsity. The conversion from a list of lengths of groups for the ordered tractogram to the required np.array is wrong.

Avoid using the ordered group structure until this issue is not solved. If you need to use group sparsity you can follow the example in the dedicated tutorial.

Hello,

I am trying to run COMMIT2 on my data preprocessed by QSIPrep and tractogram obtained from MRtrix3. I followed the tutorial but got an error when building the linear operator.

The tractogram was obtained using the iFOD2 algorithm, based on anatomically constrained tractography.

Code input

import os
import commit
from commit import trk2dictionary
import amico 
import numpy as np

# Directories
derivatives_dir="path/to/my/derivatives"
dwi_dir=os.path.join(derivatives_dir,"qsiprep")
fod_dir=os.path.join(derivatives_dir,"fod")
tracto_dir=os.path.join(derivatives_dir,"tractography")
connectomes_dir=os.path.join(derivatives_dir,"connectome")

sub="sub-016"

# Input files
dwi_path=os.path.join(dwi_dir,sub,f"dwi/{sub}_space-T1w_desc-preproc_dwi.nii.gz")
bval_path=os.path.join(dwi_dir,sub,f"dwi/{sub}_space-T1w_desc-preproc_dwi.bval")
bvec_path=os.path.join(dwi_dir,sub,f"dwi/{sub}_space-T1w_desc-preproc_dwi.bvec")
tracto_path=os.path.join(tracto_dir,sub,"tracks_10M.tck")

os.chdir(os.path.join(connectomes_dir,sub))

# Get only connecting fibers
if not os.path.isfile(os.path.join(connectomes_dir,sub,f"{sub}_parcels.csv")) :
    os.system( f'tck2connectome -force -nthreads 0 -assignment_radial_search 2 \
            -out_assignments fibers_assignment.txt \
            {tracto_path} \
            functional/hcp_mmp1_in_T1w_dwi.nii.gz {sub}_parcels.csv' )

if not os.path.isdir(os.path.join(connectomes_dir,sub,"bundles")) :
    os.mkdir(os.path.join(connectomes_dir,sub,"bundles"))
    os.system(( f'connectome2tck -force -nthreads 0 -exclusive -files per_edge -keep_self {tracto_path} assignments_{sub}_parcels.csv bundles/bundle_' ))

if not os.path.isfile( f'{tracto_dir}/{sub}/fibers_connecting.tck' ) :
    C = np.loadtxt( f'{sub}_parcels.csv', delimiter=',' ) # NB: change 'delimiter' to suits your needs
    CMD = 'tckedit -force -nthreads 0'
    for i in range(C.shape[0]):
        CMD_i = 'tckedit -force -nthreads 0'
        for j in range(i,C.shape[0]):
            if C[i,j] > 0 :
                CMD_i += ' bundles/bundle_%d-%d.tck' %(i+1,j+1)
        os.system( CMD_i + ' bundles/fibers_connecting_%d.tck' % (i+1) )
        CMD += ' bundles/fibers_connecting_%d.tck' %(i+1)
    os.system( CMD + f' {tracto_dir}/{sub}/fibers_connecting.tck' )

# Importing the tractogram
trk2dictionary.run(
    filename_tractogram = os.path.join(tracto_dir,sub,"fibers_connecting.tck"),
    filename_mask       = os.path.join(dwi_dir,sub,f"anat/{sub}_label-WM_probseg_flirt_bin.nii.gz"),
    verbose=3
)

# Load the diffusion data
amico.util.fsl2scheme(bval_path, bvec_path, 'DWI.scheme')
mit = commit.Evaluation()
mit.load_data(dwi_path, 'DWI.scheme', b0_thr=5)

# Set the forward model
mit.set_model( 'StickZeppelinBall' )
d_par       = 1.7E-3             # Parallel diffusivity [mm^2/s]
d_perps_zep = [ 0.51E-3 ]        # Perpendicular diffusivity(s) [mm^2/s]
d_isos      = [ 1.7E-3, 3.0E-3 ] # Isotropic diffusivity(s) [mm^2/s]
mit.model.set( d_par, d_perps_zep, d_isos )
mit.generate_kernels( regenerate=True )
mit.load_kernels()

# Load the sparse data-structure and build A
mit.load_dictionary(os.path.join(tracto_dir,sub,"COMMIT"))
mit.set_threads(1)
mit.build_operator()

# perform the fit
mit.fit( tol_fun=1e-3, max_iter=1000, verbose=True )
mit.save_results( path_suffix="_COMMIT1" )

# Retrieve the streamline contributions estimated by COMMIT for later use in COMMIT2:
x_nnls, _, _ = mit.get_coeffs( get_normalized=False )

Terminal output

-> Creating the dictionary from tractogram:

   * Configuration:
	- Segment position = COMPUTE INTERSECTIONS
	- Coordinates shift in X = 0.000 (voxel-size units)
	- Coordinates shift in Y = 0.000 (voxel-size units)
	- Coordinates shift in Z = 0.000 (voxel-size units)
	- Min segment len  = 0.001 mm
	- Min streamline len    = 0.00 mm
	- Max streamline len    = 250.00 mm
	- Do not blur streamlines
	- Output written to "/derivatives/tractography/sub-015/COMMIT"
	- Temporary files written to "/derivatives/tractography/sub-015/COMMIT/temp"
	- Using parallel computation with 20 threads

   * Loading data:
	- Tractogram
		- geometry taken from "/derivatives/qsiprep/sub-015/anat/sub-015_label-WM_probseg_flirt_bin.nii.gz"
		- 130 x 155 x 131
		- 1.5000 x 1.5000 x 1.5000
		- 5863925 streamlines
	- Filtering mask
		- 130 x 155 x 131
		- 1.5000 x 1.5000 x 1.5000
	- No dataset specified for EC compartments

   * Exporting IC compartments:
     [ 5673556 streamlines kept, 514133882 segments in total ]                    

   * Exporting EC compartments:
     [ 0 voxels, 0 segments ]
	
   [ 281.7 seconds ]

-> Writing scheme file to [ DWI.scheme ]

-> Loading data:
	* Acquisition scheme:
		- diffusion-weighted signal
		- 199 samples, 13 shells
		- 14 @ b=0, 23 @ b=1490.0, 19 @ b=2990.0, 20 @ b=1495.0, 11 @ b=3005.0, 31 @ b=1500.0, 16 @ b=2985.0, 16 @ b=2995.0, 17 @ b=1505.0, 24 @ b=3000.0, 4 @ b=2980.0, 1 @ b=1485.0, 2 @ b=3010.0, 1 @ b=1510.0
	* Signal dataset:
		- dim    : 130 x 155 x 131 x 199
		- pixdim : 1.500 x 1.500 x 1.500
		- values : min=-2287.08, max=39701.55, mean=362.34
   [ 19.6 seconds ]

-> Preprocessing:
	* Normalizing to b0... [ min=-2071.32, max=4038.11, mean=0.42 ]
	* Keeping all b0 volume(s)... [ 130 x 155 x 131 x 199 ]
   [ 1.2 seconds ]

-> Simulating with "Stick-Zeppelin-Ball" model:
   [ 0.5 seconds ]

-> Resampling LUT for subject ".":
	* Keeping all b0 volume(s)...
          [ OK ]
	* Normalizing... [ OK ]
   [ 0.1 seconds ]

-> Loading the dictionary:
	* Segments from the tracts... [ 5673556 fibers and 514133882 segments ]
	* Segments from the peaks...  [ 0 segments ]
	* Isotropic contributions...  [ 157794 voxels ]
	* Post-processing...          [ OK ]
   [ 161.9 seconds ]

-> Distributing workload to different threads:
	* Number of threads : 1
	* A operator...  [ OK ]
	* A' operator... [ OK ]
   [ 0.1 seconds ]
[ ERROR ] The selected model has EC compartments, but no peaks have been provided; check your data

This error originates from the file core.pyx line 691 when the following condition is met: if self.DICTIONARY['EC']['nE'] <= 0 and self.KERNELS['wmh'].shape[0] > 0 :

Having no EC compartments, the value self.DICTIONARY['EC']['nE'] is set to 0. Thus, I do not understand this part but I may be missing something here.

On a side note, the b-values of my data should be 1500s/mm2 and 3000s/mm2 but they oscillate around these values in the b-values file, resulting in more than 2 shells when loading the data. I am not sure if this could have an impact on my error, though.

Please let me know if I can provide any further information.

Thank you very much,

Sébastien

The regularisation variable is named regulatisation

https://github.com/daducci/COMMIT/blob/master/commit/solvers.py#L189

Do you know where this comes from? I checked out the master branch then pip install. Older version works fine. @daducci @matteofrigo

With the current master branch, I get the error running: import commit.

In [2]: import commit
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-4a13c031fad3> in <module>()
----> 1 import commit

/home/ggirard/research/COMMIT/commit/__init__.py in <module>()
----> 1 from core import Evaluation
      2 __all__ = ['core','models','solvers','proximals','trk2dictionary']
      3 
      4 from pkg_resources import get_distribution
      5 __version__ = get_distribution('commit').version

/home/ggirard/research/COMMIT/commit/core.py in <module>()

/home/ggirard/research/COMMIT/commit/solvers.py in <module>()
     11 eps = np.finfo(float).eps
     12 
---> 13 from commit.proximals import (non_negativity,
     14                              omega_group_sparsity,
     15                              prox_group_sparsity,

ImportError: No module named proximals

Hi!

Recent paper "A new method for accurate in vivo mapping of human brain connections using microstructural and anatomical information" (https://advances.sciencemag.org/content/6/31/eaba8245) reports under "Code and data availability" section that the code of COMMIT2 is also available in this repository. Can you please guide how to use COMMIT2?

Tabinda

The pull request daducci/AMICO#37 has introduced the parameter doMergeB0 in the resample function of each model. The code in COMMIT still uses the old syntax without this parameter, and so has to be updated accordingly.

Hi all.

This is not an issue because I solved it but it might be helpful for other Mac Users when installing commit. With new Mac, you might get this error when running pip install .

/Users/desm2239/anaconda2/bin/../include/c++/v1/stdio.h:108:15: fatal error: 'stdio.h' file not found
#include_next <stdio.h>
^~~~~~~~~
1 error generated.
error: command 'clang' failed with exit status 1

This is because your stdio.h is not in your /usr/include when Xcode installs stuff.

Now, on top of this, you cannot create a symlink or a directory in /usr/include. New MacOSX, after El Capitan block you.

So, you must disable SIP: https://apple.stackexchange.com/questions/208478/how-do-i-disable-system-integrity-protection-sip-aka-rootless-on-macos-os-x

Next, you can create a symlink /usr/include using something like this.
find /Applications/Xcode.app -name stdio.h
sudo ln -sf /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/ /usr/include

pip install .
should now work!

Hope this helps others...

The group norm assigned by the parser of the regularisation term is wrong.

When attempting to run the example code for creating the group_idx variable for COMMIT2 using Numpy version 1.24.0, I encounter an "inhomogeneous shape error":

>>> group_idx = np.array( [np.arange(tmp[i],tmp[i+1]) for i in range(len(tmp)-1)] )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2255,) + inhomogeneous part.

This error might potentially also be an issue for COMMIT2, because the structureIC parameter of commit.solvers.init_regularisation is expected to be an inhomogeneous numpy array, as shown in the example provided in the documentation np.array([[0,2,5],[1,3,4],[0,1,2,3,4,5],[6]]).

It appears that this issue is caused by Numpy deprecating the construction of inhomogeneous arrays in version 24. While a long-term solution may require modifications to the code, a short-term solution could advice users to use Numpy version 1.23.5 until the issue is resolved.