daviddmc / nesvor

NeSVoR is a package for GPU-accelerated slice-to-volume reconstruction.

Hi! I am currently a fifth-year Ph.D. student at MIT majoring in Electrical Engineering and Computer Science. My research focuses on machine learning and developing robust and efficient algorithms driven by clinical problems. Applications include motion-robust 3D rendering of the human brain, real-time quality assessment in MR scans as well as pose estimation and motion characterization of fetuses. I am advised by Prof. Elfar Adalsteinsson and collaborate closely with Prof. Polina Golland and Prof. P. Ellen Grant.

I also did summer internships at Google and Meta, working on automated Ads bidding and large-scale video recommendation systems respectively.

Prior to MIT, I received my Bachelor's degree from Tsinghua University in 2018. I also spent a summer as a research assistant at Stanford, where I was advised by Prof. John Pauly and Prof. Greg Zaharchuk.

nesvor's Issues

Efficiency with deepspeed

Possibly naive suggestion, I wonder if we can lower VRAM usage and/or improve speed using https://github.com/microsoft/DeepSpeed ?

Can the overall method be applied to different medical images?

As the title suggests, I'm wondering if the model could be fine-tuned for different types of medical images. Can it be applied to larger, RGB images like H&E images? To my knowledge 2D H&E slices are much bigger than 2D MRI slices. Thanks!

Wrong Phantom in tests/slice_acquisition

Hi,
I tested both versions of Phantom in the code "modified-shepp-logan" and "shepp_logan" generated by phantom3d.py, and the outputs are completely off. Please consider replacing it with the correct Phantom.

Visualizing learnt hash grid at different level

Dear @daviddmc,
I was trying to reproduce the visualization you made Fig.13 in the nesvor paper, i.e. the different levels of the learnt hashgrid. However I have not managed so far to obtain similar visualization. Is there some piece of code included in nesvor I could use to do that. If not, may you indicate me how to proceed ? So far I obtained patterns that are replicated, i.e. one brain then half a brain superimposed in the same figure, which seems weird.
Best,
Alex

Three facets of data

Dear @daviddmc,

I have information on three facets of a site with specific location coordinates. I was wondering, can we try the program directly with data from other organs.

Thanks!

Wrong checkpoint for DynUNet

I ran nesvor segment-stack --input-stacks volumes/case2_day1_stack.nii.gz --output-stack-masks volumes/case2_day1_mask.nii.gz but the downloaded checkpoint (nesvor/preprocessing/masking/brain_segmentation.py) is completely wrong and all keys are unexpected.

Computation of PSNR score : Low values

Dear @daviddmc,
Thank you for this open source code!
I try to reproduce your experiments by computing the PSNR score between the reconstructed volume from NeSVoR and the original high resolution data that I sub-sampled.
I get surprisingly low PSNR value, that seem to be due to a shift in the intensity distributions between the two images.
By searching in the code, I did identify the line 525 in svort/inference.py , (function run_svort) which looks like a normalization of the intensities and could thus induce the shift in the distribution.
Could you please explain me why the normalization at line 525 is needed, and how you did handle the problem of shift in the distribution for computing the PSNR in your paper?
Thanks very much for your help
Best,
Steven

"SVD did not converge" error when using nesvor segment-stack

I have an issue when using nesvor segment-stack. It fails, giving a SVD error when trying to save the . I have tried with different images, so it is not related to the input. I am using the docker image, version 0.5.0. I have also tried it in a HPC environment (converting the image to a singularity container), getting the same error.

This is the full error stack:

WARNING:root:Fail to load tinycudann. Will use pytorch implementation.
/usr/local/lib/python3.10/dist-packages/nibabel/nifti1.py:1088: RuntimeWarning: invalid value encountered in divide
  R = RZS / zooms
/usr/local/lib/python3.10/dist-packages/numpy/linalg/linalg.py:2139: RuntimeWarning: invalid value encountered in det
  r = _umath_linalg.det(a, signature=signature)
ERROR:root:Unhandled exception:
Traceback (most recent call last):
  File "/usr/local/bin/nesvor", line 33, in <module>
    sys.exit(load_entry_point('nesvor', 'console_scripts', 'nesvor')())
  File "/usr/local/NeSVoR/nesvor/cli/main.py", line 23, in main
    run(args)
  File "/usr/local/NeSVoR/nesvor/cli/main.py", line 48, in run
    getattr(commands, command_class)(args).main()
  File "/usr/local/NeSVoR/nesvor/cli/commands.py", line 75, in main
    self.exec()
  File "/usr/local/NeSVoR/nesvor/cli/commands.py", line 257, in exec
    outputs(
  File "/usr/local/NeSVoR/nesvor/cli/io.py", line 93, in outputs
    m.save(p)
  File "/usr/local/NeSVoR/nesvor/image/image.py", line 187, in save
    save_nii_volume(path, output_volume, affine)
  File "/usr/local/NeSVoR/nesvor/image/image_utils.py", line 110, in save_nii_volume
    img = nib.nifti1.Nifti1Image(volume, affine)
  File "/usr/local/lib/python3.10/dist-packages/nibabel/nifti1.py", line 1848, in __init__
    super().__init__(dataobj, affine, header, extra, file_map, dtype)
  File "/usr/local/lib/python3.10/dist-packages/nibabel/analyze.py", line 909, in __init__
    super().__init__(dataobj, affine, header, extra, file_map)
  File "/usr/local/lib/python3.10/dist-packages/nibabel/spatialimages.py", line 531, in __init__
    self.update_header()
  File "/usr/local/lib/python3.10/dist-packages/nibabel/nifti1.py", line 2273, in update_header
    super().update_header()
  File "/usr/local/lib/python3.10/dist-packages/nibabel/nifti1.py", line 1884, in update_header
    super().update_header()
  File "/usr/local/lib/python3.10/dist-packages/nibabel/spatialimages.py", line 565, in update_header
    self._affine2header()
  File "/usr/local/lib/python3.10/dist-packages/nibabel/nifti1.py", line 1894, in _affine2header
    hdr.set_qform(self._affine, code='unknown')
  File "/usr/local/lib/python3.10/dist-packages/nibabel/nifti1.py", line 1100, in set_qform
    P, S, Qs = npl.svd(R)
  File "<__array_function__ internals>", line 200, in svd
  File "/usr/local/lib/python3.10/dist-packages/numpy/linalg/linalg.py", line 1642, in svd
    u, s, vh = gufunc(a, signature=signature, extobj=extobj)
  File "/usr/local/lib/python3.10/dist-packages/numpy/linalg/linalg.py", line 98, in _raise_linalgerror_svd_nonconvergence
    raise LinAlgError("SVD did not converge")
numpy.linalg.LinAlgError: SVD did not converge

Note that the rest of the tool works great: when using Monaifbs separately and providing NeSVoR with the masks, everything works smoothly. So a workaround right now is to compute the masks outside of the docker.

It is not a memory issue, and I have manually checked the images and they have no issues (Monaifbs works well with the same images when used in my own installation).

Understanding n-level-bias parameter defaults value

Hello,

thanks for developing and making available nesvor. I had a question regarding the n-level-bias parameter of the reconstruct command

• what was the exact meaning of the n-level-bias ?
• its default value is set to 0: does this mean bias field is not corrected by default ? Or does it mean only level 0 of the hasgrid is used to estimate the bias ?

Best,
Alex

TCNN was not compiled with half-precision support!

Dear @daviddmc,

I use a tool (https://github.com/daviddmc/NeSVoR) that relies on tiny cuda.

However, I get the following error :
RuntimeError: TCNN was not compiled with half-precision support.

How can I compile TCNN with half-precision support ?

CUDA Version: 11.7
NVIDIA TITAN Xp

Thanks!

ROI in the data is too large for SVoRT

Dear @daviddmc,

Thanks for making this code public!
I am currently trying to reconstruct an adult brain from HCP using nesvor. However, I get the warning "[WARNING] ROI in the data is too large for SVoRT" when I want to pre-register the slices via SVoRT. The model then skips SVoRT and resorts to regular stack registration. I'm using 3 Stacks with spacing[1.25, 1.25, 3.3], resulting in a rough dimension of [170, 150, 55]. Would you happen to know how to handle this problem?

Thanks!

What parameters I should set to improve the separation of the fetal brain regions and eliminate the unwanted areas？

Thank you for your fantastic work! I've successfully run your open-source code for 3D reconstruction. However, after the reconstruction of the fetal brain, there are always some regions that are not of interest, while others are well-separated. Currently, I'm using the following parameters with nesvor reconstruct:
nesvor reconstruct
--input-stacks stack-1.nii.gz ... stack-N.nii.gz
--thicknesses ...
--output-volume volume.nii.gz
--output-resolution 0.8
--segmentation
--bias-field-correction
Could you please advise on what parameters I should set to improve the separation of the fetal brain regions and eliminate the unwanted areas?

TypeError: 'ABCMeta' object is not subscriptable

TypeError: 'ABCMeta' object is not subscriptable

torch 1.10.0+cu113
cuda 11.2
python 3.8.10

nesvor reconstruct --input-slices ../nesvor/casenii
--output-slices ../nesvor/outnii
--output-resolution 0.8
--registration svort
--segmentation
--bias-field-correction

seems like a ez question..

Cannot reproduce results on FeTA using your slicing strategy.

Hi,
I tried using nesvor to reconstruct volume with stacks of slices sampled from the FeTA dataset. Stack assessment gave high ncc ( > 0.92).
I enabled registration and bias field correction and sampled 3 orthogonal stacks with gaps over the whole volume using modified code from you slice_acquistion tests, but the results are terrible. Even worse, if I enable segmentation the whole stack is masked out.
It's hard to infer how else stacks could be extracted from the paper. Here's my code and results:

nesvor reconstruct \
--input-stacks stacks/feta_stack1_gap=3.nii.gz stacks/feta_stack2_gap=3.nii.gz stacks/feta_stack3_gap=3.nii.gz \
--output-volume volumes/feta_rec_nesvor_gap=3.nii.gz \
--bias-field-correction \
--registration svort

 def get_cg_recon_test_data(self, angles, volume: torch.Tensor=None, return_stack_as_list=True):
        

        vs = 240 if volume is None else max(volume.shape) # z axis

        gap = s_thick = 3#not the slice thickness
        res = 1
        res_s = 1.5
        n_slice = int((np.sqrt(3) * vs) / gap) + 4
        ss = int((np.sqrt(3) * vs) / res_s) + 4
        
        if volume is None:
            print(f"Creating a Phantom of {(vs, vs, vs)} with resolution {res}")
            if self.phantom == "default":
                volume = phantom3d(n=vs, phantom="shepp_logan")
            else:
                volume = shepp_logan((vs, vs, vs))
        else:
            print('Using provided volume')
            
        volume = torch.tensor(volume, dtype=torch.float32).cuda().contiguous()
        volume = pad_to_length(volume, vs)
        volume = volume.unsqueeze(0).unsqueeze(0) #preprocess for svort
        
        psf = get_PSF(res_ratio=(res_s / res, res_s / res, s_thick / res)).cuda()

        stacks = []
        transforms = []
        
        #get slices for each triplet of angles
        for i in range(len(angles)):
            angle = (
                torch.tensor([angles[i]], dtype=torch.float32)
                .cuda()
                .expand(n_slice, -1)
            )
            tz = (
                torch.arange(0, n_slice, dtype=torch.float32).cuda()
                - (n_slice - 1) / 2.0
            ) * gap
            tx = ty = torch.ones_like(tz) * 0.5 #pick middle coordinates? but why 0.5 not 0.5 * dim
            t = torch.stack((tx, ty, tz), -1)
            transform = RigidTransform(torch.cat((angle, t), -1), trans_first=True)
            # sample slices
            mat = mat_update_resolution(transform.matrix(), 1, res)
            slices = slice_acquisition(
                mat, volume, None, None, psf, (ss, ss), res_s / res, False, False
            )
            if return_stack_as_list:
                slices = slices.squeeze()
            stacks.append(slices) #shape: (h, 1, w, l)
            transforms.append(transform)

        params = {
            "psf": psf,
            "slice_shape": (ss, ss),
            "res_s": res_s,
            "res_r": res,
            "interp_psf": False,
            "volume_shape": (vs, vs, vs),
        }
        
        #save stacks to test Nesvor
        if len(stacks) == 3:
            self._save_stacks(stacks, gap=gap)

nesvor segment-stack issue when using it standalone

Using "nesvor segment-stack" standalone command (https://nesvor.readthedocs.io/en/latest/commands/segment-stack.html) generates masks with different sform/qform and origin, which, in turn, makes the "nesvor reconstruct" fail with the error:

	2024-03-13 15:55:26 [INFO] Data loading starts ...
	2024-03-13 15:55:26 [INFO] loading stacks
	2024-03-13 15:55:26 [ERROR] Unhandled exception:
	Traceback (most recent call last):
	  File "/usr/local/bin/nesvor", line 33, in <module>
	    sys.exit(load_entry_point('nesvor', 'console_scripts', 'nesvor')())
	  File "/usr/local/NeSVoR/nesvor/cli/main.py", line 23, in main
	    run(args)
	  File "/usr/local/NeSVoR/nesvor/cli/main.py", line 48, in run
	    getattr(commands, command_class)(args).main()
	  File "/usr/local/NeSVoR/nesvor/cli/commands.py", line 75, in main
	    self.exec()
	  File "/usr/local/NeSVoR/nesvor/cli/commands.py", line 160, in exec
	    input_dict = self.preprocess()
	  File "/usr/local/NeSVoR/nesvor/cli/commands.py", line 132, in preprocess
	    input_dict, self.args = inputs(self.args)
	  File "/usr/local/NeSVoR/nesvor/cli/io.py", line 24, in inputs
	    stack = load_stack(
	  File "/usr/local/NeSVoR/nesvor/image/image.py", line 651, in load_stack
	    raise Exception(
	Exception: Error: the sizes/resolutions/affine transformations of the input stack and stack mask do not match!

Note that this does not happen when using just "nesvor reconstruct" without the previously computed masks. The error only happens when: 1) using nesvor segment-stack to generate masks and 2) using those masks as input to nesvor reconstruct.

I have replicated this with various stacks, but unsure if this is a problem in the qform/sform configuration of this specific data or a general thing. This is the header information and the origin of the original stack:

And this is the one from the corresponding mask generated by nesvor segment-stack:

This is bothersome if we want to use the various functions included in the library to run every step at a time.

A quick fix that work for me is just to copy the header from the mask to the original stack, and invert the x axis. This seems to make everything work, but it is dirty.

I believe the problem comes from the transformations on the affine matrix that nesvor performs when loading/saving a nifti file, so if you can take a look at it, that would be great. Thanks!

--device option not passed down to tiny-cuda-nn

nesvor has the option --device to specify which GPU is to be used by nesvor. However, the GPU selection is not passed down to the tiny-cuda-nn functions, which will always attempt to use GPU#0.

We have a machine with 2 GPUs. GPU#0 is currently being used, so we try to run nesvor --device 1. However, the following exception occurs:

Traceback (most recent call last):
  File "/opt/conda/bin/nesvor", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/nesvor/cli/main.py", line 23, in main
    run(args)
  File "/opt/conda/lib/python3.10/site-packages/nesvor/cli/main.py", line 48, in run
    getattr(commands, command_class)(args).main()
  File "/opt/conda/lib/python3.10/site-packages/nesvor/cli/commands.py", line 75, in main
    self.exec()
  File "/opt/conda/lib/python3.10/site-packages/nesvor/cli/commands.py", line 162, in exec
    model, output_slices, mask = train(input_dict["input_slices"], self.args)
  File "/opt/conda/lib/python3.10/site-packages/nesvor/inr/train.py", line 36, in train
    model = NeSVoR(
  File "/opt/conda/lib/python3.10/site-packages/nesvor/inr/models.py", line 309, in __init__
    self.build_network(bounding_box)
  File "/opt/conda/lib/python3.10/site-packages/nesvor/inr/models.py", line 350, in build_network
    self.inr = INR(bounding_box, self.args, self.spatial_scaling)
  File "/opt/conda/lib/python3.10/site-packages/nesvor/inr/models.py", line 147, in __init__
    self.encoding = build_encoding(
  File "/opt/conda/lib/python3.10/site-packages/nesvor/inr/models.py", line 48, in build_encoding
    raise e
  File "/opt/conda/lib/python3.10/site-packages/nesvor/inr/models.py", line 39, in build_encoding
    encoding = tcnn.Encoding(
  File "/opt/conda/lib/python3.10/site-packages/tinycudann/modules.py", line 315, in __init__
    super(Encoding, self).__init__(seed=seed)
  File "/opt/conda/lib/python3.10/site-packages/tinycudann/modules.py", line 161, in __init__
    initial_params = self.native_tcnn_module.initial_params(seed)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

I used nvtop to monitor GPU usage during the runtime of NeSVoR.

• nesvor started around -110s
• nesvor crashed around -60s
• we can observe a bump in GPU0 memory consumption right before nesvor crashes