biomedia-mira / istn Goto Github PK

I was running the code on provided 2d data in affine and bspline mode. Affine transformation seems to preserve structure. However in Bspline, deformation is huge and the image structure is also not maintained. Training was done without changing any parameter from the repo.
Original image

Warped image

What could be the reason behind such deformation? I compared the bspline code with airlab and looks like diffeomorphism and Diffusion Regulariser are not added. I was hoping adding them might make the deformation look reasonable. Is there anything else that needs to be taken care of.

Thank you for the help.

I am interested to use the ISTN to try aligning brain scans, leveraging the location of mitochondria as landmarks. However I have run into issues in the code. In my understanding from the ISTN paper tied to this github project, segmentations are landmarks which are more tree/segment shaped and masks are landmarks which are more point shaped. In the paper, both are demonstrated to have good uses with ISTNs for alignment of images and promoting the ITN's ability to make good representations of the input. I am hoping to leverage a configuration that best optimizes for the mitochondria as landmarks, which leads me to want to use a mask; especially if the ISTN can be better optimized for point like structures with masks. The mitochondria tend to have a very point-like presense in the brain tissue as tubular structures. Thus I am more inclined to try using them as a mask rather than like segments. However in trying to pass the mitochondria as masks and leaving the segmentation as None in the istn-reg.py commandline arguments, I run into many issues in the code throughout that seem to require a lot more than just a hotfix or two. Thus I am wondering is there is a way to run the code with masks instead of segmentation? Also which loss function would you think is the best bet for getting good results with mitochondria as masks would you say? Some preliminary results have surprisingly shown the supervised loss function is not doing so well compared to the others.

I noted that the research participated in the Learn2Reg LungCT challenge in 2021. I observed that its performance is still ranked highly on today's LeaderBoard.
However, when attempting to replicate the results with the open-source code on the LungCT dataset, the outcomes are significantly different.
I would be grateful if you could share the version of the code that was utilized during the Learn2Reg challenge.

In def affine_matrix(self, x) for the STN3D class (line 330), the rot Z code block is repeated:

# self.rot = self.rotation(x)
rot = torch.tanh(self.rotation(x)) * (math.pi / 4.0)
# rot Z
angle_1 = rot[:, 0].view(-1)
rotation_matrix_1 = torch.zeros([b, 4, 4], dtype=torch.float)
rotation_matrix_1[:, 0, 0] = torch.cos(angle_1)
rotation_matrix_1[:, 0, 1] = -torch.sin(angle_1)
rotation_matrix_1[:, 1, 0] = torch.sin(angle_1)
rotation_matrix_1[:, 1, 1] = torch.cos(angle_1)
rotation_matrix_1[:, 2, 2] = 1.0
rotation_matrix_1[:, 3, 3] = 1.0
# rot X
angle_2 = rot[:, 1].view(-1)
rotation_matrix_2 = torch.zeros([b, 4, 4], dtype=torch.float)
rotation_matrix_2[:, 1, 1] = torch.cos(angle_2)
rotation_matrix_2[:, 1, 2] = -torch.sin(angle_2)
rotation_matrix_2[:, 2, 1] = torch.sin(angle_2)
rotation_matrix_2[:, 2, 2] = torch.cos(angle_2)
rotation_matrix_2[:, 0, 0] = 1.0
rotation_matrix_2[:, 3, 3] = 1.0
# rot Z
angle_3 = rot[:, 2].view(-1)
rotation_matrix_3 = torch.zeros([b, 4, 4], dtype=torch.float)
rotation_matrix_3[:, 0, 0] = torch.cos(angle_3)
rotation_matrix_3[:, 0, 1] = -torch.sin(angle_3)
rotation_matrix_3[:, 1, 0] = torch.sin(angle_3)
rotation_matrix_3[:, 1, 1] = torch.cos(angle_3)
rotation_matrix_3[:, 2, 2] = 1.0
rotation_matrix_3[:, 3, 3] = 1.0

Should the last operation rotate around the Y axis instead?

# rot Y
angle_3 = rot[:, 2].view(-1)
rotation_matrix_3 = torch.zeros([b, 4, 4], dtype=torch.float)
rotation_matrix_3[:, 0, 0] = torch.cos(angle_3)
rotation_matrix_3[:, 0, 2] = torch.sin(angle_3)
rotation_matrix_3[:, 2, 0] = -torch.sin(angle_3)
rotation_matrix_3[:, 2, 2] = torch.cos(angle_3)
rotation_matrix_3[:, 1, 1] = 1.0
rotation_matrix_3[:, 3, 3] = 1.0

Hello,

I am working on a Ubuntu linux subsystem for windows. I have installed all the requirements from the requirements folder and have successfully run all the 2D examples written in the comments of istn-reg.py and gotten the plot of the output validation curves and training curves. However I cannot run the 3D examples code:

python istn-reg.py --mode3d --loss i --out output3d/istn-i --model output3d/istn-i/train/model --config data/brain3d/config.affine.json --train data/brain3d/train.csv --train_seg data/brain3d/train.seg.csv --train_msk data/brain3d/train.msk.csv --val data/brain3d/val.csv --val_seg data/brain3d/val.seg.csv --val_msk data/brain3d/val.msk.csv --test data/brain3d/test.csv --test_seg data/brain3d/test.seg.csv --test_msk data/brain3d/test.msk.csv

I get this error:

...

TRAINING data...

Reading source image .nii.gz
Traceback (most recent call last):
File "istn-reg.py", line 487, in
train(args)
File "istn-reg.py", line 251, in train
resampler_seg=config.resampler_seg)
File "/home/yahtzee/istn/pymira/img/datasets.py", line 122, in init
source = sitk.ReadImage(src_path, sitk.sitkFloat32)
File "/home/yahtzee/.local/lib/python3.6/site-packages/SimpleITK/SimpleITK.py", line 8876, in ReadImage
return _SimpleITK.ReadImage(*args)
RuntimeError: Exception thrown in SimpleITK ReadImage: /tmp/SimpleITK/Code/IO/src/sitkImageReaderBase.cxx:99:
sitk::ERROR: The file ".nii.gz" does not exist.

I am I doing something wrong? am I missing a dependency or anything?

Thank you

I was successful in having the program run to completion and viewing the results on tensorboard for the original example plus custom data sets of 70 image pairs (main and seg for both training and validation) that were 128x128, 256x256, 1024x1024, and 512x600. However when I ran it for the image set where each image is 1334x1553 pixels, I got this strange cryptic error. Is there a constraint on this istn-reg tool that I need to know, is there a bug, or is there something otherwise that I need to do in order to run the program with my desired image set.

pip install -r requirements.txt won't run unless I remove pkg-resources=0.0.0 from file. https://stackoverflow.com/questions/39577984/what-is-pkg-resources-0-0-0-in-output-of-pip-freeze-command According to this link this line being added to the requirements.txt is a bug from pip3 providing incorrect metadata to pip.

I've been using the BSpline3D network. When I try to use the predicted parameters in SimpleITK, the transformation result do not match. STN network heavily warps the images while in SimpleITK they seem as not being warped. Can anyone help me with the reasoning of such behaviour.

Hi. It is, actually, not an issue related to the ISTN code. I want to know if you have any plan to release the code in Github for your paper-"Atlas-ISTN: Joint Segmentation, Registration and Atlas Construction with Image-and-Spatial Transformer Networks."

I run :
python istn-reg.py --config data/synth2d/config.json --transformation affine --loss u --out output/stn-u --model output/stn-u/train/model
I get the error :
empty(): argument 'size' must be tuple of ints, but found element of type Tensor at pos 2

Great job,and I have a question.
In code Line 70
Is the parameter 0.1 to control the translation amount in a small range?
Looking forward your reply!
Best wishes!

Trying to run:

python istn-reg.py --mode3d --loss i --out output3d/istn-i --model output3d/istn-i/train/model --config data/synth3d/config.json --train data/synth3d/train.csv --train_seg data/synth3d/train.seg.csv --val data/synth3d/val.csv --val_seg data/synth3d/val.seg.csv

There is an indentation error on line 180 in pymira/nets/stn.py that will cause running the script to fail, giving:

Traceback (most recent call last):
  File "istn-reg.py", line 39, in <module>
    from pymira.nets.stn import STN2D, BSplineSTN2D, STN3D, BSplineSTN3D
  File "/local/home/Jonathan/istn/pymira/nets/stn.py", line 180
    self.fc.weight.data.zero_()
                              ^
TabError: inconsistent use of tabs and spaces in indentation

I was running code with Brain MRI functional images with no segmentations and masks. I was wondering if the STN layer only controls the rotation of the image. Since I don't use the segmentation data, ITN provides me a bad result as I attached.