eigenvivek / diffdrr Goto Github PK

It looks very nice in notebook, but the framerate is much too high when saved to disk

My comments

• Xray-to-CT is not an example of slice-to-volume registration. An Xray is not a tomographic slice of the volume; it is a projection. Change to 2D-to-3D registration or Xray-to-CT registration.
• Rewrite convergence experiment to use momentum.
• Remake convergence figure.
• Add acknowledgements (email polina to ask what grants to add)

Reviewer 1

• Try comparison to Plastimatch GPU (I swear it wasn't working at time of writing).

Reviewer 2

• Understand correlation between failure to converge and initialization parameters
• Maybe show comparison to gradient-free optimization?
• Better explain how using differentiable DRRs can get you to real-time speed (especially with pretraining, reference voxelmorph -- we are pluggable in deep learning framework including for registration and reconstruction)
• Try bigger volumes

Reviewer 3

• 2nd para of the introduction has mixed multiple topics

Similar to how seaborn or sklearn open toy datasetes

Instead of multiresolution rays, use a multiresolution voxel grid (ie, octree represent the CT)

Nalini's advice:

1. Check the training loss
2. Make sure the network can learn a single example
3. Use batchnorm
4. Use residual connections
5. Try a deeper network
6. Try to predict just a few parameters (eg, just project through an axis-aligned slice)

Probably an indexing error with the volume

Downgrade ~1.4

some combination of this

• https://dev.to/epassaro/caching-anaconda-environments-on-github-actions-2d08
• https://github.com/epassaro/cache-conda-envs
• https://github.com/actions/cache
• https://github.com/conda-incubator/setup-miniconda

Currently, the random subsample of pixels is initialized with the DRR generator. Instead, the random sample should be generated every time the forward model is called. This ensures that the entire image can be sampled over the course of an optimization run, instead just a fixed subset.

The last PR (#65) might have broken 3D plotting functions

Switching to einsum notation will break some of the indexing for getting source point and detector plane

generate two DRRs
sum sq diff as loss func

params, perturb by a vector (randomly generated)
then compute the finite difference

two gradients (auto and finite)
compute difference
compute length of difference

scatterplot
x : length of gradient (for finite differences)
y : length of difference

might want to do this for every parameter separately because they have different scales

Use time.perf_counter to time code, not time.time

A few notebooks were never updated to the new API and it's confusing for first-time users. Need to fix any breaking changes in the notebooks.

• plastimatch
• can also just try the cpu?

• arxiv
• code style
• black
• ci

Something in animation function is producing too much whitespace around saved GIFs

Need to figure out how to remove this

Want to be able to compute just a subset of the pixels

also would be helpful to have sparse implementations of loss functions too!

• Use central differences to compare to autodiff
• Maybe compare the timing?

Just want to verify that the gradients are correct

• Continuous integration
• Code coverage
• Badges for both

• Figure out a testing package
• Add tests to make sure core API is never broken

• Pick out a doc provider (sphinx, readthedocs, etc)
• Doctests? (#49)
• Add badge

• Nelder-Mead vs gradient-based descent
• Try training a NN to predict DRR parameters
• Make optimization timing plot

https://github.com/v715/DiffDRR/blob/main/src/drr.py

Seems to be a bug when a ray doesn't intersect the volume, causes some CUDA indexing error that makes the program crash

Use einsum notation to do all the tensor computations in DRR generation

This notation make the computation of batched DRRs much easier as well (#30)

Allow "cuda:x" as input to get_device, in addition to "cuda".

Extend the API to allow multiple set of parameters be passed simultaneously

Also check that XCorr2 will work with batches in current form

• Implement all loss metrics (L2, NCC)
• Try to make a 3D plot for the loss landscape?

The 3D volume and the 2D detector plane are stored as their physical analogs (3D voxelgrid and 4D tensor of spatial coordinates, respectively). This is helpful for conceptualization, but there is no need to structure the code based on the geometry of the real world.

• The voxelgrid could be completely flattened, or hashed, so long as it's any structure with lookup.
• The spatial coordinates should be reshaped from (b, n1, n2, 3) where n1,n2 are image dimensions to (b, m, 3) where m=n1xn2. This would be especially useful for getting sparse computation of DRRs working (#22).

Once a large tensor like alpha, alphamids or step_length is no longer necessary, it should be deleted to free up memory on the GPU

This would also potentially allow us to overcome memory issues for large DRRs

Given an input DRR from unknown parameters, train a NN to predict the parameters (ie, an pre-initialization network)

• Generate multiple (parameter, DRR) pairs and train NN

Start w/ the architecture in Fig 4 from alansary et al