Our goal is to perform an inpainting for computer tomographic (CT) images, to reduce the dose a patient has to undergo during an examination. We take advantage of recent developments in the architecture of neural nets and utilize so called deep convolutional generative adversarial neural nets (DCGANs) as well as an autoencoder (AE) which is trained in an adversarial fashion.
Radon transforms are images as one obtains them from CT scans before the reconstruction. Figure 1 shows a reconstructed CT scan in the axial plane of a patient.
Before the reconstruction, an image rather has the shape of many stacked sine functions. This typical appearance, as seen below, results from the detector and the x-ray source which rotate around the patient.
The shown radon transform is the result of transforming a simulated section through randomly created ellipses. For a proof of concept we stick to these simulated sections before applying the algorithm to real patient data.
In order to reduce the radiation, a patient has to undergo, one can simply reduce the number of projections. An example of a radon transform with a reduced number of projections is shown in figure 3.
To compensate for the reduced information that one obtains from such a radon transform, we test two methods for inpainting the unkown regions.
For the inpainting we test 2 methods. The first method is based on a DCGAN architecture from [3], and learns to generate snippets of radon transformations from a random input. To obtain the most likely inpainting, one optimizes for the random input, as shown in [2]. The snippets are then put together to inpaint a whole radon transformation. The second method is based on an AE that makes use of dilated convolutions to increase the receptive field and is trained in an adversarial fashion using both, a global and a local discriminator network [4].
We can see that compared to the generative approach (Figure 4, center), the autoencoded approach (Figure 4, right) inpaints the radon transform with a global structure.
The reconstructed images overall represent the finding of the lacking global structure for the generative inpainting, which results in ring artifacts (Figure 5, center). Both resulting reconstruction are, due to the removed information, a little blurry.
One huge advantage of the AE over the DCGAN approach is that the AE needs no further optimization step to perform the inpainting and is thus faster by at least a factor of 10. Given that CT images usually involve a huge number of images, further improvements to the DCGAN approach with for example greater image snippets might not be as conducive. The AE encoder on the other hand shows promising results and should further be investigated with some small adjustments, as explained in the conclusion here. We especially want to point out that with the unofficial release of PyTorch 0.4.0 and above, there will be Fourier methods added. Using these Fourier methods, one can add a layer to the neural network that performs a reconstruction of the inpainted radon transform, as explained in [5], thus making it possible to not only account for the radon space but for the object space as well. An example of a Fourier based reconstruction is presented in figure 6.
[2] Semantic Image Inpainting with Deep Generative Models
[3] Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
[4] Globally and Locally Consistent Image Completion
[5] Fourier Methods in CT: Projection and Reconstruction Algorithms
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent
