fiy2w / mri_seq2seq

Synthesis Models for Multi-Sequence MRIs

Seq2Seq: Sequence-to-Sequence Generator

We are committed to exploring the application of synthesis or fusion for multi-sequence MRI (also including other modalities such as CT) in clinical settings.

Seq2Seq is a series of dynamic multi-domain models that can translate an arbitrary sequence to a target sequence.

• If you are looking for a straightforward way to use it without much thought, please try nnSeq2Seq.
• To learn more information about our work, please refer to our publications.

nnSeq2Seq

Referring to nnU-Net, we propose nnSeq2Seq, a tool for adaptively training Seq2Seq models with a given dataset. It will analyze the provided training cases and automatically configure a matching synthesis pipeline. No expertise is required on your end! You can easily train the models and use them for your application.

What functions does nnSeq2Seq have?

• Input one/multiple images $\rightarrow$ output one image
  • Missing sequence/modality synthesis
    • Input: [CT] $\rightarrow$ Output: [PET]
    • Input: [T1, T2, DWI, ADC] $\rightarrow$ Output: [DCE]
  • Segmentation
    • Input: [DCE] $\rightarrow$ Output: [Tumor]
    • Input: [T1, T1Gd, T2, FLAIR] $\rightarrow$ Output: [Tumor]
  • Inpainting
  • Deblur and super-resolution
  • vector quantized common (VQC)-latent space compression
    • Input: [CT] $\rightarrow$ Output: [VQC]
    • Input: [T1, T2] $\rightarrow$ Output: [VQC]
• Explainability and visualization
  • Imaging differentiation map
  • Synthesis-based sequence contribution
  • Task-specific sequence contribution

How to get started?

Read these:

• Installation instructions
• Dataset conversion
• Usage instructions

Additional information:

Publications

Follow for our publications, which contain new features that have not yet been added to nnSeq2Seq.

If you use Seq2Seq or some part of the code, please cite (see bibtex):

• Seq2Seq: an arbitrary sequence to a target sequence synthesis, the sequence contribution ranking, and associated imaging-differentiation maps.

  Synthesis-based Imaging-Differentiation Representation Learning for Multi-Sequence 3D/4D MRI
  Medical Image Analysis.

• TSF-Seq2Seq: an explainable task-specific synthesis network, which adapts weights automatically for specific sequence generation tasks and provides interpretability and reliability.

  An Explainable Deep Framework: Towards Task-Specific Fusion for Multi-to-One MRI Synthesis
  MICCAI2023.

mri_seq2seq's Issues

visualization

Hello, I would like to know how this visualization of the attention map is implemented, is it convenient to provide the relevant code. I hope you get better and better in your scientific research.

How to implement many-to-one during testing

How to implement many-to-one during testing