roka9 / vilmedic

ViLMedic (Vision-and-Language medical research) is a modular framework for vision and language multimodal research in the medical field

---

ViLMedic: a framework for research at the intersection of vision and language in medical AI

Installation

conda create --name vilmedic python==3.9 -y
git clone https://github.com/jbdel/vilmedic
python setup.py develop

Documentation

Learn more about ViLMedic here.

Model Zoo

ViLMedic hosts a zoo of pretrained models.

from vilmedic import AutoModel
model, processor = AutoModel.from_pretrained("selfsup/convirt-mimic")
batch = processor.inference(seq=["no acute cardiopulmonary process"],
                            image=["my_chest_xray.jpg"])

out = model(**batch)
print(out.keys())
# dict_keys(['loss', 'loss_l', 'loss_v', 'linguistic', 'visual'])

Name	dataset	Report preprocessing
Radiology report generation
rrg/biomed-roberta-baseline-mimic	mimic-cxr	r2gen
rrg/biomed-roberta-baseline-indiana	indiana	r2gen
rrg/baseline-padchest	padchest	-
Radiology report summarization
rrs/biomed-roberta-baseline-mimic	mimic-cxr	rouge
rrs/biomed-roberta-baseline-indiana	indiana	r2gen
Self-supervision
selfsup/convirt-mimic	mimic-cxr	r2gen
selfsup/convirt-mimic-balanced	mimic-cxr	r2gen
selfsup/convirt-padchest-16	padchest	gloria
selfsup/convirt-padchest-32	padchest	gloria
selfsup/convirt-indiana-16	indiana	r2gen
selfsup/convirt-indiana-32	indiana	r2gen
selfsup/convirt-indiana-64	indiana	r2gen
selfsup/gloria-chexpert	CheXpert	gloria
selfsup/gloria-mimic-48	mimic-cxr	r2gen
selfsup/simclr-mimic-16	mimic-cxr
selfsup/simclr-mimic-32	mimic-cxr
selfsup/simclr-mimic-64	mimic-cxr
selfsup/vae-mimic	mimic-cxr
selfsup/vae-indiana	indiana
selfsup/vae-padchest	padchest
Medical VQA
mvqa/mvqa-imageclef	ImageCLEF-VQAMed

Implemented solutions

ViLMedic replicates solutions from the multimodal medical literature.

Solutions
Medical Visual Question Answering
SYSU-HCP at VQA-Med 2021
Radiology report generation
Generating Radiology Reports via Memory-driven Transformer
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports
Improving Factual Completeness and Consistency of Image-to-text Radiology Report Generation
Radiology report summarization
Multimodal Radiology Report Summarization
Multimodal self-supervised Learning
Contrastive Learning of Medical Visual Representations from Paired Images and Text
DALLE: Zero-Shot Text-to-Image Generation
CLIP: Learning Transferable Visual Models From Natural Language Supervision
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition

Citation

If you use ViLMedic in your work or use any models published in ViLMedic, please cite:

@misc{Delbrouck2021ViLmedic,
  author =       {Jean-Benoit Delbrouck and Khaled Saab and Juan Manuel Zambrano Chaves and Pierre Joseph Marcel Chambon and Sabri Eyuboglu
 and Maya Varma and Jared Alexander Dunnmon and Curtis Langlotz and Akshay Chaudhari and Daniel Rubin},
  title =        {ViLMedic: A multimodal framework for vision and language medical research},
  howpublished = {\url{https://github.com/jbdel/vilmedic}},
  year =         {2021}
}

License

ViLMedic is MIT-licensed. The license applies to the pre-trained models as well.