jeixhuang / med-palm

Towards Generalist Biomedical AI

Med-PaLM

A responsible path to generative AI in healthcare: Unleash the power of Med-PaLM 2 to revolutionize medical knowledge, answer complex questions, and enhance healthcare experiences with accuracy, safety, and equitable practices.

Getting Started

pip install MedPalm

Usage

import torch
from medpalm.model import MedPalm

#usage
img = torch.randn(1, 3, 256, 256)
text = torch.randint(0, 20000, (1, 4096))

model = MedPalm()
output = model(img, text)
print(output.shape)

📝 Note: Modify the examples to suit your data and project needs.

📚 Datasets

Here is a comprehensive markdown table containing metadata and details for all the datasets mentioned in the MED PALM paper

Dataset	Modality	Description	# Training examples	# Test examples	Tasks
MultiMedBench	Multimodal	Benchmark for biomedical AI			14 biomedical tasks
MedQA	Text	US licensing exam questions	10,178	1,273	Question answering
MedMCQA	Text	Indian medical exam questions	182,822	4,183	Question answering
PubMedQA	Text	Biomedical literature questions	0	500	Question answering
MIMIC-III	Radiology reports	Radiology reports for ICU patients	58,405 reports	13,057 reports	Report summarization
VQA-RAD	Radiology images	QA pairs on radiology images	1,797 QA pairs	451 QA pairs	Visual question answering
Slake-VQA	Radiology images	English-Chinese QA pairs	9,849 samples	2,070 samples	Visual question answering
Path-VQA	Pathology images	QA pairs on pathology images	19,755 QA pairs	6,761 QA pairs	Visual question answering
MIMIC-CXR	Chest X-ray	Images and reports	353,542	4,834	Report generation, classification
PAD-UFES-20	Dermatology images	Skin lesion smartphone images	1,838 images	460 images	Image classification
CBIS-DDSM (mass)	Mammography	Mammogram mass patches	1,318 images	378 images	Image classification
CBIS-DDSM (calcification)	Mammography	Mammogram calcification patches	1,544 images	326 images	Image classification
VinDr-Mammo	Mammography	Mammogram studies	16,000 images	4,000 images	Image classification
PrecisionFDA (training)	Genomics	Genomic variant images	197,038 images		Image classification
PrecisionFDA (evaluation)	Genomics	Genomic variant images		13,030 images	Image classification
Montgomery County	Chest X-ray	Chest X-rays	0	138 images	TB detection evaluation
MIMIC-CXR (human evaluation)	Chest X-ray	Chest X-ray images and reports		246 cases	Human evaluation

---

💼 Commercial Use-Cases

Med Palm has thousands of potential use cases the 3 below are simple, for more detailed applications check out my new blog article on MedPalm's use in the real world. Click here to learn more

• Clinical Diagnostics: Combining medical imaging, patient tales 📖, and genes, we're aiming for top-notch diagnostic solutions.

• Healthcare Research: Dive deep into diverse datasets and discover something new with Med-PaLM by your side! 🤿

• Telemedicine: Quick, reliable, and remote! 🌍 Med-PaLM's here to revolutionize telehealth.

Contributing to Med Palm 🤖🌟

Help with the todo list!

---

License

Med-PaLM's is under the MIT license. Check out the details here.

Citation

@misc{2307.14334,
Author = {Tao Tu and Shekoofeh Azizi and Danny Driess and Mike Schaekermann and Mohamed Amin and Pi-Chuan Chang and Andrew Carroll and Chuck Lau and Ryutaro Tanno and Ira Ktena and Basil Mustafa and Aakanksha Chowdhery and Yun Liu and Simon Kornblith and David Fleet and Philip Mansfield and Sushant Prakash and Renee Wong and Sunny Virmani and Christopher Semturs and S Sara Mahdavi and Bradley Green and Ewa Dominowska and Blaise Aguera y Arcas and Joelle Barral and Dale Webster and Greg S. Corrado and Yossi Matias and Karan Singhal and Pete Florence and Alan Karthikesalingam and Vivek Natarajan},
Title = {Towards Generalist Biomedical AI},
Year = {2023},
Eprint = {arXiv:2307.14334},
}

Todo

• Implement a training script using Huggingface's accelerate and deepspeed