xueyu-ubc / xraypulse

copy of xraypulse

XrayPULSE

---

Key Features

This repository provides the official implementation of XrayPULSE:

Key feature bulletin points here

• An attempt to extend PULSE to a biomedical multimodal conversational assistant.
• XrayPULSE is fintuned on Xray-Report paired datasets in Chinese

Details

Our model is based on PULSE. We utilize MedCLIP as our medical visual encoder and Q-former (BLIP2) following a simple linear transformation as the adapter to inject the image to PULSE. For aligning the frozen visual encoder and the LLM by the adapter, we generate Chinese-version Xray-Report paired data from free-text radiology reports of two datasets (MIMIC-CXR and OpenI) with the help of chatGPT. To facilitate research in biomedical multimodal learning, we will release the data to the public.

Get Started

Installation

git clone https://github.com/openmedlab/XrayPULSE.git
cd XrayPULSE

Environment

conda env create -f env.yml
conda activate xraypulse

Prepare the pretrained weights

You can find the pretrained model weights.

• PULSE_Model
• Pretrained_XrayPULSE_Checkpoint

The weights of PULSE would be in a single folder in a structure similar to the following:

pulse_weights
├── config.json
├── generation_config.json
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json 
├── pytorch_model.bin.index.json
├── pytorch_model-00001-of-00002.bin
├── pytorch_model-00002-of-00002.bin 

Then, set the path of pulse_weights to "bloom_model" in the model config file "xraypulse/configs/models/xraypulse.yaml"

And add the path of the pretrained checkpoint in "demo_configs/xraypulse_demo.yaml".

Run Demo

bash run_demo.sh

🙏 Acknowledgement

This project is built upon the gaint sholders of XrayGPT. Great thanks to it!

We used medical aware image encoder from MedCLIP.

The model architecture of XrayGPT follows BLIP2.

🛡️ License

This project is under the CC-BY-NC 4.0 license. See LICENSE for details.