openmedlab / xraypulse

XrayPULSE

---

Key Features

This repository provides the official implementation of XrayPULSE:

Key feature bulletin points here

• An attempt to extend PULSE to a biomedical multimodal conversational assistant.
• XrayPULSE is fintuned on Xray-Report paired datasets in Chinese

Details

Our model is based on PULSE. We utilize MedCLIP as our medical visual encoder and Q-former (BLIP2) following a simple linear transformation as the adapter to inject the image to PULSE. For aligning the frozen visual encoder and the LLM by the adapter, we generate Chinese-version Xray-Report paired data from free-text radiology reports of two datasets (MIMIC-CXR and OpenI) with the help of chatGPT. To facilitate research in biomedical multimodal learning, we will release the data to the public.

Get Started

Installation

git clone https://github.com/openmedlab/XrayPULSE.git
cd XrayPULSE

Environment

conda env create -f env.yml
conda activate xraypulse

Prepare the pretrained weights

You can find the pretrained model weights.

• PULSE_Model
• Pretrained_XrayPULSE_Checkpoint

The weights of PULSE would be in a single folder in a structure similar to the following:

pulse_weights
├── config.json
├── generation_config.json
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json 
├── pytorch_model.bin.index.json
├── pytorch_model-00001-of-00002.bin
├── pytorch_model-00002-of-00002.bin 

Then, set the path of pulse_weights to "bloom_model" in the model config file "xraypulse/configs/models/xraypulse.yaml"

And add the path of the pretrained checkpoint in "demo_configs/xraypulse_demo.yaml".

Run Demo

bash run_demo.sh

🙏 Acknowledgement

This project is built upon the gaint sholders of XrayGPT. Great thanks to it!

We used medical aware image encoder from MedCLIP.

The model architecture of XrayGPT follows BLIP2.

🛡️ License

This project is under the CC-BY-NC 4.0 license. See LICENSE for details.

xraypulse's Issues

run webui demo bug

When running webui of run_demo.py, after running successfully, the web interface appears to upload medical images, and the error will occur

没办法复现，报错

Traceback (most recent call last):
File "C:\Users\Administrator\Desktop\XrayPULSE-main\demo.py", line 62, in
model = model_cls.from_config(model_config).to('cuda:{}'.format(args.gpu_id))
File "C:\Users\Administrator\Desktop\XrayPULSE-main\xraypulse\models\xray_pulse.py", line 348, in from_config
model = cls(
File "C:\Users\Administrator\Desktop\XrayPULSE-main\xraypulse\models\xray_pulse.py", line 71, in init
self.visual_encoder, self.ln_vision = self.init_vision_encoder(
File "C:\Users\Administrator\Desktop\XrayPULSE-main\xraypulse\models\blip2.py", line 65, in init_vision_encoder
visual_encoder = create_eva_vit_g(
File "C:\Users\Administrator\Desktop\XrayPULSE-main\xraypulse\models\eva_vit.py", line 433, in create_eva_vit_g
state_dict = torch.load(cached_file, map_location="cpu")
File "D:\biancheng_soft\conda1\envs\all\lib\site-packages\torch\serialization.py", line 797, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
File "D:\biancheng_soft\conda1\envs\all\lib\site-packages\torch\serialization.py", line 283, in init
super().init(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory
是不是缺少了什么配置，或者预训练模型。。

XrayPULSE中的PULSE_MODEL多卡加载

XrayPULSE中的PULSE_MODEL支持多卡加载吗？

使用的是PULSE-7Bv5，GPU为Tesla T4，15G显存，仍然OOM……

RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

Great work, but I get this error when I try to run the demo. I looked it up because the file XrayPULSE_ckpt.pth is corrupted. Can you reproduce it using the provided file 'XrayPULSE_ckpt.pth'?

Chinese version dataset request

Thank you for your good work!

Could you share the Chinese-version Xray-Report paired data (MIMIC-CXR and OpenI) ? Thanks very much

Installation部分有小错误

git clone https://github.com/openmedlab/XrayPULSE.git
cd XrayPLUSE

这里cd后面拼写错误