🤗 Hugging Face   |   🤖 ModelScope   |    📑 Blog   |    📑 Paper   

We foresee a future where content understanding and generation are seamlessly unified within a single model. To this end, we have launched the WePOINTS project. WePOINTS is a suite of multimodal models designed to create a unified framework that accommodates various modalities. These models are being developed by researchers at WeChat AI, leveraging the latest advancements and cutting-edge techniques in multimodal models.

2024.11.02 Add the demo script to filter the pre-training data by perplexity.
2024.10.15 We released POINTS with Qwen2.5-7B 🔥🔥🔥.
2024.10.05 We open-sourced the inference code of POINTS🔥🔥🔥.
2024.09.07 We released the paper about the first vision-language model, POINTS🚀🚀🚀.
2024.05.20 We released the paper to reveal some overlooked aspects in vision-language models🚀🚀🚀.

git clone https://github.com/WePOINTS/WePOINTS.git
cd WePOINTS
pip install -e .

We provide the usage of POINTS, using Hugging Face 🤗 transformers library.

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import CLIPImageProcessor
from PIL import Image
import torch
import requests
from io import BytesIO


image_url = 'https://github.com/user-attachments/assets/83258e94-5d61-48ef-a87f-80dd9d895524'
response = requests.get(image_url)
image_data = BytesIO(response.content)
pil_image = Image.open(image_data)
prompt = 'please describe the image in detail'
model_path = 'WePOINTS/POINTS-Yi-1-5-9B-Chat'
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path, trust_remote_code=True, device_map='cuda').to(torch.bfloat16)
image_processor = CLIPImageProcessor.from_pretrained(model_path)
generation_config = {
    'max_new_tokens': 1024,
    'temperature': 0.0,
    'top_p': 0.0,
    'num_beams': 1,
}
res = model.chat(
    pil_image,
    prompt,
    tokenizer,
    image_processor,
    True,
    generation_config
)
print(res)

We use VLMEvalKit to evaluate the performance of our models. Please follow the installation instructions to install the toolkit. Then, you can evaluate POINTS using the following command:

# before running the evaluation below, please enter the root directory of VLMEvalKit
torchrun --nproc-per-node=8 --master_port=8888 --no-python python run.py --data MMMU_DEV_VAL MMBench_DEV_EN MMBench_TEST_EN_V11 MMBench_TEST_CN_V11 HallusionBench OCRBench AI2D_TEST MMStar MMVet MathVista_MINI MME RealWorldQA LLaVABench POPE  --model POINTS-Yi-1.5-9B-Chat --verbose --work-dir ./

POINTS is the pioneering work that proposes applying model soup to models fine-tuned with different instruction datasets. This approach combines the benefits of these models and enhances the performance of the final averaged model. The following code snippet demonstrates how to use model soup to combine the models:

from wepoints.utils import model_soup

# models fine-tuned with different instruction datasets
model_paths_or_names = [
  'first_model_path',
  'second_model_path',
  'third_model_path'
]
save_path = '/path/to/save/model'
model_soup(model_paths_or_names, save_path)

CATTY is a brand new strategy to split a large-resolution image into small patches of the same size. Compared to previsou approaches, CATTY can preserve the original image aspect ratio.

from PIL import Image
from wepoints.utils.images.catty import split_image_with_catty

image_path = '/path/to/local/image.jpg'
# used to save the split images for debugging
save_folder = '/path/to/save/folder'
image = Image.open(image_path)
sub_images = split_image_with_catty(image, save_folder=save_folder, do_resize=True)

We provide a script to filter the pre-training data by perplexity using Qwen2VL. Please first the download the demo pre-training data here. And then run the following command:

python scripts/pretrain_filtering_with_ppl.py --model_name Qwen2VL --model_path /path/to/model --original_file_path data.jsonl --filtered_file_path filtered_data.jsonl

If you find our work helpful, feel free to cite us:

@article{liu2024points,
  title={POINTS: Improving Your Vision-language Model with Affordable Strategies},
  author={Liu, Yuan and Zhao, Zhongyin and Zhuang, Ziyuan and Tian, Le and Zhou, Xiao and Zhou, Jie},
  journal={arXiv preprint arXiv:2409.04828},
  year={2024}
}

@article{liu2024rethinking,
  title={Rethinking Overlooked Aspects in Vision-Language Models},
  author={Liu, Yuan and Tian, Le and Zhou, Xiao and Zhou, Jie},
  journal={arXiv preprint arXiv:2405.11850},
  year={2024}
}