theocoombes / clipcap

Using pretrained encoder and language models to generate captions from multimedia inputs.

ClipCap

Using pretrained encoder and language models to generate captions from multimedia inputs, allowing high fidelity text generation using the rich textual detail already learned by pretrained LMs on tasks such as image captioning, VQA, audio captioning and more.

More details and results to come soon.

Installation

By default, the encoders remained uninstalled for ease of access. View the data preprocessing documentation for info on how to install these.

pip install git+https://github.com/TheoCoombes/ClipCap.git

Supported Encoders

• CLIP for tasks such as Image Captioning, VQA etc.
• CLAP for tasks such as Audio Captioning, Audio Question Answering, etc.

Data Preprocessing

You can run the data preprocess script using the command below. (More info)

python3 -m clipcap.preprocess --help

Training

You can run the training script using preprocessed data with the command below. (More info)

python3 -m clipcap.train --help

Acknowledgments

This repository is heavily based on @rmokady's original implementation of ClipCap and also contains modified versions of @rom1504's clip-inference and embedding-reader libraries. Many thanks to both for their amazing work :)

TODO

Improved documentation and eval + inference scripts to come soon.

clipcap's Issues

inference

hello,thx for your codes,can you share the packages version and the pretrained model,thx

train and release models

to begin with, train on coco, and have clipcap.load_pretrained("coco_global_vit_b_32")

Evaluation using pre-trained model

Hello,

I love your work, really impressive stuff! I'm working on something similar and was wondering if you might have a pretrained model I could play around with for some basic tests.

Thanks!

minimal usage instruction

Install with pip install ClipClap

use with

import clipclap

model = clipclap.load_pretrained()
text = clipclap.generate(PIL.open("https://some/img"))

text = clipclap.generate(my_clip_embedding)

(this is an example, please tune the API as you like)

Release a pretrained model and add inference example