autodistill / autodistill-fuyu

Fuyu multi-modal language model for use with Autodistill.

Autodistill Fuyu Module

This repository contains the code supporting the Fuyu base model for use with Autodistill.

Fuyu-8B, developed by Adept, is a multimodal language model. You can ask Fuyu a question about an image and retrieve a response. The Autodistill Fuyu integration enables you to use Fuyu for image classification.

Read the full Autodistill documentation.

Read the Fuyu Autodistill documentation.

Installation

To use Fuyu with autodistill, you need to install the following dependency:

pip3 install autodistill-fuyu

Quickstart

from autodistill_fuyu import Fuyu

# define an ontology to map class names to our Fuyu prompt
# the ontology dictionary has the format {caption: class}
# where caption is the prompt sent to the base model, and class is the label that will
# be saved for that caption in the generated annotations
# then, load the model
base_model = CLIP(
    ontology=CaptionOntology(
        {
            "person": "person",
            "a forklift": "forklift"
        }
    )
)
predictions = model.predict("image.png")

base_model.label("./context_images", extension=".jpeg")

Fuyu is licensed under a CC-BY-NC license.

🏆 Contributing

We love your input! Please see the core Autodistill contributing guide to get started. Thank you 🙏 to all our contributors!