jacobmarks / concept-space-traversal-plugin

Traverse the space of concepts with a multi-modal similarity index in FiftyOne

Concept Space Traversal Plugin

This plugin allows you to "traverse" the concept space of a similarity index by adding text prompts (with configurable strength) to a base image.

Note: This plugin requires a similarity index that supports prompts (i.e. embeds text and images) to be present on the dataset. You can create one with:

import fiftyone as fo
import fiftyone.brain as fob

dataset = fo.load_dataset("my_dataset")
fob.compute_similarity(
    dataset,
    brain_key="my_brain_key",
    model="clip-vit-base32-torch",
    metric="cosine",
    )

Watch On Youtube

Installation

fiftyone plugins download https://github.com/jacobmarks/concept-space-traversal-plugin

Refer to the main README for more information about managing downloaded plugins and developing plugins locally.

Operators

open_traversal_panel

• Opens the concept space traversal panel on click
• Only activated when the dataset has a similarity index

traverser

• Runs the Traverser on the dataset

get_sample_url

• Returns the resource URL of the media file associated with a sample, from the sample's ID.

💪 Development

This plugin was a joint creation between myself and Ibrahim Manjra. Couldn't have done it without his JavaScript wizardry!