laion-ai / aesthetic-predictor

A linear estimator on top of clip to predict the aesthetic quality of pictures

LAION-Aesthetics_Predictor V1

A linear estimator on top of clip to predict the aesthetic quality of pictures

Normal results:

Aesthetic results:

visualization over laion2B http://3080.rom1504.fr/aesthetic/aesthetic_viz.html

integrated into https://rom1504.github.io/clip-retrieval/

import os
import torch
import torch.nn as nn
from os.path import expanduser  # pylint: disable=import-outside-toplevel
from urllib.request import urlretrieve  # pylint: disable=import-outside-toplevel
def get_aesthetic_model(clip_model="vit_l_14"):
    """load the aethetic model"""
    home = expanduser("~")
    cache_folder = home + "/.cache/emb_reader"
    path_to_model = cache_folder + "/sa_0_4_"+clip_model+"_linear.pth"
    if not os.path.exists(path_to_model):
        os.makedirs(cache_folder, exist_ok=True)
        url_model = (
            "https://github.com/LAION-AI/aesthetic-predictor/blob/main/sa_0_4_"+clip_model+"_linear.pth?raw=true"
        )
        urlretrieve(url_model, path_to_model)
    if clip_model == "vit_l_14":
        m = nn.Linear(768, 1)
    elif clip_model == "vit_b_32":
        m = nn.Linear(512, 1)
    else:
        raise ValueError()
    s = torch.load(path_to_model)
    m.load_state_dict(s)
    m.eval()
    return m

That model takes clip embeddings as input

https://github.com/rom1504/embedding-reader/blob/main/examples/aesthetic_inference.py https://github.com/rom1504/embedding-reader/blob/main/examples/aesthetic_inference_emb.py

aesthetic-predictor's Issues

What is sa_0_4?

I was aware of @crowsonkb's ava aesthetic models, but these ones are new to me, what is the dataset/where the sa_0_4 come from?

efficient way to customize the model for more types of images

I have a use case where I have some type of images where this model doesn't work well, but works well for many of my images. If I want to add some more images to the training data and retrain what is the most efficient approach ?

fine-tuning the nn.Linear only ?

Did you train end-to-end or did you just tune the last Linear layer here only (and freeze the rest) ?

aes score exceed 10

I use clip feature of my image, and send it into the aes-predictor. While i get the image score of 20+ or 30+, which is labeled 0-10 of the trainning data.

Different from the result as your colab demo

I just run your colab demo but get different results. In my run, the aesthetic score is 5.04. But in your colab demo, it is 4.03. Is there any wrong with the provided checkpoint?

ViT-B/16 pre-computed embeddings?

Since you have the ViT-B/16 aesthetic model, was just wondering whether you would have the ViT-B/16 pre-computed embeddings too

Would you share the extracted clip-embedding of some LAION data?

It would be really helpful for downstream used of a well-prepared subset. Appreciated it.