google / butteraugli

butteraugli estimates the psychovisual difference between two images

butteraugli

A tool for measuring perceived differences between images

Introduction

Butteraugli is a project that estimates the psychovisual similarity of two images. It gives a score for the images that is reliable in the domain of barely noticeable differences. Butteraugli not only gives a scalar score, but also computes a spatial map of the level of differences.

One of the main motivations for this project is the statistical differences in location and density of different color receptors, particularly the low density of blue cones in the fovea. Another motivation comes from more accurate modeling of ganglion cells, particularly the frequency space inhibition.

Use

Butteraugli can work as a quality metric for lossy image and video compression. On our small test corpus butteraugli performs better than our implementations of the reference methods, psnrhsv-m, ssim, and our yuv-color-space variant of ssim. One possible use is to define the quality level setting used in a jpeg compressor, or to compare two or more compression methods at the same level of psychovisual differences.

Butteraugli is intended to be a research tool more than a practical tool for choosing compression formats. We don't know how well butteraugli performs with major deformations -- we have mostly tuned it within a small range of quality, roughly corresponding to jpeg qualities 90 to 95.

Interface

Only a C++ interface is provided. The interface takes two images and outputs a map together with a scalar value defining the difference. The scalar value can be compared to two reference values that divide the value space into three experience classes: 'great', 'acceptable' and 'not acceptable'.

Build instructions

Install Bazel by following the instructions. Run bazel build -c opt //:butteraugli in the directory that contains this README file to build the command-line utility. If you want to use Butteraugli as a library, depend on the //:butteraugli_lib target.

Alternatively, you can use the Makefile provided in the butteraugli directory, after ensuring that libpng and libjpeg are installed. On some systems you might need to also install corresponding -dev packages.

The code is portable and also compiles on Windows after defining _CRT_SECURE_NO_WARNINGS in the project settings.

Command-line utility {#cmdline-tool}

Butteraugli, apart from the library, comes bundled with a comparison tool. The comparison tool supports PNG and JPG images as inputs. To compare images, run:

butteraugli image1.{png|jpg} image2.{png|jpg}

The tool can also produce a heatmap of differences between images. The heatmap will be output as a PNM image. To produce one, run:

butteraugli image1.{png|jpg} image2.{png|jpg} heatmap.pnm

butteraugli's Issues

new_http_archive is not defined when build

when i git clone the master and try to build it , i follow the instructions
install bazel success , and my environment is mac os

➜  butteraugli git:(master) bazel version
Build label: 0.23.1
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Mar 4 10:40:32 2019 (1551696032)
Build timestamp: 1551696032
Build timestamp as int: 1551696032

then i follow the command ,but it appeared below ,
i am sorry for boring you ,but i can not find way
the code is just git clone from the respository and run on the master

➜  butteraugli git:(master)
➜  butteraugli git:(master)  bazel build -c opt //:butteraugli
ERROR: /Users/admin/work/learn/30.picture_/code_/butteraugli/WORKSPACE:3:1: name 'new_http_archive' is not defined
ERROR: /Users/admin/work/learn/30.picture_/code_/butteraugli/WORKSPACE:11:1: name 'new_http_archive' is not defined
ERROR: /Users/admin/work/learn/30.picture_/code_/butteraugli/WORKSPACE:19:1: name 'new_http_archive' is not defined
ERROR: Error evaluating WORKSPACE file
ERROR: error loading package '': Encountered error while reading extension file 'tools/build_defs/repo/http.bzl': no such package '@bazel_tools//tools/build_defs/repo': error loading package 'external': Could not load //external package
ERROR: error loading package '': Encountered error while reading extension file 'tools/build_defs/repo/http.bzl': no such package '@bazel_tools//tools/build_defs/repo': error loading package 'external': Could not load //external package
INFO: Elapsed time: 0.098s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
➜  butteraugli git:(master)

Make{Low, High}FreqColorDiff{Dx, Dy}() functions incorrectly initialize the lut array

The three functions declare the lut table using std::array<>, but do not assign 0 to the zeroth element of the array. I'm not a C++11 spec expert but under Visual Studio 2017 this leads to guard values /trash being assigned to lut[0]. This has a nasty effect on mask computations, as it now gets filled with INF values which ultimately leads to a division by zero exception.

Issues with taking the maximum

Currently the score that Butteraugli outputs is the maximum over the entire image. If I understood correctly, the rationale is that errors (high scores) are noticeable, so even if there is a large error in even a small part of the image, this is unforgivable. However, this has a few implications:

• The maximum is sensitive to small changes. The pixel with the maximum error could be an outlier. If an image is altered only slightly, the worst error might be in a different part of the image, and the score will change by a lot. Something like the 90th or 98th percentile would still penalize large errors in small regions, while being more stable.
• The converse of penalizing large errors in small regions, is that improvements on large regions are not reflected in the score at all, if there is no improvement in the region where the score is the worst.

Of course, capturing both in one single number is difficult. Outputting a distribution or even just a few percentiles would be more useful, but it would also make the output harder to interpret and compare.

Clarification needed on the input RGB format

The butteraugli.h header contains this comment:

// Value of pixels of images rgb0 and rgb1 need to be represented as raw
// intensity. Most image formats store gamma corrected intensity in pixel
// values. This gamma correction has to be removed, by applying the following
// function:
// butteraugli_val = 255.0 * pow(png_val / 255.0, gamma);
// A typical value of gamma is 2.2. It is usually stored in the image header.
// Take care not to confuse that value with its inverse. The gamma value should
// be always greater than one.
// Butteraugli does not work as intended if the caller does not perform
// gamma correction.

Is this correct and a conversion to "raw intensity" is needed for the input images?
And if so is the result of the above mentioned gamma correction that the input is in linear RGB? So for example when applied to standard definition video with matrix '601' the frames should be converted to linear RGB before applying Butteraugli? Thanks!

Is there any suggest about the butteraugli compare result value?

I using butteraugli compare two image, the result value is "0.971608", according the result value, how can I know the two image match is perfect or ok or bad?

Evaluate butteraugli on TID-2013 and publish results

http://www.ponomarenko.info/tid2013.htm
Metrics are ranked by their Spearman correlation with Mean Opinion Score.

Someone who has hardware powerful enough to make it practical could do this. It would be a real proof (or maybe disproof) that this metric is superior to others.

VS 2017 does not accept ConstRestrict

The compilation error is:

1>butteraugli.h(137): error C2219: syntax error: type qualifier must be after '*' 1>butteraugli.h(258): error C2219: syntax error: type qualifier must be after '*' 1>butteraugli.h(258): note: see reference to alias template instantiation 'ConstRestrict<ComponentType*>' being compiled 1>butteraugli.h(302): note: see reference to class template instantiation 'butteraugli::Image<ComponentType>' being compiled

It looks like somebody has already done some research on this issue here: https://stackoverflow.com/questions/42869470/should-msvc-reject-an-templated-using-without-instantiation . From what they're saying, it seems like it's a genuine flaw in the code but I haven't had time to dig into the C++ spec.

A simple workaround for this glitch is changing the alias declaration to something along the lines of:

using ConstRestrict = T const;

Tag a release so this can go on homebrew

See related discussion on Homebrew/homebrew-core#1377

Butteraugli Score 0.000

Hi, I get a butteraugli 0 score when processing this images.
Butteraugli can be in a range between 1.0 and 1.5 (when quality is 95), but I think 0.0 is wrong.
What's wrong?

my terminal

input image

guetzli output image (-q 95)

reopen

reopen

it's too slow (takes more than a minute to compare 12MP images)

I used the makefile to build it and… it's prohibitively slow.

a install problem.

ERROR: in target '//external:cc_toolchain': no such package '@local_config_cc//': Traceback (most recent call last):
File "C:/users/administrator/appdata/local/temp/_bazel_administrator/-qthx2dm/external/bazel_tools/tools/cpp/cc_configure.bzl", line 788
_find_env_vars(repository_ctx, vc_path)
File "C:/users/administrator/appdata/local/temp/_bazel_administrator/-qthx2dm/external/bazel_tools/tools/cpp/cc_configure.bzl", line 567, in _find_env_vars
_find_vcvarsall_bat_script(repository_ctx, vc_path)
File "C:/users/administrator/appdata/local/temp/_bazel_administrator/-qthx2dm/external/bazel_tools/tools/cpp/cc_configure.bzl", line 560, in _find_vcvarsall_bat_script
repository_ctx.path(vcvarsall).exists
Illegal char <"> at index 97: C:\users\administrator\appdata\local\temp_bazel_administrator-qthx2dm\external\local_config_cc"D:\Program Files\VS2017"\VC\Auxiliary\Build\VCVARSALL.BAT.
INFO: Elapsed time: 0.774s

memory leak?

butteraugli/butteraugli.cc:73: error: MEMORY_LEAK
memory dynamically allocated by call to malloc() at line 61, column 47 is not reachable after line 73, column 1.

Google butteraugli

TID2013 scores

FYI, scores for 290e395 measured against TID2013 database:

TID2013 Category	Spearman correlation
Noise	-0.822
Actual	-0.853
Simple	-0.915
Exotic	-0.543
New	-0.818
Color	-0.843
Full	-0.634

Butteraugli

Butteraugli 😍😍❤️✨️

Migrate libjpeg to libjpeg turbo

libjpeg-turbo is a JPEG image codec that uses SIMD instructions (MMX, SSE2, NEON, AltiVec) to accelerate baseline JPEG compression and decompression on x86, x86-64, ARM, and PowerPC systems. On such systems, libjpeg-turbo is generally 2-6x as fast as libjpeg, all else being equal.

macOS Monterey build failed

external/png_archive/pngwutil.c:14:10: fatal error: 'pngpriv.h' file not found
#include "pngpriv.h"

Please add version output

Please add a command line parameter to print the version of butteraugli.

Easy to use, but hard to learn

Butteraugli is easy to use, but very hard to understand. Is it possible to release some introduction material? It will be very helpfu..!

Error while trying to build

I am using g++ (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
I get the following error while trying to build using MakeFile

butteraugli_main.cc:190:65: error: invalid conversion from ‘JSAMPROW {aka char*}’ to ‘const uint8_t* {aka const unsigned char*}’ [-fpermissive]
const uint8_t* const BUTTERAUGLI_RESTRICT row = buffer[0];
~~~~~~~~^
butteraugli_main.cc:209:65: error: invalid conversion from ‘JSAMPROW {aka char*}’ to ‘const uint8_t* {aka const unsigned char*}’ [-fpermissive]
const uint8_t* const BUTTERAUGLI_RESTRICT row = buffer[0];

Question about butteraugli's threshold value

Dear all:

There is something we are not certain about. We test butteraugli on this image.
Original:

Compressed JPEG compression quality 90

According to butteraugli's result, the distmap's value is above good threshold and even bad threshold, so the difference can be visible. But according to our experiment, human cannot observe the difference at all. And the butteraugli' results keep the same even when the quality reached 98. Is there any problems with our understanding. Many thanks.

Compilation issues under VS2017

Apart from the two major issues I've reported in other tickets, I also had to resolve the following glitches in the code to get it to build:

1. CacheAlignedUniquePtrT definition: T[] had to be changed to T
2. Image definition: I had to manually specify copy assignment operator for the class.
3. ButteraugliComparator: I had to unconstify the member fields.

Please consider modifying the source code so that other folks don't have to go through the same process. Thanks.

Butteraugli failing due to difference in number of channels

butteraugli bright-gray-room.png bright-gray-room.png_mozjpeg_q99.jpg
Different number of channels: 4 vs 3

Test-files : https://drive.google.com/file/d/0B_s7n4-sdChAZ1JhcmlvblJMcUU/view?usp=sharing

cite publications about this programme

Hi,

Do you know is there any publications of this butteraugli programme people can cite. Many thanks.

Integrating with OSS-Fuzz

Greetings butteraugli developers and contributors,

We’re reaching out because your project is an important part of the open source ecosystem, and we’d like to invite you to integrate with our fuzzing service, OSS-Fuzz. OSS-Fuzz is a free fuzzing infrastructure you can use to identify security vulnerabilities and stability bugs in your project. OSS-Fuzz will:

• Continuously run at scale all the fuzzers you write.
• Alert you when it finds issues.
• Automatically close issues after they’ve been fixed by a commit.

Many widely used open source projects like OpenSSL, FFmpeg, LibreOffice, and ImageMagick are fuzzing via OSS-Fuzz, which helps them find and remediate critical issues.

Even though typical integrations can be done in < 100 LoC, we have a reward program in place which aims to recognize folks who are not just contributing to open source, but are also working hard to make it more secure.

We want to stress that anyone who meets the eligibility criteria and integrates a project with OSS-Fuzz is eligible for a reward.

To help you getting started, we can attach our internal fuzzer for your project that you are welcome to use directly, or to use it as a starting point.

If you're not interested in integrating with OSS-Fuzz, it would be helpful for us to understand why—lack of interest, lack of time, or something else—so we can better support projects like yours in the future.

If we’ve missed your question in our FAQ, feel free to reply or reach out to us at [email protected].

Thanks!

Tommy
OSS-Fuzz Team

Compile for Windows

Can someone compile this for WIndows please??
Thank you!!

No error when comparing images with transparent areas

When you compare images of different formats , the output is a value.I think an error should be displayed in such a situation.

warning: 'assume_aligned' attribute directive ignored

This warning appears during compilation on RedHat and ubuntu

More info : http://stackoverflow.com/questions/41694735/warning-assume-aligned-attribute-directive-ignored

butteraugli: command not found

Hi,
I'm struggling to understand the usage for buggeraugli. After install bazel, I run the command:
bazel build -c opt //:butteraugli, and it printed:“Build completed successfully, 84 total actions”. Then, I run the command butteraugli ori.png rec.png. "butteraugli: command not found." In this directory, there are bazel-bin, bazel-butteraugli, bazel-genfiles, bazel-out,bazel-testlogs. Is there anything wrong?
Could your please give me some advices?
Best regards.

Results

Not sure why this is happening, but butteraugli seems to ignore visible color differences in this case -

Original Image

90% Compression using guetzli

Notice how the green is clearly desaturated compared to the original.
Butteraugli score - 1.540664

90% Compression using mozjpeg

The greens, as well as the picture as a whole are much more closer to the original image.
Butteraugli score - 1.581931