otto-de / jlineup

An easy-to-use visual acceptance test tool to automatically detect design changes of your website. Works on the command line or as a web service.

Status

About

JLineup is a tool which is useful for automated visual regression tests of web pages, especially in continuous delivery pipelines. It can be used as a simple command line tool or as a small web service which is controlled via REST API.

JLineup shoots and compares screenshots of a web page at two consecutive points in time. It does a pixel by pixel comparison of both runs and generates a HTML and a JSON report. Behind the scenes, it uses Selenium and a browser of choice (currently Chrome, Chromium and Firefox are supported).

JLineup has no other dependencies than web browser (Firefox or Chrome/Chromium) and a JVM. Experimental: There's also a self-contained Linux AMD64 build of the CLI version that doesn't even require a JVM.

Example

Let's take this little example config for a check of otto.de during a deployment:

{
  "urls": {
    "https://www.otto.de": {
      "paths": [ 
	    "/"
      ],
      "devices" : [ {
        "width" : 850,
        "height" : 600,
        "pixel-ratio" : 1.0,
        "device-name" : "DESKTOP",
        "touch" : false
      }, {
        "width" : 1000,
        "height" : 850,
        "pixel-ratio" : 1.0,
        "device-name" : "DESKTOP",
        "touch" : false
      }, {
        "width" : 1200,
        "height" : 1000,
        "pixel-ratio" : 1.0,
        "device-name" : "DESKTOP",
        "touch" : false
      } ]
    }
  },
  "wait-after-page-load" : 0.5,
  "browser" : "chrome-headless"
}

JLineup runs before and after the deployment and generates a report like this:

There's also a JSON report, which is great if you want to check things by script:

Example JLineup JSON Report

Quick Howto

JLineup CLI comes as executable Java Archive. Java 17 or higher has to be available to run it.

Open a terminal and download it like this:

wget https://repo1.maven.org/maven2/de/otto/jlineup-cli/4.12.1/jlineup-cli-4.12.1.jar -O jlineup.jar

Then type

java -jar jlineup.jar --help

to see the command line help.

See the CLI documentation for more details and a small tutorial.

Integration example

This is an example, how JLineup can be helpful in your automated build and deploy pipeline. Let's assume, this is part of a continuous integration pipeline:

Browser Compatibility

JLineup 4.11 was tested successfully with

• Chrome 126.x
• Firefox 127.x

Chrome or Firefox have to be installed on the system if you want to use one of them.

Documentation

JLineup as CLI tool

JLineup as web server

JLineup Job Configuration

Third Party Libraries

JLineup uses some third party tools and libraries

Selenium

• Selenium is licensed under the Apache 2.0 License.

Webdrivermanager

• Webdrivermanager is licensed under the Apache 2.0 License.

Jackson

• Jackson is licensed under the Apache 2.0 License.

Logback

• Logback is licensed under the Eclipse Public License.
• The SLF4J API is licensed under the MIT License.

Thymeleaf

• Thymeleaf is licensed under the Apache 2.0 License.

Edison Microservice

• Edison Microservice is licensed under the Apache 2.0 License.

Spring Boot

• Spring Boot is licensed under the Apache 2.0 License.

GraalVM Community Edition

• The binary cli version is built with GraalVM Community Edition. GraalVM CE brings it's own Product License.

Image Compare Viewer

• Image Compare Viewer is licensed under the MIT License.

Google Fonts

• Google Fonts are used in the HTML report. These are the terms.

AWS SDK

• AWS SDK for Java 2.0 is licensed under the Apache 2.0 License.

Pixelmatch

Some code from Pixelmatch was ported to Java for JLineup.

• Pixelmatch is licensed under the ISC License.

Looks-same

Some code from Looks-same was ported to Java for JLineup.

• Looks-same is licensed under the MIT License.

Historic Facts

JLineup is a configuration compatible replacement for Lineup, implemented in Java. The original Lineup was a Ruby tool, but is not maintained any more.

Credit for original Lineup goes to Finn Lorbeer.

Contact

If you have questions or proposals, please open an issue or write an email to marco DOT geweke AT otto.de

jlineup's Issues

Does the max-diff value gets rounded internally?

I have a comparision with a very small difference value (0.00036101398601399). When I use exactly that value for the max-diff config value, the check fails. Increasing it to something like 0.00037101398601399 works, so I wondered if there is some rounding done internally?

JLinueup should delete chrome profile after run to free up diskspace

Currently chrome profiles are not deleted after a jlineup run. We noticed that a profile may take up ~300MB of disk space, so that the disk runs full quite rapidly when you have many jlineup runs.

Strange behavior when url parameters are used as url key

after step does not recognice existing before step when running with delay

Hi,

I just noticed a strange behavior that seems to be existing since 4.4.0. 4.3.4 works fine:

I build a service that runs two comparisions: one where the after step runs right after the before step (no issues), and one where the before step runs, then some software updates are triggered, and after the updates are finished (might take a few minutes), the after step is exectued. This works with 4.3.4, but since 4.4.0, the after step seems to not notice the before step. It just creates additional folders for the screenshots, instead of adding the after screenshots in the folders created by the before step, and the report.json only contains screenshot-after-file-name entries, screenshot-before-file-name is null.

Persistence

JLineup Web doesn't have any persistence besides the screenshots and the generated reports on disk. To make it possible to restart the server without loosing currently running or pending jobs, we need a persistence.

User-Agent in der jlineup.json setzen

Hallo,

momentan ist es nicht möglich den User-Agent in der jlineup.json zu setzen.
Wäre ggf. ein Feature Request?

Gruß
David

display browser version in jlineup report

It would be great to see the browser version in order to see if the bug shown in a report depends on a certain version.

Fix code scanning alert - Implicit narrowing conversion in compound assignment

Tracking issue for:

• https://github.com/otto-de/jlineup/security/code-scanning/1

report.json beinhaltet nicht das max-diff

In der report.json fehlt momentan die Information ob ein Test fehlgeschlagen ist. Es wird zwar das difference ausgewiesen, dieses muss dann aber manuell wieder gegen das max-dif aus der lineup.json verglichen werden. Wäre es nicht Sinnvoll ein failure Status oder das max-diff auch in das report.json mit aufzunehmen?

Gruß
David

Create a readme.md file

Hi
Could you create a readme.md file to help with the start and using the jlineup tool and some tips as to how would you use it and also test scenarios where some page interaction is required to go to the desired page like login or click or something else ??

thanks ....

`device-config` needs to be `device` in docs

First, thanks a lot for this tool!

I switched from the window-widths to device config recently and tried the device-config key like described in the docs (https://github.com/otto-de/jlineup/blob/master/docs/CONFIGURATION.md#device-config), but got an error for that. The key device worked.

Create a web frontend for jobs

The webserver part of JLineup should have a nice overview of running and recent Jobs. At the moment, the edison status detail indicator is (mis)used for this.

Make job errors accessible through the job status

At the moment, errors and exceptions, that are thrown during a JLineup web run, go to the configured log on the web server. It would be nice to have at least the root error cause in the job status.

Feature Request: Call custom URL before and after a step

It would be nice to call custom URLs before and / or after JLineup run steps. This would enable you to create and remove test data through custom controllers in your 'JLineupped' service.

CLI jar size suddenly increased

Since two or three versions, the CLI jar increased in size by ~30 MB, please investigate.

Reveal run id after before run

It would be great to retrieve the 'before' run id after requesting the /runs endpoint either via body response or redirect in order to get the url to request for the 'after' phase

faulty dependency in 3.0.0-rc1 pom

In the pom of 3.0.0-rc1 jlineup-core is not properly referenced.
Is:

<dependency>
    <groupId>de.otto</groupId>
    <artifactId>core</artifactId>
    <version>3.0.0-rc1</version>
    <scope>compile</scope>
</dependency>

Should:

<dependency>
    <groupId>de.otto</groupId>
    <artifactId>jlineup-core</artifactId>
    <version>3.0.0-rc1</version>
    <scope>compile</scope>
</dependency>

This leads to problems if you want to use jlineup in gradle:
Execution failed for task ':jlineupBefore'.

Could not resolve all files for configuration ':runtime'.
> Could not find de.otto:core:3.0.0-rc1.
Searched in the following locations:
https://repo1.maven.org/maven2/de/otto/core/3.0.0-rc1/core-3.0.0-rc1.pom
https://repo1.maven.org/maven2/de/otto/core/3.0.0-rc1/core-3.0.0-rc1.jar
Required by:
project : > de.otto:jlineup:3.0.0-rc1