GitHubHelp

Visual regression tests issues

#5 · closed · 3 comments

View on GitHub ↗

PromoFaux

## What happened? Visual regression tests appear to fail on the `main` branch. Initially I thought it was the PR branches I had created, but tested on a freshly cloned copy of this repo ## What did you expect to happen? Visual regression tests to pass! ## Steps to reproduce 1. Checkout this repo 2. docker compose up 3. bash test.sh all ## Environment - Stift version / commit: 800fbec5faa166e6fb35dd1727cd85078ce5a0ce - Browser + version: Chrome 146.0.7680.178 - OS: WSL (Windows) - Self-hosted (Docker / Caddy / other) or hosted instance: Self-hosted ## Console / server logs ``` stift-1 | Starting nginx... stift-1 | nginx: [alert] could not open error log file: open() "/var/lib/nginx/logs/error.log" failed (13: Permission denied) stift-1 | Starting Stift API server... stift-1 | Admin token: 1b2be5ee-8416-43f9-939c-af013bd8ee96 (auto-generated; set ADMIN_TOKEN env var to override) stift-1 | Default max projects per user: 50 stift-1 | Registration: open stift-1 | Stift API server listening on port 3001 stift-1 | 127.0.0.1 - - [09/Apr/2026:21:22:54 +0000] "GET /api/config HTTP/1.1" 200 135 "-" "Wget" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:22:55 +0000] "GET / HTTP/1.1" 200 540 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:22:55 +0000] "GET /assets/index-BjRrZwqS.css HTTP/1.1" 200 5553 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:22:55 +0000] "GET /assets/index-BVWNWbk2.js HTTP/1.1" 200 391745 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:22:55 +0000] "GET /stift.svg HTTP/1.1" 200 739 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:22:55 +0000] "GET /api/config HTTP/1.1" 200 140 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:22:56 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:22:56 +0000] "GET /api/config HTTP/1.1" 200 140 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:00 +0000] "GET /api/config HTTP/1.1" 200 140 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:01 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:01 +0000] "GET /api/config HTTP/1.1" 200 140 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:04 +0000] "GET /api/config HTTP/1.1" 200 140 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:05 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:05 +0000] "GET /api/config HTTP/1.1" 200 140 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:08 +0000] "GET /api/config HTTP/1.1" 200 140 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:09 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:10 +0000] "GET /api/config HTTP/1.1" 200 140 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:11 +0000] "GET /examples/server-rack.jpg HTTP/1.1" 200 250819 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:13 +0000] "GET /api/config HTTP/1.1" 200 140 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:14 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:14 +0000] "GET /api/config HTTP/1.1" 200 140 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:17 +0000] "GET /api/config HTTP/1.1" 200 140 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:18 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:18 +0000] "GET /api/config HTTP/1.1" 200 140 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 172.20.0.1 - - [09/Apr/2026:21:23:20 +0000] "GET /examples/solar-panels.jpg HTTP/1.1" 200 185977 "http://host.docker.internal:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.7103.25 Safari/537.36" stift-1 | 127.0.0.1 - - [09/Apr/2026:21:23:21 +0000] "GET /api/config HTTP/1.1" 200 135 "-" "Wget" stift-1 | 127.0.0.1 - - [09/Apr/2026:21:23:49 +0000] "GET /api/config HTTP/1.1" 200 135 "-" "Wget" stift-1 | 127.0.0.1 - - [09/Apr/2026:21:24:16 +0000] "GET /api/config HTTP/1.1" 200 135 "-" "Wget" stift-1 | 127.0.0.1 - - [09/Apr/2026:21:24:43 +0000] "GET /api/config HTTP/1.1" 200 135 "-" "Wget" stift-1 | 127.0.0.1 - - [09/Apr/2026:21:25:11 +0000] "GET /api/config HTTP/1.1" 200 135 "-" "Wget" ``` ``` Running all tests... === Visual regression tests === added 2 packages in 1s FAIL: Bug Report (diff: 99.47%, threshold: 1.00%) Actual saved to /screenshots/example1_actual.png FAIL: PCB Inspection (diff: 99.16%, threshold: 1.00%) Actual saved to /screenshots/example2_actual.png FAIL: Weld Analysis (diff: 99.41%, threshold: 1.00%) Actual saved to /screenshots/example3_actual.png FAIL: Server Room Audit (diff: 99.48%, threshold: 1.00%) Actual saved to /screenshots/example4_actual.png FAIL: Bridge Inspection (diff: 99.44%, threshold: 1.00%) Actual saved to /screenshots/example5_actual.png FAIL: Solar Panel Array (diff: 99.48%, threshold: 1.00%) Actual saved to /screenshots/example6_actual.png Some visual regression tests failed. ``` ## Screenshots <img width="2112" height="1382" alt="Image" src="https://github.com/user-attachments/assets/cdf1a91c-5d5c-4be8-b9f4-84069dd7c183" />

Comments

m13v

99% diff on every test screams baseline mismatch, not actual regressions. the baselines were probably captured on a different resolution, OS, or browser version than what the test runner is using now. the fix is to re-record baselines inside the same Docker container that runs the tests, so the rendering environment is identical. the fact that you're running Chrome 146 in a Docker container on WSL adds two layers of rendering variability (WSL's display handling plus Docker's headless Chrome). try adding --disable-gpu and --font-render-hinting=none flags to the Chromium launch to get more deterministic output. also check if the screenshots are being captured at the same viewport size as the baselines, Playwright defaults can differ from whatever generated the original reference images.

m13v

we open sourced a framework that handles the browser rendering consistency problem across CI environments. the browser lifecycle management with deterministic viewport and rendering config: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/core/browser.ts and the test execution engine that normalizes screenshot output: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/core/freestyle.ts

DL6ER

Fixed in e1b1ae0. The reference images were captured with default Chromium rendering settings, which produce different pixels on different machines due to font hinting, GPU acceleration, and sub-pixel positioning. That's why every test showed ~99% diff - not actual regressions, just rendering environment mismatch. Added four flags to all Playwright Chromium launches: ``` --disable-gpu --font-render-hinting=none --disable-lcd-text --disable-font-subpixel-positioning ``` These force CPU rasterization with integer pixel positioning and no platform-specific font tricks, producing identical output across Linux, macOS, WSL, and CI without having to add another dependency. Thanks for the suggestion though @m13v Reference screenshots have been re-generated with these flags. On a clean checkout you should now be able to run `bash test.sh visual` and get 0.00% diff on all six examples, as long as the Stift instance is running and reachable at the URL the test expects (default `http://host.docker.internal:8080`). Thanks for reporting!