asgaardlab / canvas-visual-bugs-testbed

Visual testing framework for PixiJS applications

Visual testing for PixiJS applications

Automated visual testing framework for PixiJS applications. Early in development and does not yet work for all PixiJS apps (see Issues).

Note: To access the replication package for our paper "Automatically Detecting Visual Bugs in HTML5 <canvas> Games":

• download the source from the p1 tag
• download the data from Zenodo

Features

• Collect snapshots from a PixiJS application; snapshots contain a screenshot paired with a scene graph
• Perform automatic visual comparisons between oracles (generated during the test) and the objects rendered to the <canvas>
• Supports automated test scripts written with Playwright in TypeScript

Note: To detect visual bugs, must set baseline (acceptable) error scores and then perform regression testing. This functionality will be added to the framework in a future release.

Prerequisites

Python (v3.4+)

Download and install the latest version of Python from this link. After installing Python, ensure Python's package manager (Pip) is installed by running the command:

python3 -m pip install ---upgrade pip

Node.js

Using the instructions found at this link, first install Node Version Manager (nvm), and then use nvm to install Node and Node Package Manager (npm).

Installation

1. Download the latest release from the releases page
2. Unzip the folder and place it inside your test directory
3. Open the folder in your terminal and run the command npm install -D

Note: Initial installation may take a long time due to Python dependencies

Usage

1. Collect snapshots by instrumenting automated test scripts
2. Run visual tests on collected snapshots with Python script

• Snapshot name needs to be known on both ends

1. Collecting snapshots

• instrument your test code to capture snapshots, which are pairs of (screenshot, scene graph)
• sampling API requires reference to Playwright browser Page instance
• sampling API requires call to inject into the Page instance before samples can be taken

Importing:

import { PixiSamplerPlaywright } from 'pixi-visual-test/sampling/PixiSamplerPlaywright'

Instantiating:

/** @param {playwright.Page} page: Playwright browser page where PixiJS app is running */
const sampler = new PixiSamplerPlaywright(page, 'test/snapshots')

Injecting:

await sampler.startExposing();

Sampling:

await sampler.takeSnapshot('<name_of_snapshot>');

2. Running visual tests

Run python3 ./pixi-visual-test test/snapshots/<name_of_snapshot>

Examples

See test/ directory for a bare minimum example test script made for our toy example PixiJS game.

To run the example, enter the command npm run test from the root directory of this repository.

canvas-visual-bugs-testbed's Issues

Possible problems with objects with spritesheet and atlas resources

No identifier for Spritesheet / Atlas resources means the framework is limited for some games (?)

Sampling does not work for games in iframes

When the web game is contained in an iframe, the PixiSamplerClient object cannot yet access the iframe's DOM.

Starting point for solving this might be this answer on StackOverflow: https://stackoverflow.com/a/36795207

Parser does not always work for all PixiJS applications

Some scene graphs are constructed differently from others, currently the PixiParser.ts code only correctly parses out the objects for certain games.