tingerrr / typst-test

A visual regression test runner for typst packages.

rust & typst enthusiast

typst-test's Issues

Diff images are not being generated for page size differences

Built from commit version: ce06d7e
Rust: 1.75.0
Platform: Windows 11 Home 22H2
Repository: https://github.com/fenjalien/metro/

Running typst-test run works as expected apart from the diff images are not being generated in the diff folders. I've linked the repository I'm having the issue with.

Allow custom default tests

Currently whenever a new test is created it is populated with "Hello World!". I then have to copy and paste in the package's import statement and other setup which is common to all tests I want to run.

I would like a way to change the default test on a per project basis.

Debounce package downloads

When running multiple tests while not having any packages cached on the system, the multiple instances of Typst can accidentally clear directories for packages another instance has just created, leading to ghostwrites. See typst#2739

While this is technically a Typst issue, it arises mostly when running typst-test in its current form.

This will be resolved with the coming rework, which will ship its own Typst world implementation.

Extract core behavior into a library crate

By extracting the core implementation of test scaffolding and test running, other projects such as typst-lsp and typst-preview could benefit from it without needing to interface with the bare bones binary interface.

Typst-preview already supports a special presentation mode for polylux, it's not too far-fetched to assume they could be interested in a special test-watching mode. Likewise, typst-lsp could allow discovering and running tests easily.

Custom user actions

Allow users to run custom actions between test stages.

On compelling use case for this is being able to test typstfmt (see #3). This avoids pulling it in as a dependency or even tying it to a command line option of typst-test at all.

With sufficiently flexible custom actions, typstfmt could generate the references from the unaltered test script and the output from the altered version, failing if the formatting introduce a visual regression.

Compile-only tests

Add some kind of mechanism to mark tests which do not have references and are meant only to be compiled.

Just omitting the references could be enough, an option on new and edit could be added to select if a test should be compared or not.

Error:unexpected character '\n' after patch version number

I encountered the same issue as in #19.

When running a test, I get the message:

typst-test ran into an unexpected error, this is most likely a bug
Please consider reporting this at https://github.com/tingerrr/typst-test/issues/new
    Error:unexpected character '\n' after patch version number

Here is what I did:

cargo install typst-test --git https://github.com/tingerrr/typst-test
typst-test init
typst-test add test-1
typst-test run test1

The installation command gives the following warning:

   Compiling typst-test v0.1.0 (/Users/xxx/.cargo/git/checkouts/typst-test-af24935a6e1ceb4c/97b1839/crates/typst-test)
warning: variants `RootNotFound` and `InitNeeded` are never constructed
   --> crates/typst-test/src/project/mod.rs:398:5
    |
396 | pub enum Error {
    |          ----- variants in this enum
397 |     #[error("project not found: {0:?}")]
398 |     RootNotFound(PathBuf),
    |     ^^^^^^^^^^^^
...
404 |     InitNeeded,
    |     ^^^^^^^^^^
    |
    = note: `Error` has a derived impl for the trait `Debug`, but this is intentionally ignored during dead code analysis
    = note: `#[warn(dead_code)]` on by default

warning: `typst-test` (bin "typst-test") generated 1 warning

I ran the commands in several of my packages and even downloaded some other packages and always got the same error message.

I tried the ci-semi-stabel branch with no luck. The ci-semi-stable tag works for me, though.

Change folder structure to one subdirectory per test

I propose changing the folder structure, such that each test file lives in its own folder. A test folder contains a Typst file (e.g. "test.typ") plus the reference image.

CeTZ currently uses its own test script, which structures the tests as given above, but we would like to use typst-test.

CeTZ's tests are recognized by recursively scanning the tests/ directory for test.typ files and, if found, use the folder of those files as test-cases. This allows nesting/grouping tests by topic by just putting them in common parent directories.

Generate better diff images

When generating a diff image, the following things could be improved:

• generate diff images on a page size mismatch (#8)
• highlight changes, bounding boxes of small change sets can be created very easily using a border finding algorithm (see for example Suzuki and Abe ¹)
• highlight which kind of change it is
  • red: deletion from reference to output
  • green: addition from reference to output
• overlay these boxes on the output or reference image to make it easier to visually compare

Footnotes

1. https://www.nevis.columbia.edu/~vgenty/public/suzuki_et_al.pdf ↩

Testing formatters

Looks really nice!

Maybe it could be use to test formatters as well!

Given a formatter and a test file

• Tests that the compilation gives the same result before and after formatting
• Else pinpoints where it broke the code file
  Apply diffs with dichotomy and test the output, it's a bit of a headache to implement because the diff types I used were hard to understand I think

It could happily live next to typstfmt when I get up to it but I think it might be better suited in typst-test

Add some figure to quantify a reference mismatch

for example the number of mismatched pixels.

This would be very helpful in the case when tests fail due to extremely minor changes. Especially PNG rendering is sometimes subject to minute differences even when PDF/SVG output is not (at least visually) affected.

Thanks for this project - I've been using it for some months now and it is amazing :)

Error running typst-test after init: unexpected character '\n' after patch version number

Error:

$ typst-test update
typst-test ran into an unexpected error, this is most likely a bug
Please consider reporting this at https://github.com/tingerrr/typst-test/issues/new

typst.toml

[package]
name = "mypackage"
version = "0.0.2"
entrypoint = "lib.typ"
authors = ["Keyneston"]
repository = "https://github.com/keyneston/mypackage"
license = "MIT"

Feature request: `--watch` flag

The typst-test program executes sufficiently rapidly that power users may benefit from a --watch flag that runs the test suite when it detects a change in any project files.

It may be more involved than just adding a flag, however, as people are likely to want to exclude or include specific files or filetypes, so a configuration service may also be needed (which is a lot of bloat)

No reference store and comparison

To avoid repository bloat, don't save images and instead store some kind of symbolic reference to another storage and a way to retrieve this reference from the given storage.

An external store comes with some security, performance and usability issues, but may be worth it in the long run.

The main motivation for this is that images are not meant to be stored as is in a git repository. Solutions such as Git LFS on GitHub are horrendously overpriced for FOSS repositories. Another issue with storing images in a git repository is that they are often optimized to account for aforementioned bloat. This optimization can take quite long even on updates which introduce only a minor change to the reference, as comparison is not perceptual, but exact.

See more on a related discussion on the Typst community discord here.

Compile tests in memory

By adding typst as a dependency, test output generation and comparison could be done entirely in memory. This seems like a sensible thing to do, here are some of the pros and cons:

Pros

• less I/O by doing in-memory reference to image comparison
  • increased performance
• typst-test can be run without an external binary for typst, reducing error paths and improving usability
• multiple tests can be parsed from a single test file with most of the config living in this file
  • avoids config files for configuring per test behavior
• allows for separation of typst-test into a library crate and binary crate
  • opens the door for others to use the same underlying implementation for test running

Cons

• fixes typst-test to a typst version for its default invocation
• the default behavior of typst-test may diverge from the user's installed typst compiler on a version mismatch
• massively increased dependency tree and binary size
• increased complexity of the implementation
  • many implementations will have to be copied verbatim from typst-cli to ensure the same world behavior

Rework and cleanup core implementation

The current implementation is a little brittle, this has a lot of downsides:

• error reporting is sometimes done using ?, other times it's swallowed and simply printed to stdout
• some actually likely error cases are not properly handled and result in a simple error message with no hint
• as it currently is, reporting a nonzero exit code on test failure would need an ugly hack
  • this is currently the primary blocker for adoption of typst-test in cetz and in code bases which test using CI in general

The core should be reworked into a solid foundation on which new features can be built. At this stage, the project is still fairly small, and a rewrite is feasible.

I want to achieve the following goals with the rewrite:

• Abstract user facing reporting sufficiently to allow in place progress per test for multiple tests
• Properly handle common error cases better
• Make code easier to follow and reason about
• Make code, which depends on FS operations, easier to test automatically
• Abstract the testing in a way that makes separation into a core library and cli frontend easier down the line

`add` command removes everything when creating a test directly in an existing folder

With a folder structure like test/higher/lower/test.typ and no test.typ in higher/. Using the add command removes everything in higher/.

False positives in test runs because of relaxed comparison

The change introduced in 97b1839 was rather hasty and caused even simple mismatches to be ignored.

A user could initialize a project, remove the d in the default example test, run the tests and observe a pass. The metrics chosen for the relaxed comparison were not good enough and need to be opt in.

ci-semis-table has since been rolled back to the previous commit.