yusufpapurcu / pepper

Pepper

Type-safe, composable, extensible test matching for Go

Inspired by Hamcrest

When writing tests it is sometimes difficult to get the balance right between over specifying the test (and making it brittle to changes), and not specifying enough (making the test less valuable since it continues to pass even when the thing being tested is broken). Having a tool that allows you to pick out precisely the aspect under test and describe the values it should have, to a controlled level of precision, helps greatly in writing tests that are “just right”. Such tests fail when the behaviour of the aspect under test deviates from the expected behaviour, yet continue to pass when minor, unrelated changes to the behaviour are made.

From The Hamcrest Introduction

Out of the box, Pepper can work just like other test libraries in Go like is, where you can make basic assertions on data.

Simple examples

Expect(t, "Pepper").To(Equal("Pepper"))

Pepper has lots of built-in matchers, which you pass in to To, for common testing operations such as examining comparable, string, io and *http.Response.

What is a Matcher[T] ? It's a function that takes a T, and returns a MatchResult

type Matcher[T any] func(T) MatchResult

Here is the definition of MatchResult

type MatchResult struct {
    Description string
    Matches     bool
    But         string
    SubjectName string
}

Here is how HaveAllCaps is defined

func HaveAllCaps(in string) matching.MatchResult {
	return matching.MatchResult{
		Description: "be in all caps",
		Matches:     strings.ToUpper(in) == in,
		But:         "it was not in all caps",
	}
}

Expect(t, "HELLO").To(HaveAllCaps)

Quite nice, but not all that different from libraries you already use. Pepper starts to come into its own when you start taking advantage of composing matchers.

Composing matchers

Expect(t, score).To(GreaterThan(5).And(LessThan(10)))

The method And on Matcher[T], lets you compose matchers. And returns the composed Matcher[T], so you can continue to chain more matchers however you like.

Expect(t, score).To(GreaterThan(5).And(Not(GreaterThan(opponentScore))))

Not negates a matcher. By using matchers, and composing them with And, Not, Or, you can write very expressive tests, cheaply.

Defining your own matchers

You can define your own matchers for your own types. Over time, the investment in writing matchers for your tests pays dividends, the cost of writing your tests decrease, as you reuse, mix and match the standard matchers and composition tools with your own.

Some will argue writing these matchers adds more code as if that's inherently a bad thing, but I would argue that the tests read far better, and don't suffer the problems you can run in to if you lazily assert on complex types.

In my experience of using matchers, over time as you find yourself testing more and more permutations of behaviour, the effort behind the matchers pays off in terms of making tests easier to write, read and maintain.

Here is an example of testing a todo-list

type Todo struct {
    Name        string    `json:"name"`
    Completed   bool      `json:"completed"`
    LastUpdated time.Time `json:"last_updated"`
}

func WithCompletedTODO(todo Todo) MatchResult {
    return MatchResult{
        Description: "have a completed todo",
        Matches:     todo.Completed,
        But:         "it wasn't complete",
    }
}
func WithTodoNameOf(todoName string) Matcher[Todo] {
    return func(todo Todo) MatchResult {
        return MatchResult{
            Description: fmt.Sprintf("have a todo name of %q", todoName),
            Matches:     todo.Name == todoName,
            But:         fmt.Sprintf("it was %q", todo.Name),
        }
    }
}

func TestTodos(t *testing.T) {
    t.Run("with completed todo", func(t *testing.T) {
        res := httptest.NewRecorder()
        res.Body.WriteString(`{"name": "Finish the side project", "completed": true}`)
        Expect(t, res.Result()).To(HaveBody(Parse[Todo](WithCompletedTODO)))
    })

    t.Run("with a todo name", func(t *testing.T) {
        res := httptest.NewRecorder()
        res.Body.WriteString(`{"name": "Finish the side project", "completed": false}`)
        Expect(t, res.Result()).To(HaveBody(Parse[Todo](WithTodoNameOf("Finish the side project"))))
    })

    t.Run("compose the matchers", func(t *testing.T) {
        res := httptest.NewRecorder()

        res.Body.WriteString(`{"name": "Egg", "completed": false}`)
        res.Header().Add("content-type", "application/json")

        Expect(t, res.Result()).To(
            BeOK,
            HaveJSONHeader,
            HaveBody(Parse[Todo](WithTodoNameOf("Egg").And(Not(WithCompletedTODO)))),
        )
    })
})

Note how we can compose built-in matchers like BeOK, HaveJSONHeader and Not, with the custom-built matchers to easily write very expressive tests that fail with very clear error messages. Pepper makes it really easy to check JSON responses of your HTTP handlers.

Due to the compositional nature of Matcher[T], we can re-use our Matcher[Todo] for tests at different abstraction levels; these matchers are not coupled to HTTP, we composed the matchers for this context. For instance, if you have a TodoRepository, or a TodoService, you can use these same matchers in the tests for them too.

Test failure readability

One of the most frustrating areas working with automated tests is how often test failure quality is poor. I'm sure every developer has into this scenario:

test_foo.go:123 - true was not equal to false

Computer, I already know that true is not equal to false. What was not false? What was true? What was the context?

Pepper makes it easy for you to write tests that give you a clear message when they fail.

t.Run("failure message", func(t *testing.T) {
    res := httptest.NewRecorder()
    res.WriteHeader(http.StatusNotFound)
    Expect(t, res.Result()).To(BeOK)
})

Here is the failing output

=== RUN   TestHTTPTestMatchers/Status_code_matchers/OK/failure_message
    matchers_test.go:292: expected the response to have status of 200, but it was 404

Embracing this approach with well-written matchers means you get readable test failures for free.

Summary

Pepper brings the following to the table

• ✅ Type-safe tests. No interface{}
• ✅ Composition to reduce boilerplate
• ✅ Clear test output as a first-class citizen
• ✅ A "standard library" of matchers to let you quickly write expressive tests for common scenarios out of the box
• ✅ Extensibility. You can write rich, re-useable matchers for your domain to help you write high quality, low maintenance tests
• ❌ Not a framework. Pepper does not dictate how you set up your tests or how you design your code. All it does is help you write better assertions with less effort.

Examples

You can find high-level examples in the GoDoc.

The matchers, which you can find in the directories section also have examples.

You'll notice in the examples the following line:

t := &SpyTB{}

This is a test spy that is used to verify the output of the matches made. The examples call LastError() to see what test output would happen, so you can see what the failures look like.

Finally, I have worked through my course Learn Go with Tests, using Pepper to write assertions, so you can find more examples at the Github repository

Trade-offs and optimisations

Type-safety

Now Go has generics, we have a more expressive language which lets us make matchers that are type-safe, rather than relying on reflection. The problem with reflection is it can lead to lazy test writing, especially when you're dealing with complex types. Developers can lazily assert on complex types, which makes the tests harder to follow and more brittle. See the curse of asserting on irrelevant detail for more on this.

The trade-off we're making here is you will have to make your own matchers for your own types at times. This is a good thing, as it forces you to think about what you're actually testing, and it makes your tests more readable and less brittle.

Due to the compositional nature of the library though, you should be able to leverage existing matchers for re-use. For example, you'll never have to make a matcher that parses JSON, you should instead use Parse in combination with a matcher of your type.

Due to Go having some constraints on where you can use generics, such as function types not being allowed to have type parameters, the API isn't as friendly as it would be, if you used interface{}/any. However, this is a trade-off I am OK with, in the name of type-safety.

Composition and re-use

Matchers should be designed with composition in mind. For instance, let's take a look at the body matcher for an HTTP response:

func HaveBody(bodyMatchers pepper.Matcher[io.Reader]) pepper.Matcher[*http.Response]

This allows the user to re-use io.Reader matchers that are already defined, compose them with And/Or/Not, and of course users can define their own Matcher[io.Reader] too.

Test failure readability

Often the most expensive part of a test suite is trying to understand what has failed when a test goes red. This is why TDD emphasises the first step of writing a failing test and inspecting the output. It's a chance for you to see what it's like if the test fails 6 months later, and you've lost all context.

Pepper strives to make it easy for you to write tests that explain exactly what has gone wrong, but it does require a little more upfront effort when writing your own matchers.

type MatchResult struct {
	Description string
	Matches     bool
	But         string
	SubjectName string
}

The bare minimum you need to write a matcher is to complete the Matches field, but to get good test failure messages, you'll need to invest a little time filling the other fields and checking the failure reads nicely.

Pepper will not bend over backwards to write perfect English. It's important that the reason for test failure is clear, but perfect grammar is not needed for this; and the complexity cost involved to make matchers "write" different sentences depending on how they are used, is not worth it.

Dot imports?

As you'll see from the examples, Pepper is using "Dot imports", where you import the Pepper package, and other matching package and alias it to .

Here is the import statement for the HTTP matchers test file as an example

import (
	"fmt"
	. "github.com/quii/pepper"
	. "github.com/quii/pepper/matchers/comparable"
	. "github.com/quii/pepper/matchers/http"
	. "github.com/quii/pepper/matchers/io"
	. "github.com/quii/pepper/matchers/json"
	. "github.com/quii/pepper/matchers/spytb"
	"net/http"
	"net/http/httptest"
	"testing"
	"time"
)

This allows you to write Expert(t, "Pepper").To(Equal("Pepper")) rather than, pepper.Expect(t, "Pepper").To(comparable.Equal("Pepper")). Of course, as a consumer of Pepper, you can choose which style you prefer.

Dot imports have a number of documented downsides, perhaps most importantly, they are not part of the Go 1 backward compatibility guarantee

If a program imports a standard package using import . "path", additional names defined in the imported package in future releases may conflict with other names defined in the program. We do not recommend the use of import . outside of tests, and using it may cause a program to fail to compile in future releases.

(Emphasis mine)

Other potential issues:

• Shaky IDE support. Some commentary on the internet says some editors/IDE struggle with dot imports, but I do not find this is true with Intellij at least. I can "jump to definition" on a dot imported symbol, and it will still take me to the correct source.
• May make code less easy to understand. In my view, I would tentatively agree, I like seeing package names generally in Go code, and it helps me design coherent packages, but in trying to make a library like Pepper, it does cause in my view a fair amount of unnecessary noise. In particular, the more composition you do, the more repeated package name instances you'll see, and given matcher composition is one of the main design goals of Pepper, this is a problem!

In short, this is a trade-off between fluency of writing test code, and a slight obfuscation as to where symbols like EqualTo, Parse etc. come from. Whilst I will use dot imports, and document examples with it, you as a consumer, are free to avoid it if you wish.

Benefits of matchers

A lot of the time people zero-in on the "fluency" of matchers. Whilst it's true that the ease of use does make using matchers attractive, I think there's a larger, perhaps less obvious benefit.

The curse of asserting on irrelevant detail

A trap many fall in to when they write tests is they end up writing tests with greedy assertions where you end up lazily writing tests where you check one complex object equals another.

Often when we write a test, we only really care about the state of one field in a struct, yet when we are greedy, we end up coupling our tests to other data needlessly. This makes the test:

• Brittle, if domain logic changes elsewhere that happens to change values you weren't interested in, your test will fail
• Difficult to read, it's less obvious which effect you were hoping to exercise

Matchers allow you to write domain specific matching code, focused on the specific effects you're looking for. When used well, with a domain-centric, well-designed codebase, you tend to build a useful library of matchers that you can re-use and compose to write clear, consistently written, less brittle tests.

How to write your own matchers

Reminder: a matcher is a function that takes the thing you're trying to match against, returning a result

type Matcher[T any] func(T) MatchResult

Here is the definition of MatchResult

type MatchResult struct {
    Description string
    Matches     bool
    But         string
    SubjectName string
}

Here is how HaveAllCaps is defined

func HaveAllCaps(in string) matching.MatchResult {
	return matching.MatchResult{
		Description: "be in all caps",
		Matches:     strings.ToUpper(in) == in,
		But:         "it was not in all caps",
	}
}

Higher-order matchers

This is fine for simple matchers where you want to assert on a static property of T. Often though, you'll want to write matchers where you want to check a particular _ value of a property_ .

For this, no magic is required, just create a higher-order function that returns a Matcher[T].

A simple example is with EqualTo

func EqualTo[T comparable](in T) matching.Matcher[T] {
	return func(got T) matching.MatchResult {
		return matching.MatchResult{
			Description: fmt.Sprintf("be equal to %v", in),
			Matches:     got == in,
			But:         fmt.Sprintf("it was %v", got),
		}
	}
}

Leveraging composition

When designing your higher-order matchers, think about how the value you are matching against could be matched with other matchers you may not have thought of.

For example, HasSize, I could've written like this:

func HaveSize[T any](size int) pepper.Matcher[[]T]

However, this needlessly couples the matcher to the specific matching I was currently catering for (logically EqualTo). Instead, we can design our matcher to be combined with other matchers that work on int.

func HaveSize[T any](matcher pepper.Matcher[int]) pepper.Matcher[[]T] {
	return func(items []T) pepper.MatchResult {
		return matcher(len(items))
	}
}

This way, users can use this matcher in different ways, like checking if a slice has a size LessThan(5) or GreaterThan(3).

With this simple change, users can also leverage the other composition tools like And:

Expect(t, catsInAHotel).To(HaveSize(GreaterThan(3).And(LessThan(10))))

Tip: When designing your matcher, consider changing the argument(s) from T to Matcher[T].

Test support

Pepper makes testing matchers easy because you inject in the testing framework into Expect, so we can spy on it.

I have found writing testable examples to be a satisfying way of both documenting and testing matchers.

func ExampleContainItem() {
	t := &SpyTB{}

	anArray := []string{"HELLO", "WORLD"}
	Expect(t, anArray).To(ContainItem(HaveAllCaps))

	fmt.Println(t.LastError())
	//Output:
}

func ExampleContainItem_fail() {
	t := &SpyTB{}

	anArray := []string{"hello", "world"}
	Expect(t, anArray).To(ContainItem(HaveAllCaps))

	fmt.Println(t.LastError())
	//Output: expected [hello world] to contain an item in all caps, but it did not
}

However, if you wish to check multiple scenarios, polluting the go doc with lots of examples may not be appropriate, in which case, write some unit tests.

Check out some of the unit tests for some of the comparison matchers

func TestComparisonMatchers(t *testing.T) {
	t.Run("Less than", func(t *testing.T) {
		t.Run("passing", func(t *testing.T) {
			Expect(t, 5).To(LessThan(6))
		})

		t.Run("failing", func(t *testing.T) {
			spytb.VerifyFailingMatcher(t, 6, LessThan(6), "expected 6 to be less than 6")
			spytb.VerifyFailingMatcher(t, 6, LessThan(3), "expected 6 to be less than 3")
		})
	})

	t.Run("Greater than", func(t *testing.T) {
		t.Run("passing", func(t *testing.T) {
			Expect(t, 5).To(GreaterThan(4))
		})

		t.Run("failing", func(t *testing.T) {
			spytb.VerifyFailingMatcher(t, 6, GreaterThan(6), "expected 6 to be greater than 6")
			spytb.VerifyFailingMatcher(t, 2, GreaterThan(10), "expected 2 to be greater than 10")
		})
	})
}

Contributing your own matchers

If you have a matcher you think would be useful to others, please consider contributing it to this library.

Please only submit matchers that work against types in the standard library. This keeps the library focused and backward compatible. It would be fantastic if over time this library matured into a rich suite of matchers so any dev can pick up Go and start writing excellent tests against the standard library, which already gets you so far in terms of getting work done.

Your PR will need the following

• At least two testable examples, one showing the matcher passing (with an empty output) and one showing the matcher failing with the expected failing output. This will help users understand how to use your matcher.
• Automated tests in general
• Go doc comments for the matcher

As discussed above, try to keep them "open" in terms of their design, so they can be composed with other matchers.