apiaryio / gavel-spec

Behavior specification for Gavel, validator of HTTP transactions

Behavior spec for Gavel, validator of HTTP transactions

What is Gavel?

Gavel detects important differences between actual and expected HTTP transactions (HTTP request and response pairs). Gavel also decides whether the actual HTTP transaction is valid or not.

Implementations

• JavaScript
• Didn't find your favorite language? Add a new one!

Documentation

• Gavel on Relish

About gavel-spec

This repository contains implementation-independent behavior specification of Gavel. It's written in Gherkin, language used by Cucumber. Two main benefits are:

• Any Gavel implementation can be tested against the specification, which ensures the behavior is uniform.
• Documentation at Relish is generated from the specification, which ensures it's always up-to-date.

Examples are made in raw HTTP to focus on implementation independence.

Usage

To use the specification in your project and test against it, install it as npm package (or include as a git submodule):

$ npm install gavel-spec
...
$ find ./node_modules/gavel-spec/features/**/*.feature
./node_modules/gavel-spec/features/expectations/body_json_example.feature
./node_modules/gavel-spec/features/expectations/body_text_example.feature
./node_modules/gavel-spec/features/expectations/headers.feature
./node_modules/gavel-spec/features/expectations/status_code.feature
...

It's also possible to get path to the directory with features from JavaScript:

var featuresPath = require('gavel-spec').featuresPath;
console.log(featuresPath);  // prints '/.../node_modules/gavel-spec/features/'

Use Cucumber for testing.

Publishing Documentation

Currently publishing of a new version of the generated documentation is done manually:

1. Install Relish CLI: gem install relish
2. Add current version: relish versions:add apiary/gavel:1.1.0 (the 1.1.0 should be the same number which appears on npm as the latest package version)
3. Publish new version of the documentation: relish push apiary/gavel:1.1.0 path ./features/

Steps 2-3 has been simplified as npm run docs:publish command.

gavel-spec's Issues

Hey i want to use Gavel in conjunction with Dredd.

Hi everyone,
Thanks for an awesome library, i want to use Gavel with Dredd, Dredd only supports validating 2xx responses, i want to skip the validation by dredd and use my own. I want to use Gavel to check the types of the Values in the response body returned vs response Body expected. It will be a great help if you can tell me how to do this! thanks !! 😀

Consider "actual" vs "real" terminology

Per @kylef's suggestion:

Most similiar assertion (test) style functions both in NodeJS (https://nodejs.org/api/all.html#assert_assert_equal_actual_expected_message), chai (https://www.chaijs.com/api/bdd/#method_fail), etc use the terminology "expected" and "actual".

Would it make sense to rename real to actual for consistency?

CLI tests won't run on Windows

The CLI behavior spec is written in a way which is very hard or maybe even impossible to get running on Windows. It uses pipes, grep, ... 😱

Missing specification for validation of request URI

Original README states something like this:

[URI](https://www.relishapp.com/apiary/gavel/docs/expectations/request-uri)

The page doesn't exists and surrounding formatting indicates it's an incomplete TO-DO item.

Rebuild Relic documentation

Current live Relic documentation of Gavel seems to use v 3, which is far behind the latest published version (7.0.2).

Describe behavior of the URI validation

During implementation of the latest spec, I've added more specifics on the behavior of URI validation. Instead of writing them into a comment, I should have write them as a scenarios in the Gherkin file and send them as a Pull Request for discussion. This didn't happen, so the spec now misses crucial examples describing specifics of URI validation. https://relishapp.com/apiary/gavel/v/3-0-0/docs/expectations/uri only contains 1:1 string comparison.

Define unified Gavel validation result shape

We must define a unified validation result Object structure for Gavel.

Motivation

1. To become independent from the JSON validators we use or may use. Any validator's output is coerced to the unified validation result structure, preventing us from breaking changes.
2. To contain common data that is currently missing (i.e. actual/expected values of individual fields, normalize pointers format, adjust fields results to be easily printed out).

Move Gavel spec to Dredd's monorepo

...once the monorepo exists :)

Missing specification for validation of request method

Original README states something like this:

[Method](https://www.relishapp.com/apiary/gavel/docs/expectations/request-method)

The page doesn't exists and surrounding formatting indicates it's an incomplete TO-DO item.

Broken link in gavel-spec/features/README.md

The link to https://www.relishapp.com/apiary/gavel/docs/expectations/body-json-schema in gavel-spec/features/README.md is broken.

I'm not sure if this is the right place to report the issue. I can't find contact info at https://relishapp.com/apiary/gavel/docs.

README is outdated

It should undergo some facelift. With respect to #13 and #14, the largest confusion now is that ABOUT.md is better README for this particular repository than README.md.

Optional keys in body response

I have this kind of error documented in my blueprint:

{
    "message": "Validation failed",
    "errors": {
        "fullname": [ "Can't be blank" ],
        "email": [ "Can't be blank" ],
        "password": [ "Can't be blank" ]
    }
}

the problem is that sometime, people will just get that

{
    "message": "Validation failed",
    "errors": {
        "email": [ "Can't be blank" ],
    }
}

and it will set an error for the body. (fullname, password are missing)
Is there something planned to handle this kind of case with optional keys?

Replace Relish links with GitHub links

Relish is closing, according to #57 (comment). Before we have time to tackle #21, we should remove Relish from everywhere, replace it with links to Gherkin files on GitHub, and close the account.

• replace all references to https://relishapp.com/ in gavel-spec docs (README, etc.)
• replace all references to https://relishapp.com/ in Gavel docs (README, etc.)
• remove publishing scripts from Gavel package.json and who knows where else
• replace all references to https://relishapp.com/ in Dredd docs (both README and Sphinx docs)
• replace all references to https://relishapp.com/ in the core app (code comments, ...)

Use valid JavaScript code in feature code blocks

The current code accounts to be evaluated in CoffeScript, which is a too specific context. I suggest to include code blocks for features written in valid JavaScript (ECMA2015), to be both copy-paste-able, and properly evaluated during GavelJS testing pipeline.

Relish is unusable for hosting documentation - find a replacement

Relish has several flaws:

• Publishing has to be manual, because the relish gem prompts for credentials interactively (maybe workaround exists?).
• HTML rendered from correct Markdown often contains erroneous output (list items squashed, incorrect formatting of code as a link label, ...).
• The generated ToC is sorted alphabetically instead of according to the contents of .nav. There is no way to sort them in a way useful for a reader.
• There is no way to get a nice name for pure markdown files in ToC - e.g. add-implementation.md is displayed as Add-implementation in the ToC and on the page itself and there's no way to override it.

For that reason, if anyone knows how to efficiently document something together with gherkin scenarios e.g. in Sphinx, please let us know. We will gladly replace Relish with something else.

@node_js tag is incorrect

It's rather JavaScript - it runs also in browser now.

Feature definitions not marked as "javascript" are not tested in "gavel.js"

We must tag all the features for JavaScript implementation with the @javascript tag. As a follow up, we could assess if such tag is even needed, as gavel-spec doesn't have any other implementations.

Does gavel have patternProperties support?

I'm using Dredd to run our documentation through CI and came across the following...

One of my resources returns datetimes as keys that I need to validate:

{
  "data": {
    "2013-10-14T09:00": {
      ...
    }
  }
}

I tried adding the following to the Schema section of my API blueprint doc, which allows Dredd/gavel to run, but not actually validate:

{
  "type": "object",
  "required": true,
  "properties": {
    "data": {
      "type": "object",
      "required": true,
      "patternProperties": {
        "^[0-9]{4}-[0-9]{2}-[0-9]{2}$": {
          "type": "object",
          "required": true,
          ...
        }
      }
    }
  }
}

I noticed that the JSON Schema spec defines a patternProperties that allows you to use a regex:

"patternProperties": {
  "^(/[^/]+)+$": {}
},

Does gavel support patternProperties? I ack'd the code and didn't see it. Happy to contribute, but just wanted to make sure gavel doesn't already handle this type of validation in some other way.

Replace graphml/png diagram with something else

• The asset isn't used anywhere at the moment
• It would be nice to represent its contents in something more convenient, such as graphviz (it even has Sphinx support) or mermaid

Provide proper loose/strict values diff behavior

There is a feature description that says Gavel should return a validation error upon validating an array that misses certain values opposed to expected array:

gavel-spec/features/expectations/body_json_example.feature

Lines 79 to 95 in 9f68503

	Scenario: Array member is missing in real JSON body
	When real HTTP body is following:
	"""
	{
	"object": {
	"a": "bau bau",
	"c": "boo boo",
	"e": "mrau mrau"
	},
	"array": [
	1
	],
	"string": "Foo bar"
	}
	"""
	Then Gavel will set some error for "body"
	And Request or Response is NOT valid

It appears this hasn't been working properly in the current version of Gavel, and thus this behavior is removed for now.

However, we must bring it back once it's properly supported in Gavel.

Background

The reason this feature never worked in gavel is:

1. There has been an invalid JSON example #39, which produced false positive upon validating this feature suit.
2. When the JSON is valid, there is no error thrown in Gavel, which fails the feature tests.
3. The validation error is not thrown because the JsonSchema created from the expected JSON has valuesScript: false, which means it allows more or less values between given and expected JSON.

Document how docs are generated

Add information on how the docs are generated and deployed.

Make this repository installable by npm

Installation by npm will make including the features to other projects easy. Anyone is welcome to contribute packaging also to other package indexes, but for gavel.js, npm is a priority.

Ideally, the package would also be versioned by Semantic Release. That could potentially make packaging to other indexes hard, but:

• There are no current requests for other types of packages
• It should be possible to script around missing version in package.json when publishing other kind of package
• It's easy to install npm packages also to non-JS projects
• From maintainer's point of view, it's very very convenient to use Semantic Release

Roadmap for this issue:

• Rename gavel to gavel-spec: #14
• Introduce package.json with package name gavel-spec
• Introduce semantic release
  • Migrate changelog (if there's one) to GitHub Releases
  • Make the repository commitizen-friendly
  • Introduce Semantic Releasing
• Remove or generate the VERSION file (is it related to http://www.relishapp.com/apiary/gavel/docs or is it present just for denoting version and nothing else relies on it?)
• Update https://github.com/apiaryio/gavel.js to use gavel spec from npm - apiaryio/gavel.js#70

Consider requirement keywords from RFC2119

Per @kylef's suggestion:

As soon as I read this with capital NOT I couldn't stop thinking about it missing SHOULD/MUST/SHALL from RFC2119, perhaps something to consider using:

#44 (comment)

Rename repository to gavel-spec

Since after several years of existence of Gavel its only implementation is Gavel.js, it would be convenient to put stress on the implementation and let this repository to fade out a bit. I propose to rename this repository from gavel to gavel-spec and rewrite its README so it clearly denotes its purpose as a container for the implementation-independent Cucumber specification.

In contrast to that, Gavel.js should be presented with all the love and documentation as the main implementation and as the project which people most likely want to use and study in case of problems.

Anyone will be still welcome to provide alternative implementations for gavel-spec, of course.