typeable / comparest

Compatibility checker for OpenAPI

compaREST

Compatibility checker for OpenAPI

Using compaREST in Github Actions

Add the following step to your Github Actions workflow:

- uses: typeable/comparest
    if: ${{ github.event_name == 'pull_request' }}
    with:
      old: old-openapi.yaml
      new: new-openapi.yaml

The new and old values should be paths to your OpenAPI specifications you want to compare.

You will get something like this in your pull requests:

For more detail please see our integration guide.

An example

Your situation

You are developing a very important server with a REST API. You have clients who use your API that you do not control. Say, you are also developing a mobile app that uses your API and you can't force someone to update to the latest version. (Or you prefer not to for UX reasons.)

You have recently released version 1.0 and things are going great: user are downloading your app, servers are processing requests.

You describe your API in a file api-1.0.0.yaml:

openapi: "3.0.0"
info:
  version: 1.0.0
  title: Swagger Petstore
  license:
    name: MIT
servers:
  - url: https://example.com
paths:
  /pets:
    get:
      parameters:
        - name: limit
          in: query
          required: false
          schema:
            type: integer
            maximum: 20
      responses:
        "200":
          description: ""
          headers:
            x-next:
              schema:
                type: string
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Pets"
    post:
      requestBody:
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/Pet"
      responses:
        "201":
          description: ""
components:
  schemas:
    Pet:
      type: object
      required:
        - id
        - name
      properties:
        id:
          type: integer
        name:
          type: string
          minLength: 3
          maxLength: 10
    Pets:
      type: array
      items:
        $ref: "#/components/schemas/Pet"

Evolving your product

Enthused over your initial success you hurry to release a new and improved version of your API and mobile app.

After a round of very intense programming you take a look at your new api-1.1.0.yaml:

openapi: "3.0.0"
info:
  version: 1.1.0
  title: Swagger Petstore
  license:
    name: MIT
servers:
  - url: https://example.com
paths:
  /pets:
    get:
      parameters:
        - name: limit
          in: query
          required: false
          schema:
            type: integer
            maximum: 30
      responses:
        "200":
          description: ""
          headers:
            x-next:
              schema:
                type: string
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Pets"
    post:
      requestBody:
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/Pet"
      responses:
        "201":
          description: ""
components:
  schemas:
    Pet:
      type: object
      required:
        - id
        - name
      properties:
        id:
          type: integer
        name:
          type: string
          minLength: 1
          maxLength: 15
        weight:
          type: integer
    Pets:
      type: array
      items:
        $ref: "#/components/schemas/Pet"

Looking at the very large and complex API description, you grow more and more concerned that your old mobile app might stop working when you update the server. But the spec is too large and too complex to reasonably assess this manually.

Assessing compatibility automatically

Luckily, you have access to compaREST which can programmatically analyze your APIs and determine what, if anything, breaks compatibility and what doesn't.

You can call it, passing the API your client will be aware of, and the API your server will serve like so:

docker run --rm -v $(pwd):/data:rw typeable/comparest --client /data/api-1.0.0.yaml --server /data/api-1.1.0.yaml --output /data/report.md

Running this command will output a file report.md, containing the compatibility report between the two APIs:

Summary

❌ Breaking changes	⚠️ Non-breaking changes
5	6

❌ Breaking changes

GET /pets

⬅️☁️ JSON Response – 200

$[*].name(String)

1. Maximum length of the string changed from 10 to 15.

2. Minimum length of the string changed from 3 to 1.

POST /pets

➡️☁️ JSON Request

$.weight

1. Values are now limited to the following types:

   • Number

2. The property was previously implicitly described by the catch-all "additional properties" case. It is now explicitly defined.

$.weight(Number)

Value is now a multiple of 1.0.

⚠️ Non-breaking changes

GET /pets

Parameter limit

JSON Schema

$(Number)

Upper bound changed from 20.0 inclusive to 30.0 inclusive.

⬅️☁️ JSON Response – 200

$[*].weight

1. Values are now limited to the following types:

   • Number

2. The property was previously implicitly described by the catch-all "additional properties" case. It is now explicitly defined.

$[*].weight(Number)

Value is now a multiple of 1.0.

POST /pets

➡️☁️ JSON Request

$.name(String)

1. Maximum length of the string changed from 10 to 15.

2. Minimum length of the string changed from 3 to 1.

You now know exactly in what situations and in what way your 1.0 version of the app will break if you deploy your 1.1 version of the server.

Additional formats

You can also produce a self-contained HTML report that you can open in your browser by simply omitting the file extension of the output file:

docker run --rm -v $(pwd):/data:rw typeable/comparest --client /data/api-1.0.0.yaml --server /data/api-1.1.0.yaml --output /data/report

CLI docs

For more detail please see our user guide.

Usage: comparest (-c|--client ARG) (-s|--server ARG)
                 [--silent | --only-breaking | --all] [-o|--output ARG]
                 [--folding-block-quotes-style | --header-style]
                 [--signal-exit-code]
  A tool to check compatibility between two OpenAPI specifications.

  Usage examples

      Compare files old.yaml with new.yaml and output the resulting report to
      stdout:

          comparest -c old.yaml -s new.yaml

      Only output breaking changes and write a styled HTML report to file
      report.html:

          comparest -c old.yaml -s new.yaml --only-breaking -o report

      Don't output anything, only fail if there are breaking changes:

          comparest -c old.json -s new.json --silent

      Write full report suitable for embedding into a GitHub comment to
      report.html:

          comparest -c old.json -s new.json --folding-block-quotes-style -o report.html

Available options:
  -h,--help                Show this help text
  -c,--client ARG          A path to the file containing the specification that
                           will be used for the client of the API. Can be either
                           a YAML or JSON file.
  -s,--server ARG          A path to the file containing the specification that
                           will be used for the server of the API. Can be either
                           a YAML or JSON file.
  --silent                 Silence all output. Only makes sense in combination
                           with --signal-exit-code.
  --only-breaking          Only report breaking changes in the output.
  --all                    Report both incompatible and compatible changes.
                           Compatible changes will not trigger a failure exit
                           code.
  -o,--output ARG          The file path where the output should be written. If
                           the option is omitted the result will be written to
                           stdout.

                           The file extension is used to determine the type of
                           the output file.

                           Supports many formats such as markdown, html, rtf,
                           doc, txt, rst, and many more.

                           Leave out the extension to produce a self-contained
                           HTML report with styling.
  --folding-block-quotes-style
                           The report tree is structured using summary/detail
                           HTML elements and indented using block quotes. This
                           style renders well on GitHub.Intended for HTML output
                           format. Markdown has rendering bugs on GitHub.
  --header-style           The report tree is structured using increasing levels
                           of headers.
  --signal-exit-code       Signal API compatibility with the exit code.

                           Exit with 0 if there are no breaking changes.
                           Exit with 1 if there are breaking changes.
                           Exit with 2 if could not determine compatibility.

comparest's Issues

Monomorphic HLists in Subtree instances

Make all CLI options actually do something

Options like --only-errors and --silent are currently no-ops. Fix this.

Handle dynamic path fragments in non-breaking changes

https://typeable.slack.com/archives/C01N72YV29L/p1624459527017500

Traces should be comonadic

Currently we sometimes pass traces around in an environment monad, using monadic idioms, and sometimes we use Traced, which is effectively a comonad, using comonadic idioms. Experience showed that the latter is the more fruitful approach.

Report unsupported issues differently

Report basic breaking changes

Report the breaking changes for the following features of OpenAPI:

• add path (endpoint)
• remove path (endpoint)

Implement the base of the data family Crumbs approach

https://typeable.slack.com/archives/C01N72YV29L/p1615291524049800

Wider server parsing/comparison support.

Things like {a|b}c will not match ac.

Properly handle source error traces

Problems with the source spec are reported as semantic errors. This is not the best thing at least because the consumer can also have source spec errors.

Properly implement security overrides in operations

blocked by biocad/openapi3#14

Regex comparison

We could have complete support for comparing regexes (in e.g. the pattern field in json schema), if we compiled the regexes to automata and did set-theoretic operations on the automata (product with intersection, product with union, negation), and used an emptiness test for an automaton. These operations can be carried out somewhat efficiently. After talking a little bit with ekmett about this he suggested that we use Binary decision diagrams and in particular ZDD, for determinization (NFA -> DFA) and NFA membership test.

Implement Link Objects

Operation parameters are processed as unique by name

In actual fact, they are unique by (name, position)

https://github.com/typeable/openapi-diff/blob/7df8b856239388b820762e859501262a3428a892/src/OpenAPI/Checker/Validate/Operation.hs#L116

Implement security schemes

Support more than just JSON in payloads

Replace variable environment lists with fixed ones.

Check non-template Params when comparing path items

Don't use summary/detail in report if there is only one subelement

Report non-breaking changes

Compress multiple "enum added" into a single list

An error needs 2 traces

Currently our "check issue" only remembers one trace, either the producer or the consumer. However we need a way to refer to both, because in some places we "flip variance", and the resulting tree of errors has the producers and the consumers jumbled up. Otherwise we need to pass the variance to the leaves, so that they know which source to include in the error.

Better security scheme rendering

There are no identifiers for security schemes in openapi. Think how to identify them in reports

Tuple arrays (OpenApiItemsArray) are not supported

Currently JSON Schema field is not checked further is the "required" parameter changed

In general: if a field was considered "additional" (was "open", allowed any value), adding a definition in the next API should not report that, for example, "type Null was removed".

Complete the full tree compare (with no schemas)

Complete all path-related tree compare. Except Schema which is quite big.

Headers not recognised as a change

https://github.com/typeable/openapi-diff/tree/master/test/golden/common/pathItem/operation/responses/change/headers/optional

Place "more general" (not jetted) issues at the top

Extract spec errors in report

Investigate performance low-hanging fruit

Running on relatively small (70 KB) inputs yields 1-2 seconds runtimes which seems unreasonably large.

Find out if there are easy ways of reducing runtime.

Implement read-only and write-only properties

Process remote JSON $ref

Create test infrastructure

each test takes a triple of schema A, schema B, expectation

Simple CLI interface

Simple CLI interface to call the tool on two schemas

Using optparse-generic seems like a reasonable first step.

Splitting heuristic

If we have a conjunction of conditions in the producer, and a disjunction of conditions in the consumer, then we will test the conjunction against every item in the disjunction and see if anything succeeds. This is not always enough. We could try to heuristically split the producer's set along some e.g. enum component into a dijsunction, and then do the check for each of them (this process could be recursive).

Fix MIME types

Currently adding a type is considered a breaking change, and removing it is considered a non-breaking change.

Implement Callbacks

Callback have some non-trivial templating in the path keys that seems to not be implemented in openapi3

Support sub-PathFragment templates

The current implementation of path templates only supports the whole path fragment being a template. The OpenApi spec also allows parts of the path fragments to be templated.

Deprecations should be reported as non-breaking features

Issues with normalizePath

Currently normalizePath cannot function, because after pattern matching on Snoc we don't know anything about b other than it being Steppable.

The current proposed solution is to instead define:
normalizeStep :: Step b c -> (Trace OpenApi b -> Trace OpenApi c)
as a method of Steppable, in turn making Steppable non-polykinded. The intention is that when defining the Step b c as a data family instance, we would know for which constructors normalizeStep returns a simple Snoc, and for which we instead restart from the root and return a const ....

An alternative solution is to define a series of things:

-- a type family:
type family Prev :: Type -> [Type]
-- a number of constraint implications (via superclass constraints):
APIStep a b => Elem a (Prev b)
(Subtree b, Elem a (Prev b)) => (Subtree a, APIStep a b)
(Subtree b, Steppable a b) => Elem a (Prev b)

This would allow to pattern match on Trace a c, obtaining Trace a b and Step b c, and given Subtree c conclude Subtree b, thus allowing recursion.

Smart comparison of Minimum/Maximum/MultipleOf

Properly merge servers

https://github.com/typeable/openapi-diff/blob/b51d410a39553e3c948fecdec15616dff7b9b7b4/src/OpenAPI/Checker/Validate.hs#L42-L47

Proper memoization with preserving context

Add some contradiction checks

Currently if the producer schema indicates that no values of a particular type are allowed, we test the consumer formula for whether it is contradictory, i.e. whether it also allows no values.

The formula is in Disjuctive Normal Form, and every conjunct is checked for contradiction -- currently this always fails (but if there are no conjuncts the check succeeds).

Implement disjointness heuristic for oneOf

Until that is done, we always implicitly assume that oneOf clauses are disjoint. No unsupported warning is shown because that would imply a warning for every single instance of oneOf.

Implement server overrides

blocked by biocad/openapi3#17

Implement more test cases

pretty-print json

Add structural compatibility check

Implement dereferencing

including json path parsing

Don't register removing a field as both breaking and non-breaking change

See this test case: https://github.com/typeable/openapi-diff/tree/master/test/golden/common/property-removed