Regression: How to fix this
Tools for validating OpenAPI (Swagger) files.
- node.js version > 10.x
You can install the latest stable release of node.js from here. For a machine with a linux flavored OS, please follow the node.js installation instructions over here
npm install -g oav
bash-3.2$ oav -h
Commands:
extract-xmsexamples <spec-path> Extracts the x-ms-examples for a
<recordings> given swagger from the .NET session
recordings and saves them in a file.
generate-uml <spec-path> Generates a class diagram of the
model definitions in the given
swagger spec.
generate-wireformat <spec-path> Transforms the x-ms-examples for a
given operation into raw
request/response format and saves
them in a markdown file.
resolve-spec <spec-path> Resolves the swagger spec based on
the selected options like allOfs,
relativePaths, examples etc.
validate-example <spec-path> Performs validation of x-ms-examples
and examples present in the spec.
validate-spec <spec-path> Performs semantic validation of the
spec.
Options:
--version Show version number [boolean]
-l, --logLevel Set the logging level for console.
[choices: "off", "json", "error", "warn", "info", "verbose", "debug", "silly"]
[default: "warn"]
-f, --logFilepath Set the log file path. It must be an absolute filepath. By
default the logs will stored in a timestamp based log file
at "C:\Users\amzavery\oav_output".
-h, --help Show help [boolean]
bash-3.2$
Model validation checks whether definitions for request parameters and responses, match an expected input/output payload of the service.
Examples of issues:
- required properties not sent in requests or responses;
- defined types not matching the value provided in the payload;
- constraints on properties not met; enumeration values that don’t match the value used by the service.
References: Azure/azure-rest-api-specs#778 , Azure/azure-rest-api-specs#755 , Azure/azure-rest-api-specs#773
Model validation requires example payloads (request/response) of the service, so the data can be matched with the defined models. See x-ms-examples extension on how to specify the examples/payloads. Swagger “examples” is also supported and data included there is validated as well. To get the most benefit from this tool, make sure to have the simplest and most complex examples possible as part of x-ms-examples.
- Please take a look at the redis-cache swagger spec as an example for providing "x-ms-examples" over here.
- The examples need to be provided in a separate file in the examples directory under the api-version directory
azure-rest-api-specs/arm-<yourService>/<api-version>/examples/<exampleName>.json
. You can take a look over here for the structure of examples. - We require you to provide us a minimum (just required properties/parameters of the request/response) and a maximum (full blown) example. Feel free to provide more examples as deemed necessary.
- We have provided schemas for examples to be provided in the examples directory. It can be found over here. This will help you with intellisene and validation.
- If you are using vscode to edit your swaggers in the azure-rest-api-specs repo then everything should work out of the box as the schemas have been added in the
.vscode/settings.json
file over here. - If you are using Visual Studio then you can use the urls provided in the settings.json file and put them in the drop down list at the top of a json file when the file is opened in VS.
- If you are using vscode to edit your swaggers in the azure-rest-api-specs repo then everything should work out of the box as the schemas have been added in the
Swagger specs validation could be split in the following:
- Schema validation
- Semantic validation
- Model definition validation
- Swagger operations execution (against mocked data or live tests)
- Human-eye review to complement the above
In the context of “azure-rest-api-specs” repo:
- #1 is being performed on every PR as part of CI.
- #2 and #3 are performed by the tool currently in openapi-validation-tools repo and by AutoRest linter. We’re working towards integrating them into CI for “azure-rest-api-specs” repo.
- #4 is not available yet, though we’re starting to work on it.
- #5 will be done by the approvers of PRs in “azure-rest-api-specs”, as this won’t be automated.
- Please don't edit this section unless you're re-configuring how oav plugs in to AutoRest AutoRest needs the below config to pick this up as a plug-in - see Autorest extensions.
pipeline:
swagger-document/model-validator:
input: swagger-document/identity
- A Live Validation mode has been added to OAV with the purpose of enabling validation of live traffic.
- Usage (here is a sample of a request-response pair):
const liveValidatorOptions = {
git: {
url: "https://github.com/Azure/azure-rest-api-specs.git",
shouldClone: true
},
directory: path.resolve(os.homedir(), "cloneRepo"),
swaggerPathsPattern: "/specification/**/resource-manager/**/*.json",
isPathCaseSensitive: false,
shouldModelImplicitDefaultResponse: true
}
const apiValidator = new oav.LiveValidator(liveValidatorOptions)
await apiValidator.initialize() // Note that for a large number of specs this can take some time.
// After `initialize()` finishes we are ready to validate
const validationResult = apiValidator.validateLiveRequestResponse(requestResponsePair)
Output of the OAV tool has been snapshotted and committed to the repo. The regression test may be run on a sample or all of https://github.com/azure/azure-rest-api-specs. If there are changes to the snapshots the build produces a git patch file as an artifact which may be used to update the snapshots.
Fast Regression (~10mins) is used for merge validation
Slow Regression (~1 hour) is run after merge and should be fixed if it fails
- Go to the failed build
- Download the artifact patch file
- In the OAV directory run
git apply <path to patch file>
- Commit the patched changes and create a pull request
- Validate that the changes look ok and don't represent a breaking change in OAV
- Merge the PR
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.