grossamos / rudra

A multilingual integration test coverage analysis tool

Rudra

Rudra is an openapi based test coverage analysis tool. It allows teams to set and enforce coverage levels for integration tests in CI/CD-pipelines.

Contributions to the project are allways welcome, please consult CONTIBUTING.md for more information.

Quickstart

Step 1: Point your integration tests to rudra

First point your integration tests to the Rudra reverse proxy. This ensures that Rudra can analyze an interpret requests happening during your integration tests.

Point your tests towards http://localhost:13750 (without Docker) or http://rudra:13750 (with Docker). When using Docker Rudra will automatically configure networking for your container to connect to it.

Step 2: Add a preperation step

Configure Rudra in a preperation Stage. This will set up Rudra for later use. Place this preperation stage after you've started your service and before you'll run your integration tests.

Remeber to replace the location of your OpenAPI spec and the instance URL. The OpenAPI Spec can also be provided via a link.

  - name: init rudra
    uses: grossamos/[email protected]
    with:
      stage: "preperation"
      openapi-source: "docs/swagger.json"
      instance-url: "http://localhost:8080"
      test-coverage: "75%"

Modify openapi-source to point to your openapi/swagger specification. This can also be a url.

Modify instance-url to point to the base of your service (everything before the basepath of your openapi spec).

Optionally set a desired test-coverage for your endpoints.

Step 3: Add evaluation step

Place the rudra evaluation stage somewhere after your integration tests have run.

  - uses: grossamos/[email protected]
    name: eval rudra
    with:
      stage: "evaluation"

This stage will fail if test coverage isn't met and can display additional information gathered during the integration tests.

Overview

Rudra works by acting as a reverse proxy between your application and integration tests. It collects and compares the requests (and responses) with an openapi spec.

The reverse proxy is set up an configured in the first "preperation" stage. Analysis and any propagation of results occurs during the "evaluation" stage.

Configuration options

Option	Description	Values	Examples
account-for-security-forbidden	Take security annotations into account and require 403 cases to be handled (default false)	boolean	true
account-for-security-unautorized	Take security annotations into account and require 401 cases to be handled (default false)	boolean	true
debug	Enables Debug mode (default false)	boolean	true
instance-url	Base of service, excluding basepath from openapi	URL	http://localhost:8080
only-account-for-pr	Indictates if only changes within a PR should be taken into account, doesn't take effekt outside a PR (default: false)	boolean	true
openapi-source	Location of openapi/swagger spec	Path or URL	docs/swagger.yaml
port	Port for rudra to listen on (default 13750)	unsigned 16 bit integer	13750
services	Configuartion for multiple services, conflicts with port, openapi-source, instance-url	instance-url; openapi-source; port;\n	see here
stage	Specifies which stage to use	preperation, evaluation	preperation
test-coverage	Coverage to enforce in evaluation stage (default 70%)	Percentage or float	0.75, 75%
groupings	Allows for certain configruations to be grouped together or ignored	path; method; status_code; ignored;\n	see here

Examples

Rudra Example

A reference pipeline can be point under https://github.com/grossamos/rudra-example. It uses a go service and postman to serve as an example of how to integrate rudra into your application.

The pipeline in rudra-example is structured as follows:

  - uses: grossamos/[email protected]
    name: init rudra
    with:
      stage: "preperation"
      openapi-source: "docs/swagger.json"
      instance-url: "http://localhost:8080"
      account-for-security-forbidden: true
      test-coverage: "90%"
      only-account-for-pr: true
# ... Integration tests ...
  - uses: grossamos/[email protected]
    name: eval rudra
    with:
      stage: "evaluation"

Multiple services

A configuration with multiple endpoints and openapi specifications could look as follows:

services: |
    http://localhost:8080; docs/swagger1.yaml; 13751;
    http://localhost:8443; docs/swagger2.yaml; 13752;

Groupings

Somtimes endpoints reuse the same logic and shouldn't need to be tested twice. Other times some configurations simply can't get tested and need to be ignored from a perspective of test coverage. The groupings feature allows you to define groups. A group only requires a single test to make all endpoints count as tested. If the ignored flag is set, they are assumed to be tested and are taken out of consideration.

An example could look as follows:

groupings: |
    /foo/bar; GET; 200; true;
    /foo/{bar}/moo; GET, POST; 200, 418; false;

rudra's Issues

Documentation: Add guide for Go OpenAPI spec generation

Add a guide for automatic OpenAPI spec generation to website.
The guide can be added to website/content and should explain how an OpenAPI spec can be generated for later use with rudra.
It doesn't directly document how to use rudra, but the guide will help users adopt rudra.

An example for how specs can be generated can be found under rudra-example

Documentation: Create FAQ

Create an FAQ for the website.

An outline for the FAQ already exists. Goal would be to provide answers to the questions posed there.

Documentation: Add Quarkus guide on how to generate an OpenAPI spec

Add a guide for automatic OpenAPI spec generation to website.
The guide can be added to website/content and should explain how an OpenAPI spec can be generated for later use with rudra.
It doesn't directly document how to use rudra, but the guide will help users adopt rudra.

Quarkus (using JAX-RS bindings) should be able to make use of swagger-core.

Evaluator: Configurations missed in openapi spec flaged as errors

Current behavior:

Missed configurations is a diff between an openapi spec and nginx logs. When running into configurations not covered by the openapi spec, this causes an error.
Ex. after having proxy issues (and resolving them) the following result is returned:

Expected behaviour:

No error and possibly returning the lacking endpoints in a warning message (that can be turned into an error with a configuration option)

Evaluator: Trailing Slash in Paths not ignored

Current behavior:

Trailing slashes are potentially not ignored.
Ex. /user/foo and /user/foo/ should be treated as equal but are not

Expected behavior

Ignore trailing slashes in evaluation

Bug: Path Variables not resolved

Current Behavior

Rudra correctly identifies paths, but not path variables.
Ex. the following path isn't being handled: /foo/{bar}/moo

Expected behaviour

/foo/{bar}/moo gets recognised as /foo/1/moo and all other forms

Documentation: Host Website

Create a hosted instance of our website.

• Dockerise Website (existing template can be used)
• Create initial CI pipeline that updates after changes to website folder
• Point get.rudra.cloud domain to instance

Networking: Bad Gateway should lead to explicit pipeline failure

Current behavior:

If an endpoint isn't available, nginx responds with its default 502 response.

Expected behavior:

Rudra makes it explicity, that tests are failing, because the host cannot be resolved.
Similar issues with timeouts, etc. should also be taken into account.

Configuration: Group Generic Functionality

Current behavior

When implementing generic functionality for multiple endpoints (ex. a middleware to handle auth) rudra requires every usage to be documented.

Expected behavior

We should only be required to test these functions once.
A configuration option combining path regex, status should be added to be able to group endpoints.
Users should then be able to indicate if they would like these to be excluded entierly or to only require them to be tested once.