testingisdocumenting / webtau Goto Github PK

There are cases where we legitimately want to skip a test. For example, a particular bit of functionality is not yet available in certain environments. In JUnit, one would use assumeThat to achieve this, would be useful to have an equivalent in webtau.

One thing to be aware of with this is that the webtau CLI return code is count failed + error + skipped therefore assumeThat like "failures" cannot count towards the regular skipped count.

myMap.should == 'wrongvalue gives no op.
It is not an issue when assertion is done in the context of http.get|post. But if the same syntax is used outside that it just gives no errors.

Need to create Groovy AST to replace calls like myVar.should == with ActualValue.value('myVar', myVar).should(equal(..)).

Need to replace existing runtime way of doing it with exception "should not reach here" inside Should and ShouldNot classes

http.get(...) {
  header.MyValue.should == 'value'
}

should produce the same result as

http.get(...) {
  header.myValue.should == 'value'
}

The coverage is based on operations (i.e. path + method), not urls.

At the moment we capture test total run time. Http calls time. But we don't capture custom steps time.
Add steps execution time to reports.

Specifically, from this particular condition in OpenAPISpecValidator.validateApiSpec:

ApiOperationMatch apiOperationMatch = apiOperationResolver.findApiOperation(path, method);
if (!apiOperationMatch.isPathFound()) {
    return;
}

Swagger validation action needs to appear in report as a step.

Trying to use grape for dependency management but it doesn't seem to fetch the specified libraries

Currently, we order the tests alphabetically in the HTML report. Would be nice to see the failed errors at the top (ideally with a copyable details).

Right now if I run a script using IDEA run, I get a null pointer on scenario call.

Caught: java.lang.NullPointerException: Cannot invoke method scenario() on null object
java.lang.NullPointerException: Cannot invoke method scenario() on null object
	at com.twosigma.webtau.WebTauGroovyDsl.scenario(WebTauGroovyDsl.groovy:39)
	at com.twosigma.webtau.WebTauGroovyDsl$scenario.callStatic(Unknown Source)

There is no horizontal scrollbar or tooltip, one has to either expand the request/response or go to steps tab. Can we either provide a tooltip on hover or maybe wrap the url if the row is expanded?

localHost:tests karol$ ~/Dev/webtau/dist/webtau scenarios/*
       env: local                                   // from command line argument

Clearly, the comment is not correct as no env was specified on command line!

At the moment API can be used inside JUnit, but report will not be generated properly.
Need to create JUnit 5 extension to properly hook reporting.

We have a couple of auth provider implementations now which basically do the same thing when asked for a token:

• check local cache file and its validity (in terms of time expiry)
• if valid, great
• if not valid, fetch new token and cache it

Would be nice for webtau to have some sort of helper class/utils to make this easier and reduce the duplication.

Currently, if tests fail, the script returns with 0 which is not correct.

We need to include element selection path with assertion error If no custom name is specified for element.

Tests like HttpGroovyTest hardcode port number for testing. Need to replace with auto port discovery.

Asserting the size of a list response body is done with body.numberOfElements(). Accessing the elements is done via body.all(). The inconsistent naming is a little confusing.

It would also be quite useful to have find and maybe other similar operations on a list body.

If you use groovy console to play with API or to prototype tests none of the config values are loaded. So you have to manually set things like base url.
While base url setup through the console itself is quite trivial, setting authentication handlers may not be.

Need to detect that webtau is in groovy mode and lazy load config before first HTTP or UI command.

Right now we validate errors against the 2xx model which is not correct.

Webtau should expose at least some of the swagger configuration options described in https://bitbucket.org/atlassian/swagger-request-validator/src/04551f1ff4a7fa0be58ed72af226181f27df7faf/swagger-request-validator-core/?at=master

If a Swagger json is specified in config (local or http) then following an http call, validate the response to the model in Swagger. This will first require pattern matching on the URL to determine the right operation.

This could be an additional tab or another variation on the [] links.

I think non-existent env should lead to an error saying "requested env does not exist" or something like that. Right now, it'll try to run tests and invoke http calls without the base url so it'll do a GET /foo for example which predictably fails. This is quite confusing and not immediately obvious what's wrong.

Proposed syntax would be something along the lines of:

http.get('/something') {
    body.shouldBe validAgainstSchema('schema.json')
}

Things to consider:

• loading schema relative to current working dir
• config for location of specs

Currently something like this fails:

def actual = //something that returns a String[]
actual.should == ['foo', 'bar']

Error:

[value]:   actual: [Ljava.lang.String;@50313382 <java.lang.String[]>
         expected: [foo, bar] <java.util.ArrayList>

Use collected URL coverage data to provide

• % coverage information on Summary page
• list of covered and non covered operations

Integrate with swagger to figure out what URLs and operations are covered.
Collect this data in a way to be consumable inside HTML report.

Document maven plugin usage.
Need mdoc support for extracting portion of xml to avoid confusion around pom parent.

It would be helpful to either:

• print a summary of failed tests before the Total: z, Passed: y, blah line
• or, have a debug flag, without which we only print test name and passed/failed instead of all HTTP invocations, bodies, etc.

There are end-to-end webtau CLI tests that are not being ran automatically.
We need to automated them

Need to figure out where to push artifacts. Maven Central or github.
Need to have a script to deploy and bump up a version of webtau.

If an http call responds with a json list rather than an object this code passes when it should fail:

http.get('emptyResponseUrl') {
    nonExistentField.shouldNot == null
}

If I print nonExistentField, it looks something like this:

[[null node]@body[0].nonExistentField, [null node]@body[1].nonExistentField]

It looks like webtau is doing a projection of that field onto the list.

The primary usecase is to warn about URLs which were invoked but not part of the spec. Some other things we could include would be to add the model in either another tab in the report or inline in steps for example.

I would like to be able to do something like:

body.someDate.should <= LocalDate.now()

Or something similar.

Example:
Set aSet = myBean.getChildrenSet()
aSet.should == [ ... table data ...] // throws exception that Set cannot be converted to a List.
aSet.toList().should == [ ... table data ...] // works fine

This prevents sub collection matching if I want to verify properties of myBean, e.g.
myBean.should == [ propA: 'expectedA', childrenSet: expectedChildrenTableData ]

This will be useful for correlating failed integration tests to the logs, which will help drill down into root cause quickly.

Timestamps could be in ISO8601 format with the system's timezone e.g. 2008-09-15T15:53:00.123-04:00

• Under tests -> Summary, please add two cards. Labels: "Scenario Start" and "Scenario End" with the values as the timestamps.
• For each HTTP call under http calls -> individual call, please add a single card (currently just has "Latency (ms)"). Label: "Request Start" with the timestamp.
• Under tests -> HTTP calls, please include Request Timestamp as a column.

To simplify feature testing we added triggerConfigHandlers

        WebTauConfig.resetConfigHandlers()
        WebTauConfig.registerConfigHandlerAsFirstHandler(cliConfigHandler)
        WebTauConfig.registerConfigHandlerAsLastHandler(cliConfigHandler)

        // during end to end testing we re-create the instance of CliApp, but our config is singleton
        // we cannot rely on constructor actions
        //
        getCfg().triggerConfigHandlers()

this mess-up custom config handlers as they are being called twice.
Need to think how to enable feature testing with fewer consequences.

Currently we have success, failure, error and skipped. Would be useful to show a % success rate.

Default console output is Ansi console output.
When output gets redirected to a file, file has noise like

[36m{�[0m
  �[35m"_

This code:

def foo = body.find { something }

can lead to:

Caused by: java.lang.NullPointerException
    at java.util.Collections$UnmodifiableCollection.<init>(Collections.java:1026)
    at java.util.Collections$UnmodifiableList.<init>(Collections.java:1302)
    at java.util.Collections.unmodifiableList(Collections.java:1287)

If my http.get call (for example) specifies a header, it would be useful to include it in the report.

Currently, unless using the JUnit option, one has to write some maven exec code in the pom and in all likelihood also invoke a script to then unzip dist and run CLI.

The primary use case for this is if you fail to obtain creds in a header provider then in all likelihood no tests will pass therefore may as well abandon the test run straight away. Right now if an exception is thrown from the header provider webtau will still try all scenarios.

One option would be to have a special exception type, e.g. AbortTestExecutionException, that can be thrown and skips further scenario execution. There may be a better way though.

Enable Travis CI to perform regular builds for PR and pushes

Header values sometimes contain a long text without spaces.
Update css to do a hard break.