kajf / bozr Goto Github PK

Ability to provide values for placeholders in on.body, on.headers, etc.
Similar to [remember] section, but taking provided values (not from response body)

Could be useful to have an ability to generate some arguments on the fly. E.g. timestamp.

Does it make sense to cleanup report output dir before next run in scope of current app ?

Add option to filter tool output and show info only about failed tests.

Enhancement.
To create data independent test-scenarios we need to use arguments, but they are working only in borders of particular test.
Thus, we use a numerous calls in the one test, but they are not represented in the log-monitor of test running. So it makes complications with troubleshooting during the test-scenario creating. Sometimes it's rather difficult to find out in which call is a problem, if test is failed, .
We can use option -i, but in this case log-monitor contains a lot of redundant information.
That's why it would be great to add optional parameter 'Name' to the calls. It could be used only if needed. In this case log-monitor should represent not only running tests, but also the calls they consist.

Replace \n usage with \r\n to support windows.

<testsuite id="19" name="GET_Functional" package="v" timestamp="2017-06-06T08:21:52.037Z" time="0" hostname="localhost" tests="19" failures="0" errors="0" skipped="19"> <properties></properties> <testcase name="n" classname="GET_Functional" time="-9.223372036854776e+09"> <skipped message="Endpoint design is in progress. Issue ID: "></skipped> </testcase> <system-out></system-out> <system-err></system-err> </testsuite>

When value is a number test always fails.
"body": { "values.1.code": 999 }

Some APIs has call per second limitation that cannot be changed. For example, there are 6 bozr tests with 7 calls. Two tests fail with the response 503 "You have exceeded the API rate limit of 5 requests per second".

I would suggest to add "waiter" feature that could be set on test level and allows to specify delay before test execution in seconds.

Need a way of informing about invalid suits (e.g. create invalid test result)

response.json:

{
  "option": ["ONE", "TWO"]
}

matcher:

{
   "option.size()": 1,
   "~option": "ONE"
}

In scenario test it would be useful to have a possibility to specify reference example: {id} in the "expect" section.

Suggested syntaxis

1. To ignore test case without specifying a reason

{
  "name": "IgnoredTestCase",
  "ignore": true,
  "calls":  []
}

2. To ignore test case with specific reason

{
  "name": "IgnoredTestCase",
  "ignore": "In development",
  "calls":  []
}

Console output:

[SKIPPED] json_body - IgnoredTestCase	 (In development)

Bozr should not parse content of the body until the data is requested by the test.
E.g. if binary data is returned should be possible to check statusCode.

Currently only ./reports is available as report output folder. Need ability to specify another path

E.g.

...
"on": {
  "url": "url",
  "method": "POST",
  "body": {
      "test": "data"
  }
}
...
"on": {
  "url": "url",
  "method": "POST",
  "body": "<test><data/></test>"
}

Issue occurred during parallel execution.
Version 0.8.9

bozr -w 2 -H http://127.0.0.1:8080/ta ./suites

fatal error: concurrent map writes

goroutine 24 [running]:
runtime.throw(0x71bc28, 0x15)
C:/opt/Go/src/runtime/panic.go:596 +0x9c fp=0xc0421cf390 sp=0xc0421cf370
runtime.mapassign(0x6c1280, 0xc042064e10, 0xc0421cf570, 0x0)
C:/opt/Go/src/runtime/hashmap.go:499 +0x66e fp=0xc0421cf430 sp=0xc0421cf390
main.BodySchemaExpectation.checkJSON(0xc042156960, 0x5c, 0xc04216b140, 0x10, 0xc042260a01)
D:/_ws_go/src/github.com/kajf/bozr/expect.go:65 +0x6ba fp=0xc0421cf570 sp=0xc0421cf430
main.BodySchemaExpectation.check(0xc042156960, 0x5c, 0xc04216b140, 0x0, 0x0)
D:/_ws_go/src/github.com/kajf/bozr/expect.go:42 +0x18e fp=0xc0421cf5e8 sp=0xc0421cf570
main.(*BodySchemaExpectation).check(0xc042377150, 0xc04216b140, 0x0, 0x0)
:3 +0x67 fp=0xc0421cf628 sp=0xc0421cf5e8
main.call(0xc0421c724e, 0x17, 0xc0423ba840, 0xc04246b346, 0x3, 0xc0420df6a0, 0x1e, 0xc0423ba870, 0xc0423ba8a0, 0x0, ..

    D:/_ws_go/src/github.com/kajf/bozr/main.go:272 +0x9ae fp=0xc0421cfaa8 sp=0xc0421cf628

main.runSuite(0xc0421c7266, 0xe, 0xc0421c724e, 0x17, 0xc0420e6e40, 0x2, 0x4, 0xc042444300, 0x4, 0x4)
D:/_ws_go/src/github.com/kajf/bozr/main.go:176 +0x481 fp=0xc0421cfed8 sp=0xc0421cfaa8
main.runSuites(0xc0422e7320, 0xc0422c7f80, 0xc042120d80, 0x7297f8)
D:/_ws_go/src/github.com/kajf/bozr/parallel.go:45 +0x129 fp=0xc0421cffc0 sp=0xc0421cfed8
runtime.goexit()
C:/opt/Go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc0421cffc8 sp=0xc0421cffc0
created by main.RunParallel
D:/_ws_go/src/github.com/kajf/bozr/parallel.go:19 +0xd2

goroutine 1 [chan receive]:
main.RunParallel(0xc0422e7320, 0x85c680, 0xc042002620, 0x7297f8, 0x2)
D:/_ws_go/src/github.com/kajf/bozr/parallel.go:30 +0x157
main.main()
D:/_ws_go/src/github.com/kajf/bozr/main.go:151 +0x698

goroutine 23 [chan send]:
main.NewSuiteLoader.func1(0xc0422466c0, 0xc0422e7320)
D:/_ws_go/src/github.com/kajf/bozr/loader.go:148 +0xe9
created by main.NewSuiteLoader
D:/_ws_go/src/github.com/kajf/bozr/loader.go:152 +0xd9

goroutine 25 [select]:
net/http.(*persistConn).roundTrip(0xc0421a3200, 0xc042279b20, 0x0, 0x0, 0x0)
C:/opt/Go/src/net/http/transport.go:1898 +0x97b
net/http.(*Transport).RoundTrip(0x885160, 0xc0421bf400, 0x885160, 0x0, 0x0)
C:/opt/Go/src/net/http/transport.go:391 +0x753
net/http.send(0xc0421bf400, 0x858f40, 0x885160, 0x0, 0x0, 0x0, 0x0, 0x8, 0xc042170880, 0x40faaf)
C:/opt/Go/src/net/http/client.go:249 +0x169
net/http.(*Client).send(0xc042379890, 0xc0421bf400, 0x0, 0x0, 0x0, 0xc042170880, 0x0, 0x1, 0x0)
C:/opt/Go/src/net/http/client.go:173 +0x10f
net/http.(*Client).Do(0xc042379890, 0xc0421bf400, 0x0, 0x0, 0xc042449800)
C:/opt/Go/src/net/http/client.go:595 +0x25b
main.call(0xc0421c3a4e, 0x17, 0xc0424497d0, 0xc0422cc2d6, 0x3, 0xc0422b5120, 0x1e, 0xc042449800, 0xc042449830, 0x0, ..

    D:/_ws_go/src/github.com/kajf/bozr/main.go:241 +0x21b

main.runSuite(0xc0421c3a66, 0x10, 0xc0421c3a4e, 0x17, 0xc04216acc0, 0x3, 0x4, 0xc042372000, 0x7, 0x8)
D:/_ws_go/src/github.com/kajf/bozr/main.go:176 +0x481
main.runSuites(0xc0422e7320, 0xc0422c7f80, 0xc042120d80, 0x7297f8)
D:/_ws_go/src/github.com/kajf/bozr/parallel.go:45 +0x129
created by main.RunParallel
D:/_ws_go/src/github.com/kajf/bozr/parallel.go:19 +0xd2

goroutine 26 [semacquire]:
sync.runtime_Semacquire(0xc042120d8c)
C:/opt/Go/src/runtime/sema.go:47 +0x3b
sync.(*WaitGroup).Wait(0xc042120d80)
C:/opt/Go/src/sync/waitgroup.go:131 +0x81
main.RunParallel.func1(0xc042120d80, 0xc0422c7f80)
D:/_ws_go/src/github.com/kajf/bozr/parallel.go:25 +0x32
created by main.RunParallel
D:/_ws_go/src/github.com/kajf/bozr/parallel.go:27 +0x118

goroutine 66 [IO wait]:
net.runtime_pollWait(0x22fd770, 0x72, 0x857348)
C:/opt/Go/src/runtime/netpoll.go:164 +0x60
net.(*pollDesc).wait(0xc0423526e0, 0x72, 0x857348, 0x0)
C:/opt/Go/src/net/fd_poll_runtime.go:75 +0x3f
net.(*ioSrv).ExecIO(0xc042052038, 0xc0423525a0, 0x716c90, 0x7, 0x729830, 0x2, 0x1, 0x1)
C:/opt/Go/src/net/fd_windows.go:196 +0xfe
net.(*netFD).Read(0xc042352540, 0xc0421a4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
C:/opt/Go/src/net/fd_windows.go:452 +0x13d
net.(*conn).Read(0xc042004ea8, 0xc0421a4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
C:/opt/Go/src/net/net.go:181 +0x77
net/http.(*persistConn).Read(0xc0421a3200, 0xc0421a4000, 0x1000, 0x1000, 0x4, 0x44f1b0, 0xc0422c6ea0)
C:/opt/Go/src/net/http/transport.go:1316 +0x152
bufio.(*Reader).fill(0xc0422c6d20)
C:/opt/Go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).Peek(0xc0422c6d20, 0x1, 0xc042451be5, 0x1, 0x0, 0xc042156a80, 0x0)
C:/opt/Go/src/bufio/bufio.go:129 +0x6e
net/http.(*persistConn).readLoop(0xc0421a3200)
C:/opt/Go/src/net/http/transport.go:1474 +0x19d
created by net/http.(*Transport).dialConn
C:/opt/Go/src/net/http/transport.go:1117 +0xa3c

goroutine 9 [select]:
net/http.(*persistConn).writeLoop(0xc0424299e0)
C:/opt/Go/src/net/http/transport.go:1704 +0x441
created by net/http.(*Transport).dialConn
C:/opt/Go/src/net/http/transport.go:1118 +0xa61

goroutine 8 [IO wait]:
net.runtime_pollWait(0x22fd830, 0x72, 0x857348)
C:/opt/Go/src/runtime/netpoll.go:164 +0x60
net.(*pollDesc).wait(0xc0422b16a0, 0x72, 0x857348, 0x0)
C:/opt/Go/src/net/fd_poll_runtime.go:75 +0x3f
net.(*ioSrv).ExecIO(0xc042052038, 0xc0422b1560, 0x716c90, 0x7, 0x729830, 0x1, 0x0, 0x0)
C:/opt/Go/src/net/fd_windows.go:196 +0xfe
net.(*netFD).Read(0xc0422b1500, 0xc0420cb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
C:/opt/Go/src/net/fd_windows.go:452 +0x13d
net.(*conn).Read(0xc042340210, 0xc0420cb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
C:/opt/Go/src/net/net.go:181 +0x77
net/http.(*persistConn).Read(0xc0424299e0, 0xc0420cb000, 0x1000, 0x1000, 0x4, 0x44f1b0, 0xc042423440)
C:/opt/Go/src/net/http/transport.go:1316 +0x152
bufio.(*Reader).fill(0xc0422a5320)
C:/opt/Go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).Peek(0xc0422a5320, 0x1, 0xc042029be5, 0x1, 0x0, 0xc042287f80, 0x0)
C:/opt/Go/src/bufio/bufio.go:129 +0x6e
net/http.(*persistConn).readLoop(0xc0424299e0)
C:/opt/Go/src/net/http/transport.go:1474 +0x19d
created by net/http.(*Transport).dialConn
C:/opt/Go/src/net/http/transport.go:1117 +0xa3c

goroutine 67 [select]:
net/http.(*persistConn).writeLoop(0xc0421a3200)
C:/opt/Go/src/net/http/transport.go:1704 +0x441
created by net/http.(*Transport).dialConn
C:/opt/Go/src/net/http/transport.go:1118 +0xa61

Add reporter that sends results to "Report Portal"
https://github.com/reportportal

If the response body is an array:
[{"el" : 1}, {"el":2}]
The following failure is shown:
Can't parse response body to Map. json: cannot unmarshal array into Go value of type map[string]interface {}

Currently only response info is provided for failed test. Need to have ability to see the details about request that was send.

=> impossible to use to build a link to another resource

Example of incorrect fmt:

Expected value [%!s(bool=false)] on path [some/path] does not match.

Critical for [expect] section since only one expectation will checked in case below

        "expect": {
          "body": {
            "~errors.key": "0_value",
            "~errors.key": "1_value",
            "~errors.key": "2_value"
           }
        }

Workaround:

        "expect": {
          "body": {
            "~errors.key": ["0_value", "~errors.key": "1_value", "2_value"]
           }
        }

Need to somehow report on invalid expectations (do not skip silently).
Maybe on json schema level

Current
[PASSED] security - OtherNotASubordinate 308.0244ms

Suggested
[PASSED] ui.employees.id.pto.GET. security - OtherNotASubordinate 308.0244ms

Remove debugMsg() function as in the calling frames debug messages is always thrown from main.go

Validate host argument. If it's invalid - throw error and don't even start tests.

provide ability to specify non-exact matchers like

"expect": {
    "absent": ['~user.cards.cardNumber']
}

E.g.:

body: {
  "!data.path.id": 123
}

body: {
   "expected": {
      "some.path": "expected.value"
   }
   "not": {
      "another.path": "shouldnt.be"
   }
}

The following test case part throws exception
"args": {
"key": 1
}

panic: interface conversion: interface is float64, not string

goroutine 1 [running]:
panic(0x8286c0, 0xc082227300)
C:/opt/Go/src/runtime/panic.go:481 +0x3f4
main.populateRememberedVars(0xc08221e480, 0x17d, 0xc082233620, 0x0, 0x0)
C:/_ws_go/src/github.com/pavel-letsiaha/t-rest/main.go:243 +0x197
main.populateRequest(0xc0821cbe26, 0x3, 0xc0821d21c0, 0x19, 0xc0821b3860, 0
C:/_ws_go/src/github.com/pavel-letsiaha/t-rest/main.go:213 +0x205
main.call(0xc082198104, 0x9, 0x8a6ef8, 0x1, 0xc08205b680, 0xc, 0xd, 0xc0821
C:/_ws_go/src/github.com/pavel-letsiaha/t-rest/main.go:148 +0x409
main.main()
C:/_ws_go/src/github.com/pavel-letsiaha/t-rest/main.go:109 +0xcf9

Workaround: pass arg value as a string

"args": {
"key": "1"
}