[Bug] `clarinet test` and `clarinet check` exits with 0 as exit code when tested contracts are invalid about clarinet

[lgalabru]

I think that an accurate version to test this would be with something like this:

$ clarinet test
$ echo $?

[whoabuddy]

Here is an example where we included clarinet check as a step before clarinet test and both succeeded when they shouldn't have:

https://github.com/citycoins/citycoin/runs/2786873970?check_suite_focus=true

[LNow]

Unfortunately #34 is not fixed.

Executing following commands should results with upss printed to the screen when we test invalid contract, yet in both cases all we see is output from clarinet.

$ clarinet check || echo 'upss'
$ clarinet test || echo 'upss'

I'm using last version available in crates.io.

[lnow@localhost citycoin]$ cargo install clarinet
    Updating crates.io index
     Ignored package `clarinet v0.10.0` is already installed, use --force to override

[whoabuddy]

Trying that locally leads to the same result, exit code is 0.

$ clarinet check
Analysis error: use of unresolved variable 'this-will-cause-an-error'
;; CityCoins on Stacks, simulating PoX in a smart contract
this-will-cause-an-error
;; error codes
(define-constant ERR-NO-WINNER u0)
(define-constant ERR-NO-SUCH-MINER u1)
(define-constant ERR-IMMATURE-TOKEN-REWARD u2)
(define-constant ERR-UNAUTHORIZED u3)
...(output truncated)...
$ echo $?
0

The same goes for clarinet test:

$ clarinet test
Analysis error: use of unresolved variable 'this-will-cause-an-error'
;; CityCoins on Stacks, simulating PoX in a smart contract
this-will-cause-an-error
;; error codes
(define-constant ERR-NO-WINNER u0)
(define-constant ERR-NO-SUCH-MINER u1)
(define-constant ERR-IMMATURE-TOKEN-REWARD u2)
(define-constant ERR-UNAUTHORIZED u3)
...(output truncated)...
$ echo $?
0

Now if I change the contract code so that its valid, but instead fail a test in our sequence, the exit code is updated as expected.

$ clarinet test
running 100 tests
test [CityCoin] SIP-010: transfer() succeeds with no memo supplied ... FAILED (440ms)

failures:

[CityCoin] SIP-010: transfer() succeeds with no memo supplied
Error: Expected ok, got (err u1)
    at consume (https://deno.land/x/[email protected]/index.ts:366:11)
    at String.expectOk (https://deno.land/x/[email protected]/index.ts:379:10)
    at file:///home/whoabuddy/citycoin/tests/citycoin_test.ts:85:34
    at fn (https://deno.land/x/[email protected]/test_suite.ts:457:17)
    at async Object.exitSanitizer [as fn] (deno:deno_runtime-0.9.3/js/40_testing.js:101:9)
    at async TestRunner.[Symbol.asyncIterator] (deno:deno_runtime-0.9.3/js/40_testing.js:275:13)
    at async Object.runTests (deno:deno_runtime-0.9.3/js/40_testing.js:352:22)
    at async file:///home/whoabuddy/citycoin/$deno$test.ts:3:1

failures:

        [CityCoin] SIP-010: transfer() succeeds with no memo supplied

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out (440ms)
$ echo $?
1

[lgalabru]

Should be addressed with latest, can you close this issue once confirmed on your end? Thanks!

[lgalabru]

(will update here once the docker image has been updated)

[whoabuddy]

I'm still seeing this issue with version clarinet v0.10.0, where invalid code doesn't change the exit code for clarinet check or clarinet test.

[whoabuddy]

Confirming this is working correctly in clarinet 0.13.0, for both clarinet check and clarinet test - thank you!

[lgalabru]

Thanks @whoabuddy!