How to show error states? about json-path-comparison

[cburgmer]

Agree, I want to be able to align on illegal selectors/intentional errors as well. The best idea so far I have is to detect a SyntaxError that some of the implementations returns (but Goessner's don't) and handle it differently from a general error in the evaluation step.

I'm also contemplating opening up the comparison to allow for more heterogenous responses, and to e.g. equate null and NotFound errors. I'll try to lay out this idea in a separate issue.

As for the "Rust: returns" I cannot reproduce what the response was at the time of writing, and even which of the two implementations I was referring to.

[glyn]

I think by now I've captured all implementations where I feel comfortably deduce "NOT_SUPPORTED". For some I had to reach into the internal implementation, something I might raise as feature requests upstream to have properly supported.

+1

I now have a local change to detect such a consensus. The problem right now is that so far a consensus was assumed to only exists for OK responses, and as such there are a few quirks that need ironing out. I might commit something smaller to get started with.

Sounds good. But let's keep this issue (or another if you prefer) open until we reach the desired goal of showing consensus on "NOT_SUPPORTED".

[remorhaz]

I think it's too language-dependent. Some languages just don't have exceptions at all, others have well-defined error-handling policy. You must just define the way to differentiate between "error" and "no error" states for each implementation. And it will be exceptionally good to differentiate between "query syntax error" and "error getting result" where possible.

[cburgmer]

Currently this issue would not show up for array_index_out_of_bounds because there is no consensus for []. If there was we would have to ask what libraries with a scalar response type should respond. Kotlin returns null. Is that equivalent?

In the code this shows as

$ ./src/unwrap_scalar.py <<< '[]'
Traceback (most recent call last):
  File "./src/unwrap_scalar.py", line 20, in <module>
    sys.exit(main())
  File "./src/unwrap_scalar.py", line 14, in main
    assert len(j) == 1
AssertionError

[cburgmer]

Clojure and Kotlin return null for $.key on {"key": null}, so equating null to no result found seems wrong. Especially as most other implementations return [null] as a result.

[glyn]

I think errors are sometimes legitimate (e.g. I personally think selector $.['key'] is invalid and this seems to be the consensus), but there is no record of consensus on an error (ignoring any error text, which is bound to vary).

One consequence of this is that when testing against regression_suite.yaml, all I can check when there isn't a consensus is to make sure the implementation doesn't panic (in Golang).

(BTW the opening comment said "Rust: returns null", but I'm not sure what you mean as there is no null in Rust.)

[cburgmer]

For #37 I've started adapting the implementations to return a more specific error. So far I've identified NOT_FOUND (for implementations that prefer errors over empty responses) and NOT_SUPPORTED (for implementations that can separate selector parsing errors over evaluation errors).
This now needs to be implemented in all of the 36 implementations to have a future impact on a potential consensus.
Not all implementations seem to be explicit about the errors, and might need issues raised with upstream to clarify.

[glyn]

This now needs to be implemented in all of the 36 implementations to have a future impact on a potential consensus.

If I counted correctly, 22 of the 36 implementations can already return NOT_SUPPORTED. This is well over the "half rounded up plus one" value used for consensus. So why wouldn't this be sufficient for there to be a consensus of NOT_SUPPORTED in some cases?

[cburgmer]

I think by now I've captured all implementations where I feel comfortably deduce "NOT_SUPPORTED". For some I had to reach into the internal implementation, something I might raise as feature requests upstream to have properly supported.

I now have a local change to detect such a consensus. The problem right now is that so far a consensus was assumed to only exists for OK responses, and as such there are a few quirks that need ironing out. I might commit something smaller to get started with.

[cburgmer]

I've pushed more changes to implement #37. (I'm tracking the work there, as the solution is more generic than what we want for this issue here.)

Following these changes we can now call out a consensus on NOT_SUPPORTED and NOT_FOUND. There's a bit more work needed to document and rework some of the bits for readability.

One thing that's now becoming more visible is that the remaining errors are now mostly real defects in the respective implementations and should probably be reported upstream.

Exceptions to the rule are currently

• Golang_github.com-oliveagle-jsonpath which doesn't allow us to detect syntax errors uniformly across all the different errors returned.
• Elixir_jaxon needs some rework in the adapter here, as the native C bindings don't work well with returning non-standard errorlevels to the unix shell.

The following implementations don't handle "not supported" either at all, or just not very well:

• Bash_JSONPath.sh
• Clojure_json-path
• Ruby_jsonpath
• Perl_JSON-Path

[cburgmer]

Closing, as we have improved this massively over time.