Validate references about kaizen-openapi-editor

[ghillairet]

I started to work on this issue, the validator will return warnings for the links ($ref values) that are invalid or point to no files.

Here's a demo of what it currently looks like

There is still some clean up to do as I will try to reuse some code from the JsonReferenceHyperlinkDetector as their logic is quite similar.

[tedepstein]

Note that these can be external references, pointing to other files or definitions residing in other files. The validator should work with those definitions as well.

[tedepstein]

@tfesenko, Marking this one as high priority, as it will empower users to quickly resolve a significant number of issues without in-depth diagnosis.

@andylowry , note that this may be preferable to fixing ZEN-2633

[tedepstein]

@ghillairet , @tfesenko , we have an important project called Swagger-Nomalizer that resolves and inlines Swagger references (among other things).

It's important that we keep reference resolution consistent across Swagger Normalizer, hyperlinking ("go to definition"), and reference validation.

Swagger-Normalizer also caches external files, for efficiency. Maybe reference validation can make use of this cache.

I wish I had thought of this before. @andylowry , can we move Swagger-Normalizer into our open source RepreZen organization on GitHub? And do you have any suggestions to save Guillaume some time in understanding how Swagger-Normalizer resolves references?

I don't think transitive reference resolution (and the contextual interpretation issues around that) are important here. SwagEdit only deals with one context at a time; i.e., the file loaded in the editor.

[andylowry]

@tedepstein wrote: @andylowry , can we move Swagger-Normalizer into our open source RepreZen organization on GitHub? And do you have any suggestions to save Guillaume some time in understanding how Swagger-Normalizer resolves references?

I could, but I'm not sure how much value it will bring, especially since resolution in the normalizer is not done correctly at the moment. Very little of the normalizer code actually does reference validation or resolution, and the rules are pretty simple:

1. References have a URL part and a pointer part, the latter being in the place normally viewed as the URI fragment (i.e. after the first '#").
2. The URL should be resolved in the context of the URL used to retrieve the containing document (e.g. a file:// URL for a workspace file, but could be an http: URL or other for files obtained from outside the workspace, assuming those are supported).
3. The pointer must start with a slash, and its components are separated by slashes.
4. Components that look like non-negative integers index array values; others select properties from object values
5. A slash within a pointer component is escaped as "~1" - and because tilde has this special meaning, a tilde within a pointer component is escaped as "~0"
6. No other characters should be escaped in the pointer, and %-escape sequencing is not used
7. Local references (i.e. pure JSON Pointers - no URL part) are always resolved within the containing file.

Paths are the biggest pain in the neck in this whole referencing scheme. To be absolutely consistent with the JSONRef spec, the reference value itself must conform with URI syntax. But URI syntax specifically excludes many punctuation chars, including the braces that occur regularly in swagger paths - and hence as the names of properties in the paths object. To properly reference such a path value you'd need a reference looking something like somefile.yaml#/paths/~1foo~1%7Bid%7D for the path /foo/{id}! The escaped slashes are bad enough, and we really can't live without them. But the escaped braces are just horrible, hence my approach of treating the pointer separately from the URL. This means that certain reference strings that we will accept - like somefile.yaml#/paths/~1foo~1{id} - are not really valid JSONRefs at all. So code that starts validating a ref by using the Java classes to parse the whole ref as a URI will fail on these. That may actually be more technically correct, but it's probably untenable.

The alternative that we're adopting in our recommended multi-file strategy is to use simple names for paths separated out into subordinate files - e.g. use fooWithId as the property name for the path object whose actual path is /foo/{id}, so the reference ends up looking like somefile.yaml#/paths/fooWithId - nice and simple. This has the downside of making the subordinate swagger file invalid, since the path property name does not contain a template for the id path parameter appearing in the path definition.

So I think in summary, the following two not-quite-kosher compromises are required by our approach:

1. Allow non-escaped punctuation in reference pointers, which means that the reference string as a whole may not actually be a syntactically valid JSONRef
2. Allow a swagger spec to contain paths properties that do not contain the templates normally required for the path parameters defined for the path.

Not pretty - not in the least.

[andylowry]

OK, so reading what I just wrote, my statement that the rules are fairly simple seems absurd. But I stand by it. The reasoning behind the compromises and how they affect the rules are complicated, but the rules themselves are fairly straightfoward, and once the URL and pointer are split apart, standard libraries support them well (e.g. using the Java URL class to parse and resolve the URL, and using the Jackson JsonNode#at methods to handle pointers).

[tedepstein]

@andylowry wrote:

I could, but I'm not sure how much value it will bring, especially since resolution in the normalizer is not done correctly at the moment. Very little of the normalizer code actually does reference validation or resolution, and the rules are pretty simple:

Actually, given all of what you just explained, I think this is quite complex. And unless we're careful, it seems quite likely that we will have inconsistencies between SwagEdit's reference validatation, Swagger-Normalizer's reference resolution.

This can lead to situations where the SwagEdit allows something that doesn't parse and display correctly in the live views. It's better for SwagEdit to err the conservative side, being more restrictive than Swagger-Normalizer. But it's best not to err at all, and have both libraries share reference resolution code.

[tedepstein]

@ghillairet , I just talked about this with Andy. We agreed that, while there's not a lot of code involved, this is still complex enough cause inconsistencies across components. So we suggest that you extract this logic from the SwaggerPreResolver class in Swagger-Normalizer into a utility class. When Andy gets back to working on Swagger-Normalizer, he will try to use the utility class, so you're both using the same logic.

We also think it would be helpful if you took Andy's explanation above, and incorporated that as JavaDoc.

[ghillairet]

There are certainly some parts that can be extracted from SwaggerPreResolver and share with SwagEdit, for example I agree that the logic for validation of references should be the same.

Although SwaggerPreResolver does a lot more than that, for example I see that it does fix references before resolving them. What is the reason behind it, do you allow users to write invalid references (for ex allow white spaces, unescaped characters, etc.) and convert them into valid ones?

What we could do is extract the validation logic in some class, SwagEdit does not really need more than that. It only needs to know that a reference can be open, and which file does it point to.

To give you an idea, this is how the validation of references is currently done in SwagEdit. It does not involve a complex logic:

1. Local references
   What is consider a local reference starts with #. Meaning that what is behind should be a valid JSON pointer.
   The reference will be consider valid only if a JsonNode can be found in the current document with that pointer.
2. External references
   The reference does not start with a #. Meaning that it is a path to a file with optionally a fragment being a JSON pointer.
   The first step is to deresolve the path against the URL of the current document, if the path is not absolute. This gives us a valid URL to a file being in the workspace or not.
   The reference will be consider valid only if the URL is well formed and the it's content can be accessed. If a pointer is present, the reference is only valid if a JsonNode can be found with that pointer in the URL content.

[andylowry]

Hi Guillaume.

The part about "fixing" references applies to so-called Simple Refs. That's e.g. where you see something like $ref: Pet in old versions of PetStore. Fixing that reference turns it into $ref: "#/definitions/Pet". These simple refs are not really valid in Swagger 2.0 but they're a vestige from earlier swagger. We're not as wild as the old swagger parser was (where even something like foo.yaml#/definitions/blah was treated as equivalent to #/definitions/foo.yaml#/defintions/blah (!). Ted made the decision that for now, a ref that consists solely of letters, digets, hyphen and underscore is treated as a simple ref and "fixed" as needed.

This is a part that I forgot about up above, and it would be good to have that baked into our common library for as long as we retain it.

Other than that, what you've suggested below sounds good. The Normalizer does not currently correctly handle local refs, and it also doesn't split the ref into URL + pointer, as I suggested in my earlier comment, so it will barf on path refs with braces.

[tedepstein]

Glad we've converged on a way forward here.

One more question: What do we do about caching? Loading external files can be expensive, especially when internet/intranet URLs. Should we coordinate a shared caching strategy, or even a shared cache, used by the validator and the normalizer?

[ghillairet]

@tedepstein @andylowry @tfesenko I started to put together a preliminary API for dealing with JSON references. The purpose of it is to be able to validate and resolve JSON references found in a JSON or YAML document. This API could be used to replace the SwaggerPreResolver class once the code that fixes references is added in this new API from SwaggerPreResolver.

The code can be found here https://github.com/RepreZen/SwagEdit/tree/task/59/com.reprezen.swagedit/src/com/reprezen/swagedit/json/references

The class that could replace SwaggerPreResolver is JsonReferenceResolver that is meant to be used to resolve all references inside a JSON document. The code that fixes Simple Refs in SwaggerPreResolver could be place in a specialized JsonReferenceFactory that would be used by the resolver to obtain the JSON references.

Currently the JsonReferenceResolver is not needed by SwagEdit but could be used to replace some code used for code completion, where we need to traverse the Swagger JSON schema. It would be used to have a fully resolved JSON schema when SwagEdit starts, instead of having to continuously resolve references when traversing the schema.

[tedepstein]

Looks like good progress, @ghillairet. Thanks for the update.

[tfesenko]

@ghillairet , please note work on SwaggerResolver in https://github.com/ModelSolv/RepreZen/pull/971

[ghillairet]

@tedepstein @andylowry @tfesenko This project could also be used to handle resolution of references inside a JSON document https://github.com/adjohnson916/jackson-json-reference

[andylowry]

@ghillairet Thanks - looks like an interesting library. I'm not sure we can adopt it as-is for a few reasons (take with a grain of salt - I've only read through most of the code in github - haven't experimented, and I could easily have misunderstood some stuff):

• Invalid references throw exceptions all the way to the top-level, which is too severe for our use. Normalizer leaves unresolvable references behind with markers (in the form of vendor exensions) identifying them as such
• Our special handling of fragments is not supported. Perhaps our handling is suspect, but we do it for, I think defensible reasons. Here's what we do:
  • Strip fragments from reference strings before attempting to treat what remains as a URI, to avoid needing to quote fragment characters disallowed by URI syntax, most importantly curly braces (fortunately, ~ is permitted in URI fragments). This is really due to the very unfortunate choice in Swagger to make paths be keys in the paths object, rather than allowing more sensible keys and making the paths themselves path item properties.
  • Handling of "simple refs" - reference strings like "Pet" (anything starting and ending with alphanum or underscore and also allowing internal hyphens) are turned into local reference according to context - e.g. #/definitions/Pet
  • Handling of "faux simple refs" that the Swagger Parser will incorrectly interpret as simple refs - e.g. defs.yaml#/defintiions/Pet would (yes, even in the current master branch!) be interpreted by SwaggerParser as #/definitions/defs.yaml#/definitions/Pet. We add ./ to the front of them to safeguard them (not needed by SwagEdit). SwaggerParser also botches URIs that contain a scheme that doesn't start with http - like HTTP:whatever or file:whatever but we can't fix those by prepending ./.
• Caching is keyed by full JSON Ref, which means different reference into the same document will each independently load and cache that full document.
• The markers left behind for unresolvable refs are part of a more general mechanism for leaving trace info about what the normalizer did in the resulting structure, e.g. hoisting of parameters from paths to operations, and resolving of references. The latter could not be done in the context of this framework.

All of this could, of course, be addressed by enhancing the framework - and ideally contributing it back. And the simple refs hacks are, hopefully, temporary and maybe could be addressed outside the library by introducing a preprocessing hook to be run whenever a JSON structure is obtained from a file or URI. That might be a good way to go - I do think the overall design is superior to mine. But I'm not sure whether it's important to do now.

@tedepstein, executive decision needed. It's probably strategically smart to move toward extending and adopting this library, assuming it has or is likely to get traction in the community. But I'm not sure it's what we should be focusing on at the moment.

[tedepstein]

@andylowry, @ghillairet, I took a look at the stats on that project: 4 forks, 3 contributors (but mostly adjohnson916 himself). Started October 2014, and not much recent activity.

Given the amount of work it would take to adapt this to our needs, I don't think it's worth the effort.

[tedepstein]

@tfesenko , I think this is done. If you agree, please close it.