Let's talk about IETF standardization of JSONPath about json-path-comparison

[cabo]

So we now have a mailing list. This is important as the IETF is set up to make decisions on mailing lists, and while the detailed wordsmithing is likely to happen on github, the mailing list provides a better way to include other IETFers into the more high-level decisions. Please subscribe if you care about the standardization project. Work on the charter is likely to happen in earnest when people have subscribed (and IETF108 is over). I'll make a first proposal.

Announcement copied below, with subscription link.

A new IETF non-working group email list has been created.

List address: [email protected]
Archive: https://mailarchive.ietf.org/arch/browse/jsonpath/
To subscribe: https://www.ietf.org/mailman/listinfo/jsonpath

Purpose:
For matching and selecting parts of a data item tree, XML has XPath. In 2007, Stefan Goessner proposed JSONPath as an analogue for JSON. This format has seen wide use, with dozens of implementations, but has not yet been formally standardized. The conditions for standardizing now seem to align. This mailing list is intended for discussing the creation of an IETF activity for this (likely in the form of a WG), initially by agreeing a proposed charter for this WG. This charter proposal will then receive further discussion in the DISPATCH WG. Concurrently, discussions about technical issues in JSONPath standardization are also welcome.

[cabo]

Hi Glyn,

formally, IETF WGs do not create test suites and reference implementations as an IETF product (well, reference implementations they sometimes do, as well as some test vectors, but I think the effort needed here goes beyond what we would put in an RFC). But that does not have to discourage us from considering those as parts of the package. I.e., we could make it explicit that we don't want to create a specification without having the other pieces in place.

We also would want to make sure the specification actually stands for itself, i.e., you don't have to consult any reference implementation to build another. An IETF specification will by definition be developed in the open, let's make sure that happens for the reference implementations and test suites as well.

My slides for today are at http://slides.cabo.space right now -- there is still a window of a couple hours where I should be able to update them, if needed.

See you at the DISPATCH meeting!

[cabo]

Quick report from the meeting today: A JSONPath mailing list will be created, we get to discuss a charter for a new WG there, and then the DISPATCH group will decide if that is good to go. A little more process than I'd like to have, but still manageable. We can do this during the summer lull.

More later.

[danielaparker]

@cabo In your slide "Aren't there other ways to do this?", a notable omission is JMESPath. JMESPath is a query language for transforming JSON documents into other JSON documents. It's supported in both the AWS and Azure CLI and has libraries available in a number of languages. Unlike JSONPath, which is an ad hoc creation, it has a formal spec with compliance tests.

Having implemented both, jsonpath and jmespath, my impression is that the JMESPath grammar, while far more expressive, can largely cover all of the JSONPath syntax with the exception of the recursive descent notation "..".

[remorhaz]

In fact, I see two basic possible goals:

1. Provide minimalistic standard based on existing consensus, so that existing implementations could call themselves "standard-compliant" with little or no efforts.
2. Provide powerful standard, trying to break as few existing queries as possible.

I believe that first option is useless, because implementations would not cut off the power they had already implemented and thus we'll end in having non-standard "extensions" of core standard. I don't see any profits in it. @cburgmer 's work on calculating consensus clearly discovered that "bloodless" transfer to anything but very minimalistic subset of current inmplementations' functionality is impossible.

If we follow the second option, we in fact declare that we're creating a new tool based on JSONPath ideas, partially compliant with existing implementations. That leaves to JSONPath implementors two viable options:

• switch to new standard (and new implementation) and still have enough power, loosing some of BC;
• turn their JSONPath "dialect" to an extension of the new standard, clearly declaring the "extras" and loosing little or none of BC. This way also allows subsequent gradual deprecation of "extras" without breaking all things in a moment.

If JSONPath users could be satisfied with less powerful, but standard tool - they'd choose JSON Pointer. That's why I stand for the second option.

[bettio]

I agree with @remorhaz, the first option is not an option, a JSONPath standard should be built on existing consensus, therefore we should work to add some constraints on corner cases and ill formed paths (e.g. I think nearly no one will complain if we forbid $.foo[010] since I don't think there are a lot of users using bracket notation with octal integers).

I think we should work towards opening an issue for each macro group/topic, like I did here: ietf-wg-jsonpath/draft-ietf-jsonpath-base#9, ietf-wg-jsonpath/draft-ietf-jsonpath-base#8.

I mention some further macro categories that should be discussed:

• Valid bracket notations $.foo[-1]
• Allowed integer formats (are 010 and 0x10 valid?)
• Special dot notation identifiers handling ($.1 on {"1": "foo"})
• Allowed filter expressions
• etc...

If you all agree I think we should continue any further discussion here and start opening more issues.

[glyn]

Thanks @cabo for raising this topic!

I believe there are four short-term goals:

• build a viable Working Group of implementers, users, and other interested parties
• maximise and maintain active participation, which probably requires developing in the open
• maintain velocity, or accelerate, compared to our internet draft work here
• agree the deliverables of the working group. Clearly, these will include an IETF Internet Draft, but may also include a compliance test suite and reference implementation. (I would prefer all three.)

I plan to join the IETF 108 meeting today.

[glyn]

Hi @cabo

I'm not sure how much we'll be able to cover in the meeting today, so let me dump my current thoughts here and please feel free to update your slides if appropriate.

• I agree that the IETF deliverable should consist of a comprehensive/complete specification. The purpose of any reference implementation or compliance test suite would be to support the development of the spec and the migration of existing implementations towards the spec. In particular, it's hard to see that any RI or CTS could be actively maintained in the long term (i.e. decades).

• The group which I assembled in June consists of 19 people who are mostly implementers/maintainers of JSONPath implementations, including @cburgmer of JSONPath comparison fame. There are only a couple of users AFAIK, so we certainly need to expand the number of users in a WG. So far three of us have expressed a willingness to help with the writing of an Internet Draft while four more are happy to review. (The other 12 seem to be taking a back seat for now but I know that some of them are keen to adopt any future standard in their implementations.)

• I created a github organsation https://github.com/jsonpath-standard for developing an internet draft and we have made some progress with a draft document, although currently I'm the only active author (although others hope to find the time). I would happily transfer ownership of this organisation to yourself if that would be helpful. (Similarly, although the draft we've been developing has my name in the filename, I'd be delighted to see Stefan Göessner's name there instead as in your draft.) I'd also personally be happy to restart the https://github.com/jsonpath-standard/internet-draft repository from scratch so that a fresh start could be made by the assembled WG, although I haven't asked the others how they would feel about that. Alternatively, if you or the IETF prefer another location for the actual editing work, that's fine.

• I believe our draft has made a good start in sorting out how to standardise the syntax and in particular the semantics and I'd be happy for any of this to be built upon if that would be helpful (although I still need to obtain sign-off from my employer before my contributions could be officially published, although this should be a formality).

• I'm a bit fuzzy on precisely how an IETF WG runs and whether the spec work is anticipated to take just a few months or quite a bit longer. I'd be happy to be involved for 6 months, but I can't necessarily commit to a longer term. Any clarification of the likely velocity of the spec work would be really helpful.

See you later!

[cabo]

Hi Glyn, thank you for putting this in! The meeting starts soon, so I’ll be a bit terse and only reply to some points, but let’s keep up the discussion. Grüße, Carsten

• The group which I assembled in June consists of 19 people who are mostly implementers/maintainers of JSONPath implementations, including @cburgmer of JSONPath comparison fame. There are only a couple of users AFAIK, so we certainly need to expand the number of users in a WG. So far three of have expressed a willingness to help with the writing of an Internet Draft while four more are happy to review. (The other 12 seem to be taking a back seat for now but I know that some of them are keen to adopt any future standard in their implementations.)

Good. Typically Internet-Drafts with this number of contributors have a small number of editors (2, 3, 4) and a list of contributors listed. We also have some interesting users lined up to join the effort; I’m really thrilled at this point.

• I created a github organsation https://github.com/jsonpath-standard for developing an internet draft and we have made some progress with a draft document, although currently I'm the only active author (although others hope to find the time). I would happily transfer ownership of this organisation to yourself if that would be helpful. (Similarly, although the draft we've been developing has my name in the filename, I'd be delighted to see Stefan Göessner's name there instead as in your draft.) I'd also personally be happy to restart the https://github.com/jsonpath-standard/internet-draft repository from scratch so that a fresh start could be made by the assembled WG, although I haven't asked the others how they would feel about that. Alternatively, if you or the IETF prefer another location for the actual editing work, that's fine.

That will be for the WG chairs to decide. I’m not going to be a WG chair for this; I would like to push this forward from the editor side. So let’s see who wants to be WG chair. If there is a new group, we’ll probably look for an experienced IETFer plus some more on the side of the specialization of the WG.

• I believe our draft has made a good start in sorting out how to standardise the syntax and in particular the semantics and I'd be happy for any of this to be built upon if that would be helpful (although I still need to obtain sign-off from my employer before my contributions could be officially published, although this should be a formality).

We often have these release processes, that is usually just a matter of scheduling for them when considering deadlines. I think we’ll need to discuss structure and content of the specification before plunging right in there. I see we have two straw men right one (one actually an Internet-Draft) that actually complement each other and could be merged. But that is not something we want to rush (as in: “this week” :-), especially in the middle of summer vacations.

• I'm a bit fuzzy on precisely how an IETF WG runs and whether the spec work is anticipated to take just a few months or quite a bit longer. I'd be happy to be involved for 6 months, but I can't necessarily commit to a longer term. Any clarification of the likely velocity of the spec work would be really helpful.

My experience is that we could be done at the end of the year from a technical perspective, but then we’d need another 6 months of patience for going through the approval and publication processes (no worries, I’ve been through this before). That is about as fast as it gets; if we have any technical hangups, it will take longer. So I think it would be good to budget the time we can invest in this, both in an absolute way and in terms of the real-time component. Grüße, Carsten

[about-code]

XML has a pretty complete set of standards in terms of processing, querying and transforming XML documents and it is very exciting to see any efforts in standardization of a JSON Query Language as powerful as XPath. Everything which helps closing the gaps that exist in comparison to XML is welcome from my point of view as an application developer.

In this regard JSONPath would fill the gap of querying JSON documents. Apart from that there's currently no standardized equivalent to XSLT for transforming/mapping/projecting JSON documents. For sure, such one could be built from JSONPath similar to how XSLT uses XPath at its core. Nevertheless, JMESPath and jq are very interesting languages in this regard, as they not only provide querying but also projections.

The W3C XML standards used to separate the concerns of querying documents (XPath) and transforming/projecting documents (XSLT). Yet a JSON query language also capable of projections could prove very useful and a great step towards feature parity with XML.