expkg-zone58 / ex-xparse

Parse expressions in XPath or XQuery into XML trees using REx generated parsers

ex-xparse

Parses text into syntax trees based using a parser selected from a set of built-in parsers.
The REx parser generator was used to generate the parsers. This release uses the -basex option of REx to generate Java for BaseX.

It can be used to implement the EXPath xparse proposal .

Requirements

This release requires BaseX 8.6+. This is a result of XQuery spec changes to map:merge

Usage

import module namespace xp="expkg-zone58:text.parse";
xp:parse("2+3",map{"lang":"xquery","version":"3.1 basex"}) 

The second argument provides options to control the parsing of the text from the first argument.

Options

Option	Type---	Values	Default	Notes
lang	xs:string	XPath, XQuery	XPath	The language to be parsed
version	xs:string		""
flatten	xs:boolean		true()	Flatten the parse tree

The selected parser is chosen from the list in parser.xml. where the lang matches and version starts-with. see xp:parser($opts)

flatten skips elements with only one child element and no text.

Available parsers

 <parsers xmlns="expkg-zone58:text.parse">
  <!-- available languages and versions with REx parser implementation options. 
    add "-tree -java -basex" -->

  <parser lang="xpath">
    <version version="3.0" ebnf="xpath-30" sym="XPath" />
    <version version="3.1" ebnf="xpath-31" sym="XPath" />
  </parser>

  <parser lang="xquery">
    <version version="3.1 basex-2022-10-06" ebnf="BaseX" sym="XQuery"
      options="-ll 2 -backtrack " >For BaseX 9 includes XQuery update and full-text. Does not include http://docs.basex.org/wiki/XQuery_Extensions#Expressions</version>
    
    <version version="3.1" ebnf="xquery-31" sym="XQuery" /> 
  
    <version version="3.0" ebnf="xquery-30" sym="XQuery" />

    <version version="1.0" ebnf="xquery-10" sym="XQuery" />

    <version version="3.0 ML" ebnf="XQueryML30" sym="XQuery"
      options="-backtrack" />
  </parser>

  <parser lang="xqdoc-comments">
    <version version="20160405" ebnf="XQDocComments" sym="Comments" />
  </parser>

  <parser lang="ecmascript">
    <version version="5" ebnf="EcmaScript" sym="Program"
      options="-ll 1 -backtrack -asi" />
  </parser>
  <parser lang="rex">
        <version version="5.49" ebnf="REx" sym="Grammar" />
    </parser>
  

</parsers>

Examples

import module namespace xp="expkg-zone58:text.parse";
xp:parse("2+3",map{"lang":"xquery"}) 

result

<XQuery>
  <MainModule>
    <Prolog/>
    <AdditiveExpr>
      <IntegerLiteral>2</IntegerLiteral>
      <TOKEN>+</TOKEN>
      <IntegerLiteral>3</IntegerLiteral>
    </AdditiveExpr>
  </MainModule>
  <EOF/>
</XQuery>

With options

import module namespace xp="expkg-zone58:text.parse";

xp:parse("1+2",map{"lang":"xpath","flatten":false()})

Result:

<XPath>
  <Expr>
    <ExprSingle>
      <OrExpr>
        <AndExpr>
          <ComparisonExpr>
            <StringConcatExpr>
              <RangeExpr>
                <AdditiveExpr>
                  <MultiplicativeExpr>
                    <UnionExpr>
                      <IntersectExceptExpr>
                        <InstanceofExpr>
                          <TreatExpr>
                            <CastableExpr>
                              <CastExpr>
                                <UnaryExpr>
                                  <ValueExpr>
                                    <SimpleMapExpr>
                                      <PathExpr>
                                        <RelativePathExpr>
                                          <StepExpr>
                                            <PostfixExpr>
                                              <PrimaryExpr>
                                                <Literal>
                                                  <NumericLiteral>
                                                    <IntegerLiteral>1</IntegerLiteral>
                                                  </NumericLiteral>
                                                </Literal>
                                              </PrimaryExpr>
                                            </PostfixExpr>
                                          </StepExpr>
                                        </RelativePathExpr>
                                      </PathExpr>
                                    </SimpleMapExpr>
                                  </ValueExpr>
                                </UnaryExpr>
                              </CastExpr>
                            </CastableExpr>
                          </TreatExpr>
                        </InstanceofExpr>
                      </IntersectExceptExpr>
                    </UnionExpr>
                  </MultiplicativeExpr>
                  <TOKEN>+</TOKEN>
                  <MultiplicativeExpr>
                    <UnionExpr>
                      <IntersectExceptExpr>
                        <InstanceofExpr>
                          <TreatExpr>
                            <CastableExpr>
                              <CastExpr>
                                <UnaryExpr>
                                  <ValueExpr>
                                    <SimpleMapExpr>
                                      <PathExpr>
                                        <RelativePathExpr>
                                          <StepExpr>
                                            <PostfixExpr>
                                              <PrimaryExpr>
                                                <Literal>
                                                  <NumericLiteral>
                                                    <IntegerLiteral>2</IntegerLiteral>
                                                  </NumericLiteral>
                                                </Literal>
                                              </PrimaryExpr>
                                            </PostfixExpr>
                                          </StepExpr>
                                        </RelativePathExpr>
                                      </PathExpr>
                                    </SimpleMapExpr>
                                  </ValueExpr>
                                </UnaryExpr>
                              </CastExpr>
                            </CastableExpr>
                          </TreatExpr>
                        </InstanceofExpr>
                      </IntersectExceptExpr>
                    </UnionExpr>
                  </MultiplicativeExpr>
                </AdditiveExpr>
              </RangeExpr>
            </StringConcatExpr>
          </ComparisonExpr>
        </AndExpr>
      </OrExpr>
    </ExprSingle>
  </Expr>
  <EOF/>
</XPath>

Versions

import module namespace xp="expkg-zone58:text.parse";
xp:parse("map{'a':42}",map{"lang":"xquery"}) 

Result

<XQuery>
  <MainModule>
    <Prolog/>
    <MapConstructor>
      <TOKEN>map</TOKEN>
      <TOKEN>{</TOKEN>
      <MapConstructorEntry>
        <StringLiteral>'a'</StringLiteral>
        <TOKEN>:</TOKEN>
        <IntegerLiteral>42</IntegerLiteral>
      </MapConstructorEntry>
      <TOKEN>}</TOKEN>
    </MapConstructor>
  </MainModule>
  <EOF/>
</XQuery>

import module namespace xp="expkg-zone58:text.parse";
xp:parse("map{'a':42}",map{"lang":"xquery","version":"3.0"}) 

Result

<ERROR b="4" e="4" s="158">lexical analysis failed
while expecting [S, EOF, '!', '!=', '#', '(', '(:', ')', '*', '+', ',', '-', '/', '//', ';', '&lt;', '&lt;&lt;', '&lt;=', '=', '&gt;', '&gt;=', '&gt;&gt;', '[', ']', 'and', 'ascending', 'case', 'cast', 'castable', 'collation', 'count', 'default', 'descending', 'div', 'else', 'empty', 'end', 'eq', 'except', 'for', 'ge', 'group', 'gt', 'idiv', 'instance', 'intersect', 'is', 'le', 'let', 'lt', 'mod', 'ne', 'only', 'or', 'order', 'return', 'satisfies', 'stable', 'start', 'to', 'treat', 'union', 'where', '|', '||', '}']
at line 1, column 4:
...{'a':42}...</ERROR>

Building

Add a new ebnf

1. create ebnf in content/ebnf
2. add entry in parsers.xml

after update to ebnf

1. run tools/rex.xq to create java files in src/java for all ebnfs referenced in parsers.xml
2. run tools/makejar.xq to create jars in src/java
3. copy jars to main/content
4. increment version in main/expath-pkg.xml
5. run tools/build.xq

History

• versions before 0.5 used the namespace http://expath.org/ns/xparse

ex-xparse's Issues

parser rejects update without {}

Since Base10 these are required

Update without { ... } is used in the DBA app in dba/databases/database.xqm before version 10.

Wont fix.

Type error (VarRef) passing string to be parsed

Stopped at C:/Users/andy/basex.home/repo/http-expkg-zone58.github.io-ex-xparse-0.6.14/content/xparse.xqm, 1/70:
[XPTY0004] expkg_zone58.text.parse.Parse_BaseX:parseXQuery(xs:string) expected, (VarRef) found.
- C:/Users/andy/basex.home/repo/http-expkg-zone58.github.io-ex-xparse-0.6.14/content/xparse.xqm, 45/22

Value of xqdoc-comments parser?

Verbose output (due to html option), better to drop from ex-parse and use hand coded solution where required as done in xqlint

declare namespace com = "java:expkg_zone58.text.parse.Parse_XQDocComments";
com:parseComments("  (:~  hello
@see here:)")

<Comments>
  <S>  </S>
  <XQDocComment>
    <TOKEN>(:~</TOKEN>
    <Contents>
      <Char> </Char>
      <Char> </Char>
      <Char>h</Char>
      <Char>e</Char>
      <Char>l</Char>
      <Char>l</Char>
      <Char>o</Char>
      <Trim>
</Trim>
    </Contents>
    <TaggedContents>
      <TOKEN>@</TOKEN>
      <Tag>see</Tag>
      <Contents>
        <Char> </Char>
        <Char>h</Char>
        <Char>e</Char>
        <Char>r</Char>
        <Char>e</Char>
      </Contents>
    </TaggedContents>
    <TOKEN>:)</TOKEN>
  </XQDocComment>
</Comments>

Performance issue with REx v5.37.1211

built with REx 5.35

import module namespace xp="http://expath.org/ns/xparse";
xp:parse("2+3",map{"lang":"xquery","version":"3.1 cr-20141218"}) 

180ms

Same ebnf using REx 5.37

import module namespace xp="http://expath.org/ns/xparse";
xp:parse("2+3",map{"lang":"xquery","version":"3.1 rex5.37"}) 

Time 126,000 ms

Restore p:map2 type

declare function p:map2($c as xs:integer, $lo as xs:integer, $hi as xs:integer)**as xs:integer**

Time 180ms

see https://twitter.com/apb1704/status/714219874441146368

And for older tail recursion issue..
http://www.mail-archive.com/basex-talk%40mailman.uni-konstanz.de/msg07212.html

no support for BaseX extension if-then without else

See https://docs.basex.org/wiki/XQuery_Extensions#If_Without_Else

namespace for REx parsers?

The current ex-parse packaging installs various REx parsers as jars into the repo, but does not make them available outside of ex-parse. An application may only be interested in a particular parser.

The parser jars can be installed individually to the repository. Currently they appear just as the ebnf name e.g "EcmaScript".

Perhaps they should be in a namespace (package)?
Name?
rex.parser.EcmaScript

Parse bug with replace

The parse fails on references to the replace function. e.g.
replace($a,"A","B")
This is due to a grammar conflicit with the use of replace in XQuery update.
workaround update source to use fn:replace

UpdatingFunctionCall parse error

invoke updating $f()

causes parse error

<ERROR b="8" e="13" s="1741">lexical analysis failed while expecting 'update' after successfully scanning 5 characters beginning at line 1, column 8:
...updating $f(1)...</ERROR>

A more readable xparse spec

The README points to https://lists.w3.org/Archives/Public/public-expath/2015Feb/att-0003/xparse.html as the location for the spec. On my browser that page is unreadable:

I wonder if John Lumley would provide his original file and if we could resurrect this as an EXPath spec + implementation? I can see this as being a very useful pedagogical and debugging tool.

Thoughts @adamretter @ChristianGruen?

UpdatingFunctionCall Basex allows omit of invoke

TheFix:

UpdatingFunctionCall
         ::= 'invoke'? 'updating' PrimaryExpr '(' ( ExprSingle ( ',' ExprSingle )* )? ')'