classicvalues / parsly

Parsing utility (lexer/tokenizer)

Parsly - parsing utility.

This library is compatible with Go 1.11+

Please refer to CHANGELOG.md if you encounter breaking changes.

The goal of this project is to simplify implementing parsers with a tokenizer with a set of commonly use token matchers.

Usage

To build simple parser:

• define token lexical registry matchers

• create a tokenizer
• add parsing logic with MatchAfterWhitespace, MatchAny or Match

    var Whitespace = parsly.NewToken(0, "Whitespace", matcher.NewWhiteSpace())
    var Number = parsly.NewToken(1, "Number", matcher.NewNumber())
    var Term = parsly.NewToken(2, "Term", matcher.NewCharset("+-"))
    var Factor = parsly.NewToken(3, "Factor", matcher.NewCharset("*/"))
 
  
    func Parse(input []byte) (root *Expression, err error) {
        cursor := parsly.NewCursor("", input, 0)
        root = &Expression{}
        expression := root
        matched := cursor.MatchAfterOptional(Whitespace, Number)
        if matched.Code != Number.Code {
            return nil, cursor.NewError(Number)
        }
        value, _ := matched.Float(cursor)
        expression.LeftOp = NewValue(value)
    
        for ; ; {
            matched = cursor.MatchAfterOptional(Whitespace, Factor, Term)
            if matched.Code == parsly.EOF {
                break
            }
            operator := matched.Text(cursor)
            if expression.Operator != "" {
                expression.RightOp = &Operand{Expression: &Expression{LeftOp: expression.RightOp}}
                expression = expression.RightOp.Expression
            }
            expression.Operator = operator
    
            matched := cursor.MatchAfterOptional(Whitespace, Number)
            if matched.Code != Number.Code {
                return nil, cursor.NewError(Number)
            }
            value, _ := matched.Float(cursor)
            expression.RightOp = NewValue(value)
        }
        return root, nil
    }
    

See example basic arithmetic AST expression parser

Matchers

This project implements the following matchers:

• char to match individual rune
• charset to match set of runes
• digit to match a digit
• letter to match a letter
• fragment to match a fragment
• fragments to match a set of fragments
• number to match a number
• quote to match a quoted fragment
• terminator to match a byte terminated sequence

License

The source code is made available under the terms of the Apache License, Version 2, as stated in the file LICENSE.

Individual files may be made available under their own specific license, all compatible with Apache License, Version 2. Please see individual files for details.