ykankaya / pest

The Elegant Parser

pest. The Elegant Parser

pest is a PEG parser with simplicity and speed in mind.

Elegant grammar

Defining a grammar for a list of alpha-numeric identifiers where the first identifier does not start with a digit is as straight-forward as:

alpha = { 'a'..'z' | 'A'..'Z' }
digit = { '0'..'9' }

ident = { (alpha | digit)+ }

ident_list = _{ !digit ~ ident ~ (" " ~ ident)+ }
          // ^
          // ident_list rule is silent which means it produces no tokens

This is then saved in a .pest grammar file and is never mixed up with Rust code which results in an always up-to-date formal definition of the grammar which is very easy to maintain.

Pairs API

The grammar can be used to derive a Parser implementation automatically. Parsing returns an iterator of nested token pairs:

extern crate pest;
#[macro_use]
extern crate pest_derive;

use pest::Parser;

#[derive(Parser)]
#[grammar = "ident.pest"]
struct IdentParser;

fn main() {
    let pairs = IdentParser::parse(Rule::ident_list, "a1 b2").unwrap_or_else(|e| panic!("{}", e));

    // Because ident_list is silent, the iterator will contain idents
    for pair in pairs {

        let span = pair.clone().into_span();
        // A pair is a combination of the rule which matched and a span of input
        println!("Rule:    {:?}", pair.as_rule());
        println!("Span:    {:?}", span);
        println!("Text:    {}", span.as_str());

        // A pair can be converted to an iterator of the tokens which make it up:
        for inner_pair in pair.into_inner() {
            let inner_span = inner_pair.clone().into_span();
            match inner_pair.as_rule() {
                Rule::alpha => println!("Letter:  {}", inner_span.as_str()),
                Rule::digit => println!("Digit:   {}", inner_span.as_str()),
                _ => unreachable!()
            };
        }
    }
}

This produces the following output:

Rule:    ident
Span:    Span { start: 0, end: 2 }
Text:    a1
Letter:  a
Digit:   1
Rule:    ident
Span:    Span { start: 3, end: 5 }
Text:    b2
Letter:  b
Digit:   2

Meaningful error reporting

Parsing "123" instead of "a1 b2" in the code above will result in the following panic:

thread 'main' panicked at ' --> 1:1
  |
1 | 123
  | ^---
  |
  = unexpected digit', src/main.rs:12

while parsing "ab *" will result in:

thread 'main' panicked at ' --> 1:4
  |
1 | ab *
  |    ^---
  |
  = expected ident', src/main.rs:12

Sheer performance

pest provides parsing performance in the same league as carefully written manual parsers. The following JSON benchmark puts it somewhere in between one of the most optimized JSON parsers, ujson4c, and a static native-speed parser, nom.

The first entry of pest scores 36ms, while the second scores 96ms since it's mapping Pairs to a custom JSON AST. While the first entry forms a perfectly usable tree, it does not process the file to a fully-processed JSON object. The second one does, but since it has an extra intermediate representation of the object, it repeats some work.

The benchmark uses a large 2MB JSON file. Tested on a 2.6GHz Intel® Core™ i5 running macOS.

Other features

• precedence climbing
• input handling
• custom errors
• runs on stable Rust

Usage

pest requires Cargo and Rust 1.23.

Add the following to Cargo.toml:

pest = "^1.0"
pest_derive = "^1.0"

and in your Rust lib.rs or main.rs:

extern crate pest;
#[macro_use]
extern crate pest_derive;

Projects using pest

• pest_meta (bootstrapped)
• brain
• Chelone
• comrak
• graphql-parser
• handlebars-rust
• Huia
• py_literal
• rouler
• RuSh
• rs_pbrt
• stache
• tera
• ui_gen
• ukhasnet-parser

Special thanks

A special round of applause goes to prof. Marius Minea for his guidance and all pest contributors, some of which being none other than my friends.