thautwarm/Typed-BNF

Backend-agnostic BNF grammar with type inference and semantic actions.

For IDE support, we provide a VSCode extension with the following features:

• Semantic-based syntax highlighting
• Go to definition/go to references/find all references

• Typed BNF Documentations

• Backend-agnostic grammar definitions with typed semantic actions.
• Type inference for semantic actions, including slots ($1), let bindings, lambdas, tuples, lists and record field access.
• Algebraic data types, records, external values and external types.
• Parametric nonterminals for reusable grammar fragments such as separated lists.
• Lexer rules with ranges, literals, references, repetition, optionality, negation and ignored tokens.
• Backend-specific name mapping through tbnf.config.js.
• Generated parsers for C#, TypeScript and Rust, plus pure BNF output for readable syntax specifications.

So far, we support several different architectures, which unveil the capability of Typed BNF's backend agnostic code generation.

(PL = programming language; PGEN = parser generator; pure bnf means it is the pure BNF for readable syntax specification )

You might check the following example/test scripts for detailed usage guide.

• test-scripts/test_typescript_lua_tu.sh: Lua parser in TypeScript. (Algebraic) Data types are encoded using tagged unions.
• test-scripts/test_typescript_lua.sh: Lua parser in TypeScript. (Algebraic) Data types are encoded using case classes.
• test-scripts/test-csharp-lua.sh: Lua parser in CSharp.
• test-scripts/test-csharp-json.sh: JSON parser in CSharp.
• test-scripts/test-rust-json-example.sh: committed Rust Cargo example generated from hello_world/Json.tbnf.
• test-scripts/run-tests.sh grammar-matrix: same grammar generated and asserted in C#, TypeScript, and Rust.

Note that JSON parsers generated for different programming languages come from the same grammar. The Rust Lua test uses runtests/lua_lr.tbnf, an LR-compatible variant of the ANTLR-oriented runtests/lua.tbnf.

Support for Python Lark and OCaml Menhir is legacy/deprecated since v0.4; the OCaml backend is currently broken against the current CLI/toolchain. See v0.3 for the last reliable legacy-backend snapshot.

Download the single executable file tbnf-VERSION-TARGET (e.g., tbnf-0.4.0-win-x64.exe, tbnf-0.4.0-osx-arm64) from the release page.

Usage: tbnf [options] <source-grammar-file>
Version: 0.4.3
Options:
  --version                 Show version and exit
  -h, --help                Show this help message and exit
  -o, --outDir DIR          Specify output directory (default: same as source file)
  -be, --backend TYPE       Backend to use
     Possible TYPE values:
       csharp-antlr         C# backend using ANTLR
       typescript-antlr     TypeScript backend using ANTLR
       rust-lrpar           Rust backend using grmtools/lrlex/lrpar
       pure-bnf             PureBNF backend

  -ae, --adt-encoding TYPE  ADT encoding
     Possible TYPE values:
       tagged-union         ADT encoding via tagged unions (default for TypeScript)
       case-class           ADT encoding via case classes (default for C#)
  -lang, --language NAME    Language name to generate (default: "mylang")
  -conf, --config PATH      Path to the 'tbnf.config.js' file (default: <outDir>/tbnf.config.js)

Examples:
  tbnf -lang mylanguage mygrammar.tbnf -be typescript-antlr -ae tagged-union
  tbnf -lang mylanguage mygrammar.tbnf -be csharp-antlr -conf tbnf.config.js

You might check out Typed BNF documentation.

For TypeScript backends, you will also need the antlr-ng compiler and antlr4ng runtime.

For C# ANTLR output, you will also need the antlr4 command line tool, install it from https://github.com/antlr/antlr4-tools. The Rust lrpar backend generates a Cargo project and requires a Rust toolchain; its grammar must be LR-compatible.

The following grammar compiles and runs for programming languages and parser architectures supported by TBNF.

extern var parseInt : str -> int
extern var parseFlt : str -> float
extern var getStr : token -> str
extern var unesc : str -> str
extern var appendList : <a> (list<a>, a) -> list<a>

type Json
type JsonPair(name: str, value: Json)

case JInt : int -> Json
case JFlt : float -> Json
case JStr : str -> Json
case JNull : () -> Json
case JList : (elements: list<Json>) -> Json
case JDict : list<JsonPair> -> Json
case JBool : bool -> Json

ignore space

digit = [0-9] ;

start : json { $1 }

int = digit+ ;
float = digit* "." int ;
str = "\"" ( "\\" _ | ! "\"" )* "\"" ;
space = ("\t" | "\n" | "\r" | " ")+;

seplist(sep, elt) : elt { [$1] }
                  | seplist(sep, elt) sep elt
                    { appendList($1, $3) }

jsonpair : <str> ":" json { JsonPair(unesc(getStr($1)), $3) }

/* CPP comments */

json : <int> { JInt(parseInt(getStr($1))) }
      | <float> { JFlt(parseFlt(getStr($1))) }
      | "null"  { JNull() }
      | <str>   { JStr(unesc(getStr($1))) }
      | "[" "]" { JList([]) }
      | "{" "}" { JDict([]) }
      | "true"  { JBool(true) }
      | "false"  { JBool(false) }
      | "[" seplist(",", json) "]" { JList($2) }
      | "{" seplist(",", jsonpair) "}" { JDict($2) }

Put a tbnf.config.js in the output directory, or pass one explicitly with -conf / --config, to define how variables, types, fields and constructors map from Typed BNF names to backend-language names.

The .NET CLI first looks for CommonJS-style exports and falls back to top-level declarations when an export is missing. Supported hooks are rename_var, rename_type, rename_ctor, rename_field, and the legacy OCaml-only start_rule_qualified_type.

For example, this TypeScript config maps Typed BNF built-ins to TypeScript runtime types:

"use strict";

module.exports = {
  rename_type(x) {
    if (x == "list") return "Array";
    if (x == "int" || x == "float") return "number";
    if (x == "str") return "string";
    if (x == "bool") return "boolean";
    if (x == "token") return "antlr.Token";
    return x + "_t";
  }
};

A top-level function works as a fallback too:

function rename_type(x) {
  if (x == "str") return "string";
  return x;
}

Key points:

1. Typed BNF has 7 built-in types: token, tuple, list, int, float, str and bool.
2. Typed BNF ships with no built-in functions, which makes it suitable to write portable grammars without ruling out semantic actions.
3. External values/types declared in a grammar must be implemented in the generated backend project.

Check out Backends.*.fs

The repository includes a Docker/Compose development environment with all parser-generator toolchains installed.

# Build and start a persistent dev container
test-scripts/docker-test.sh up

# Run a shell in the container
test-scripts/docker-test.sh shell

# Run tests through docker compose
test-scripts/docker-test.sh test smoke
test-scripts/docker-test.sh test all
test-scripts/docker-test.sh test typecheck
test-scripts/docker-test.sh test csharp-json
test-scripts/docker-test.sh test grammar-matrix
test-scripts/docker-test.sh test 'rust-*'

# List all available suites
test-scripts/docker-test.sh list

# Execute arbitrary commands in the running container
test-scripts/docker-test.sh exec 'deno run -A build.ts aot'

# Stop the container; named caches are kept
test-scripts/docker-test.sh down

all runs the stable suites, including C#/TypeScript and the cross-backend grammar matrix. Use rust-* for the full Rust backend regression suite. Legacy Python/OCaml and Julia backends are listed as skipped because they are not reliable test targets for the current CLI/toolchain.

• .NET 8.0 SDK
• Deno
• Antlr4 (for C# ANTLR output)
• Antlr4NG & Antlr-NG (for TypeScript backends)
• Rust and Cargo (for Rust lrpar output)

deno run -A build.ts build

All distributions are built into the dist folder.

> ls -lhp dist | grep -v /$ | awk '{print $5 "\t" $9}'
70M     TBNF.CLI.exe
44K     TBNF.CLI.pdb
132K    TBNF.Core.pdb
75M     tbnf-0.4.0-linux-arm64
69M     tbnf-0.4.0-linux-x64
75M     tbnf-0.4.0-osx-arm64
69M     tbnf-0.4.0-osx-x64
70M     tbnf-0.4.0-win-x64.exe

The grammar for Typed BNF is also implemented using Typed BNF.

deno run -A build.ts bootstrap-once