gerhobbelt / jison-lex

generates lexical analyzers. used by jison.

jison-lex [OBSOLETED]

A lexical analyzer generator used by jison. It takes a lexical grammar definition (either in JSON or Bison's lexical grammar format) and outputs a JavaScript lexer.

deprecation notice

From today (2017/oct/15) the jison-lex repository is obsoleted for the jison-lex package/codebase: the primary source is the jison monorepo's packages/jison-lex/ directory. See also GerHobbelt/jison#16.

(For a comparable argument, see also "Why is Babel a monorepo?")

Issues, pull requests, etc. for jison-lex should be filed there; hence we do not accept issue reports in this secondary repository any more.

This repository will track the primary source for a while still, but be very aware that this particular repository will always be lagging behind!

install

npm install jison-gho

Then the jison-lex library is located in the subdirectory packages/jison-lex/ of the jison-gho monorepo, i.e. .../node_modules/jison-gho/packages/jison-lex/.

Alternatively, the entire jison-lex API is also available via the jison API itself as can be seen from this internal jison code snippet:

import Lexer from '../packages/jison-lex';
import ebnfParser from '../packages/ebnf-parser';
import lexParser from '../packages/lex-parser';
import grammarPrinter from './util/grammar-printer.js';
import helpers from '../packages/helpers-lib';
var rmCommonWS = helpers.rmCommonWS;
var camelCase  = helpers.camelCase;
var code_exec  = helpers.exec;
import XRegExp from '@gerhobbelt/xregexp';
import recast from '@gerhobbelt/recast';
import astUtils from '@gerhobbelt/ast-util';
import json5 from '@gerhobbelt/json5';

// Also export other APIs: the JISON module should act as a 'facade' for the others,
// so applications using the JISON compiler itself can rely on it providing everything
// in a guaranteed compatible version as it allows userland code to use the precise
// same APIs as JISON will be using itself:
Jison.Lexer = Lexer;
Jison.ebnfParser = ebnfParser;
Jison.lexParser = lexParser;
Jison.codeExec = code_exec;
Jison.XRegExp = XRegExp;
Jison.recast = recast;
Jison.astUtils = astUtils;
Jison.JSON5 = json5;
Jison.prettyPrint = grammarPrinter;
Jison.rmCommonWS = rmCommonWS;
Jison.mkStdOptions = mkStdOptions;
Jison.camelCase = camelCase;
Jison.autodetectAndConvertToJSONformat = autodetectAndConvertToJSONformat;
...
Jison.Parser = Parser;

export default Jison;

hence you can get at it this way, for example:

import jisonAPI from 'jison-gho';
// get a reference to the full `jison-lex` API:
const jisonLexAPI = jisonAPI.Lexer;

build

To build the parser yourself, follow the install & build directions of the monorepo.

Note about ES6/rollup usage vs. ES5

All dist/ library files are 'self-contained': they include all 'local imports' from within this jison monorepo in order to deliver a choice of source files for your perusal where you only need to worry about importing external dependencies (such as recast).

As such, these dist/ files should be easier to minify and/or use in older (ES5) environments.

rollup

Iff you use rollup or similar tools in an ES6/ES2015/ES2017 setting, then the package.json::module has already been set up for you to use the original sources instead!

usage

Usage: jison-lex [file] [options]

file     file containing a lexical grammar

Options:
   -o FILE, --outfile FILE       Filename and base module name of the generated parser
   -t TYPE, --module-type TYPE   The type of module to generate (commonjs, js)
   --version                     print version and exit

programmatic usage

var JisonLex = require('@gerhobbelt/jison-lex');

var grammar = {
  rules: [
    ["x", "return 'X';" ],
    ["y", "return 'Y';" ],
    ["$", "return 'EOF';" ]
  ]
};

// or load from a file
// var grammar = fs.readFileSync('mylexer.l', 'utf8');

// generate source
var lexerSource = JisonLex.generate(grammar);

// or create a parser in memory
var lexer = new JisonLex(grammar);
lexer.setInput('xyxxy');
lexer.lex();
// => 'X'
lexer.lex();
// => 'Y'

license

MIT

related repositories

• jison / jison-gho @ NPM
• jison-lex @ NPM
• lex-parser @ NPM
• ebnf-parser @ NPM
• jison2json @ NPM
• json2jison @ NPM
• jison-helpers-lib @ NPM

• secondary source repositories

  • jison-lex
  • lex-parser
  • ebnf-parser
  • jison2json
  • json2jison
  • jison-helpers-lib