hexacta / tokenizer

Split text into tokens.

tokenizer

Split text into tokens.

Install

$ npm install --save hx-tokenizer

Usage

const tokenizer = require("hx-tokenizer");
const tokens = tokenizer.tokenize("Lorem ipsum, something.");
// tokens == ["Lorem", "ipsum", ",", "something", "."]
const text = tokenizer.join(tokens);
// text == "Lorem ipsum, something."

API

tokenizer.tokenize(text)

tokenizer.join(tokens)

License

MIT © Hexacta