azu / retext

Extensible system for analysing and manipulating natural language

retext is an extensible natural language system—by default using parse-latin to transform natural language into a TextOM object model. Retext provides a pluggable system for analysing and manipulating natural language in JavaScript. NodeJS and the browser. Tests provide 100% coverage.

Rather than being a do-all library for Natural Language Processing (such as NLTK or OpenNLP), retext aims to be useful for more practical use cases (such as censoring profane words or decoding emoticons, but the possibilities are endless) instead of more academic goals (research purposes). retext is inherently modular—it uses plugins (similar to rework for CSS) instead of providing everything out of the box (such as Natural). This makes retext a viable tool for use on the web.

Installation

npm:

npm install retext

Component.js:

component install wooorm/retext

Bower:

bower install retext

Duo:

var Retext = require('wooorm/retext');

UMD (globals/AMD/CommonJS) (uncompressed and compressed):

<script src="path/to/retext.js"></script>
<script>
  var retext = new Retext();
</script>

Usage

The following example uses retext-emoji (to show emoji) and retext-smartypants (for smart punctuation).

/* Require dependencies. */
var Retext = require('retext');
var emoji = require('retext-emoji');
var smartypants = require('retext-smartypants');

/* Create an instance using retext-emoji and -smartypants. */
var retext = new Retext()
    .use(emoji, {
        'convert' : 'encode'
    })
    .use(smartypants);

/* Read a document. */
retext.parse(
    'The three wise monkeys [. . .] sometimes called the ' +
    'three mystic apes--are a pictorial maxim. Together ' +
    'they embody the proverbial principle to ("see no evil, ' +
    'hear no evil, speak no evil"). The three monkeys are ' +
    'Mizaru (:see_no_evil:), covering his eyes, who sees no ' +
    'evil; Kikazaru (:hear_no_evil:), covering his ears, ' +
    'who hears no evil; and Iwazaru (:speak_no_evil:), ' +
    'covering his mouth, who speaks no evil.',
    function (err, tree) {
        /* Handle errors. */
        if (err) {
            throw err;
        }

        /* Log the text content of the tree (the transformed input). */
        console.log(tree.toString());
        /**
         * This logs the following:
         *   The three wise monkeys […] sometimes called the three
         *   mystic apes—are a pictorial maxim. Together they
         *   embody the proverbial principle to (“see no evil,
         *   hear no evil, speak no evil”). The three monkeys are
         *   Mizaru (🙈), covering his eyes, who sees no evil;
         *   Kikazaru (🙉), covering his ears, who hears no evil;
         *   and Iwazaru (🙊), covering his mouth, who speaks no evil.
         */
    }
);

API

Retext(parser?)

var Retext = require('retext');
var ParseEnglish = require('parse-english');
var retext = new Retext(new ParseEnglish());

/* There, ol’ chap. */
retext.parse('Some English', function (err, tree) {/* ... */});

Return a new Retext instance with the given parser (defaults to an instance of parse-latin).

Retext#use(plugin, options?)

Takes a plugin—a humble function to transform the object model. Optionally takes an options object, but it’s up to plugin authors to support settings.

Retext#parse(value, options?, done(err, tree))

Parses the given source and, when done, passes either an error (the first argument), or the (by used plugins, modified) document (the second argument) to the callback.

plugin

A plugin is simply a function, with function(retext, options?) as its signature. The first argument is the Retext instance a user attached the plugin to. The plugin is invoked when a user uses the plugin (not when a document is parsed) and enables the plugin to modify the internal Object Model (retext.TextOM) or the parser (retext.parser).

The plugin can return another function: function(NLCSTNode, options, next?). This function is invokeded when a document is parsed. It’s given the document as created by Retext#parse() before it’s given to the user.

Plugins

• retext-content — Append, prepend, remove, and replace content into/from Retext nodes;

• retext-cst — (demo) — Encoding and decoding between AST (JSON) and TextOM object model;

• retext-directionality — (demo) — Detect the direction text is written in;

• retext-dom — (demo) — Create a (living) DOM tree from a TextOM tree;

• retext-double-metaphone — (demo) — Implementation of the Double Metaphone algorithm;

• retext-emoji — (demo) — Encode or decode Gemojis;

• retext-find — Easily find nodes;

• retext-inspect — (demo) — Nicely display nodes in console.log calls;

• retext-keywords — (demo) — Extract keywords and keyphrases;

• retext-lancaster-stemmer — (demo) — Implementation of the Lancaster (Paice/Husk) algorithm;

• retext-language — (demo) — Detect the language of text;

• retext-link — (demo) — Detect links in text;

• retext-live — Change a node based on a (new?) value;

• retext-metaphone — (demo) — Implementation of the Metaphone algorithm;

• retext-porter-stemmer — (demo) — Implementation of the Porter stemming algorithm;

• retext-pos — (demo) — Part-of-speech tagger;

• retext-range — Sequences of content within a TextOM tree between two points;

• retext-search — (demo) — Search in a TextOM tree;

• retext-sentiment — (demo) — Detect sentiment in text;

• retext-smartypants — (demo) — Implementation of SmartyPants;

• retext-soundex — (demo) — Implementation of the Soundex algorithm;

• retext-syllable — (demo) — Syllable count;

• retext-visit — (demo) — Visit nodes, optionally by type;

• retext-walk — Walk trees, optionally by type.

Desired Plugins

Hey! Want to create one of the following, or any other plugin, for retext but not sure where to start? I suggest to read retext-visit’s source code to see how it’s build first (it’s probably the most straight forward to learn), and go from there. Let me know if you still have any questions, go ahead and send me feedback or raise an issue.

• retext-date — Detect time and date in text;

• retext-frequen -words — Like retext-keywords, but based on frequency and stop-words instead of a POS-tagger;

• retext-hyphen — Insert soft-hyphens where needed; this might have to be implemented with some sort of node which doesn’t stringify;

• retext-location — Track the position of nodes (line, column);

• retext-no-pants — Opposite of retext-smartypants;

• retext-no-break — Inserts non-breaking spaces between things like “100 km”;

• retext-profanity — Censor profane words;

• retext-punctuation-pair — Detect which opening or initial punctuation, belongs to which closing or final punctuation mark (and vice versa);

• retext-summary — Summarise text;

• retext-sync — Detect changes in a textarea (or contenteditable?), sync the diffs over to a retext tree, let plugins modify the content, and sync the diffs back to the textarea;

• retext-typography — Applies typographic enhancements, like (or using?) retext-smartypants and retext-hyphen;

• retraverse — Like Estraverse.

Parsers

• parse-latin (demo) — default;

• parse-english (demo) — Specifically for English;

• parse-dutch (demo) — Specifically for Dutch;

Benchmark

On a MacBook Air, it parses about 2 big articles, 25 sections, or 230 paragraphs per second.

           retext.parse(value, callback);
  230 op/s » A paragraph (5 sentences, 100 words)
   25 op/s » A section (10 paragraphs, 50 sentences, 1,000 words)
    2 op/s » An article (100 paragraphs, 500 sentences, 10,000 words)

Related

• textom
• nlcst

License

MIT © Titus Wormer