parinfer / parinfer.js

Let's simplify the way we write Lisp

Parinfer.js

Pronounced "par-in-fur". "par" rhymes with car and star; "fur" rhymes with blur and stir

See the Parinfer Home Page for the original animated demo page.

Parinfer is a proof-of-concept editor mode for Lisp programming languages. It simplifies the way we write Lisp by auto-adjusting parens when indentation changes and vice versa. The hope is to make basic Lisp-editing easier for newcomers and experts alike, while still allowing existing plugins like Paredit to satisfy the need for more advanced operations.

npm package

This library is published on npm under the package name parinfer

## using npm
npm install parinfer

## using yarn
yarn add parinfer

Usage

Parinfer consists of a couple pure functions of your text, returning new text with corrected parens or indentation.

// 'parinfer' is a global object if not used as Node module.
var parinfer = require('parinfer');

// Run Indent Mode on the given text:
var result = parinfer.indentMode("(def foo [a b");
console.log(result.text);
// prints:
// (def foo [a b])

// Run Paren Mode on the given text:
var result = parinfer.parenMode("(def foo\n[a b\nc])");
console.log(result.text);
// prints:
// (def foo
//  [a b
//   c])

Integrating with an Editor or REPL

See integrating.md

API

smartMode(text[, options])

indentMode(text[, options])

parenMode(text[, options])

Runs Indent Mode or Paren Mode on the given text. Smart Mode is currently something in between.

Arguments:

• text is the full text input.
• options is an object with the following properties:
  • commentChars - a character (ie: string of length 1) or array of characters that should be considered comments in the code (defaults to [";"])
  • openParenChars - array of characters for open-parentheses (defaults to ["(","[","{"])
  • closeParenChars - array of characters for close-parentheses (defaults to [")","]","}"])
  • cursorLine - zero-based line number of the cursor
  • cursorX - zero-based x-position of the cursor
  • prevCursorLine and prevCursorX is required by Smart Mode (previous cursor position)
  • selectionStartLine - first line of the current selection
  • changes - ordered array of change objects with the following:
    • lineNo - starting line number of the change
    • x - starting x of the change
    • oldText - original text that was replaced
    • newText - new text that replaced the original text
  • forceBalance - employ the aggressive paren-balancing rules from v1 (defaults to false)
  • partialResult - return partially processed text/cursor if an error occurs (defaults to false)

Returns an object with the following properties:

• success is a boolean indicating if the input was properly formatted enough to create a valid result
• text is the full text output (if success is false, returns original text unless partialResult is enabled)
• cursorX/cursorLine is the new position of the cursor (since parinfer may shift it around)
• error is an object populated if success is false:
  • name is the name of the error, which will be any of the following:
    • "quote-danger"
    • "eol-backslash"
    • "unclosed-quote"
    • "unclosed-paren"
    • "unmatched-close-paren"
    • "unhandled"
  • message is a message describing the error
  • lineNo is a zero-based line number where the error occurred
  • x is a zero-based column where the error occurred
  • extra has lineNo and x of open-paren for unmatched-close-paren
• tabStops is an array of objects representing Tab stops, which is populated if a cursor position or selection is supplied. We identify tab stops at relevant open-parens, and supply the following extra information so you may compute extra tab stops for one-space or two-space indentation conventions based on the type of open-paren.
  • x is a zero-based x-position of the tab stop
  • argX position of the first argument after x (e.g. position of bar in (foo bar)
  • lineNo is a zero-based line number of the open-paren responsible for the tab stop
  • ch is the character of the open-paren responsible for the tab stop (e.g. (,[,{)
• parenTrails is an array of object representing the Paren Trails at the end of each line that Parinfer may move
  • lineNo is a zero-based line number
  • startX is a zero-based x-position of the first close-paren
  • endX is a zero-based x-position after the last close-paren

Test API

You can use our testing API for a fast, visual way to specify options and verify results. This allows all metadata required by and returned from Parinfer to be specified inside the text using our annotation syntax.

See here for Annotation Syntax details

// Currently only supported in Node
var parinferTest = require('parinfer/test');

Test Example

The following code is a quick way to verify behavior of Indent Mode. The | is parsed as the cursor and removed from the text before processing.

parinterTest.indentMode(`
(def foo
  "|
  "(a b)
      c")
`);

This returns the processed text below, with | reinserted to show cursor result, and an ^ error annotation line since a string was not closed:

(def foo
  "|
  "(a b)
      c")
       ^ error: unclosed-quote

Test Usage

parinferTest.smartMode(inputText, extras); // returns string
parinferTest.indentMode(inputText, extras); // returns string
parinferTest.parenMode(inputText, extras);  // returns string

extras allows us to specify options for which there is no annotation syntax yet:

• forceBalance
• partialResult
• printTabStops

You can also use the input/output functions directly:

parinferTest.parseInput(inputText, extras); // returns {text, options}
parinferTest.parseOutput(inputText, extras); // returns result

parinferTest.printOutput(result, extras);   // returns string

// `result` is returned by main indentMode or parenMode functions

Development

Code: parinfer.js is implemented in ECMAScript 5 for easy speed and portability. Also:

Documentation: Code is documented in code.md.

Performance: To run a performance stress test:

node test/perf.js

Testing: See test/cases/ directory for testing details. Or just run the following:

npm install
npm test

A stable core for editor plugins

Want to use Parinfer on a team? Introduce Parlinter as your project's linter!

The behavior and implementation of the Parinfer library is stable and canonicalized. To allow different editors to use it, we have ported the implementation to the languages required by the plugin APIs of most major text editors. All language ports pass the same comprehensive test suite to help ensure consistent behavior.

implemented in	link	relevant editor
JavaScript	parinfer.js (here)	Atom, VSCode, LightTable
Rust	parinfer-rust	Vim
Python	parinfer.py	Sublime Text
Kotlin (JVM)	parinfer-jvm	Cursive IDE, Nightcode
Emacs Lisp	parinfer-elisp	Emacs
Vim Script	parinfer-viml	Vim
Lua	parinfer-lua	TextAdept

Status Update 2019 (Smart Mode)

Smart Mode (available in [demo]) was an experiment to eliminate switching between Indent Mode and Paren Mode—by looking at a change and determining whether to run Indent Mode or Paren Mode. It is well tested and worked great in our sandboxes, but we found that the majority of editor APIs do not allow us to integrate Smart Mode's rules safely.

For example, if we don't catch a search/replace change in multiple locations of your document, but we infer from the next typing operation that we should run Indent Mode, then Smart Mode will make its decision without knowing the previous search/replace operation took place—thereby breaking its promise of choosing the best mode, and unsafely modifying your code.

The larger problem is that Smart Mode requires the synchronous interception of every type of change coming from the editor. It must decide the right thing to do for input changes at single/multiple cursors, search/replace, copy/paste, advanced macro operations, buffer refreshes from changes on disk, and maybe some others we haven't thought of yet. The interface for receiving these kinds of changes from the editor are not consistent—they either come in asynchronously or sychronously or not at all. This forces us to resort to computing diffs, a lossy mapping from changes to patches.

We have made separate attempts to implement Smart Mode in Cursive, Vim, Atom, and Emacs through some wrangling that made integration very difficult and delicate, and ultimately incomplete. Editors simply are not yet designed to allow an ideal version of Parinfer to exist—probably because nothing like Parinfer has demanded them before. The practicality of requesting these (likely non-trivial) changes on the editor is to be determined.

License

MIT License

parinfer.js's Issues

exposing API to JS

need a way to expose parinfer-lib to JS. @oakmac, could you reiterate your thoughts on this?

only skip the cursor forward for semicolon

I only want to do two things with the cursor:

• do not move it if an unmatched right-paren was attempted
• move cursor to inserted semicolon if commenting out inferred parens

multiple editors not working

possible that we need to share a cm instance:

• second editor allows normal text input (without formatting correctly)
• first editor does not allow moving of the cursor, inserting text at front (appending correct formatting)

Problem with quoted semicolons

I've been working with hiccup this week and I found a problem in insert mode that keeps biting me. I'm not sure if #49 is describing existing bugs or just trying to find a better way of handling this stuff, but I wanted to raise an issue for this because it's coming up so often in my codebase.

Here's the test-case

in

(for [col columns]
  "|
  [:div.td {:style "max-width: 500px;"}])

out

(for [col columns]
  "
  [:div.td {:style "max-width: 500px;"}])

The test error is

expected----------------------------------
(for [col columns]
  "
  [:div.td {:style "max-width: 500px;"}])
actual------------------------------------
(for [col columns]
  "
  [:div.td {:style "max-width: 500px);"}])
expected: "(for [col columns]\n  \"\n  [:div.td {:style \"max-width: 500px;\"}])"
  actual: "(for [col columns]\n  \"\n  [:div.td {:style \"max-width: 500px);\"}])"
    diff: - "(for [col columns]\n  \"\n  [:div.td {:style \"max-width: 500px;\"}])"
          + "(for [col columns]\n  \"\n  [:div.td {:style \"max-width: 500px);\"}])"

new sections needed

• usage for single line REPLs
• pasting code (indent mode and paren mode)
• how paren-mode preserves indentation
• where parinfer breaks down

Paren Mode cursor courtesy

needs to hold parens on the right side of the cursor

stop stripping whitespace at end of file

License

It would be nice to know the license for this through a LICENSE file or at least in the README.md. If there's a license in the source I haven't found it yet.

return cursor position

the API may have to return the corrected cursor position under the following circumstances:

• not allowed to insert paren
• not allowed to remove paren
• commenting an inferred paren

we can add this to the test case output blocks with a pipe |.

animate/edit buttons for each animated editor

allow typing unmatched delims

I think this will be good for giving the user freedom to create a closing paren before an open paren if they want. We can just highlight it red until they match it, like editors will commonly do.

special cases for repositioning cursor

if change is insertion, place cursor after the first occurrence of that character starting at the previous cursor position

• prevents advancing cursor forward when inserting bad delimiters
• prevents backshifting when inserting a semicolon behind a trailing delimiter

space and shift+space

when multiple lines are selected:

• space - indent one space
• shift+space - dedent one space

strange behavior when pressing Tab

only advances the cursor one character, but inserts two spaces ahead of it

don't remove delims on editing line

consider escape state when testing chars

\,))

is being transformed to this

\)),

whitespace? function doesn't consider that \, is a literal character not the whitespace comma.

problem is that escape state can be checked prior to push-char, but not after.

need a lot of ecape char tests

multiple cursors

supporting this can have an effect on #39, in that we might have to just handle changes sequentially

Separate Parinfer as a library

Starting this issue to track separation of "Parinfer the library" from "Parinfer the website".

Specifically, I'm looking for a library version of Parinfer in order to create an Atom editor plugin that will allow it's use.

editors waiting on this:

• Replete
• Light Table
• Atom
• Gorilla REPL
• Neovim

Unit tests

State caching for performance

Editor is slow for large enough files. Need to cache the state at each line to reduce unnecessary work.

tab and shift+tab should move between indent levels

(defn foo
  [a b
  (let [x (+ a b
    x

( (    [  (

x should be able to quickly move between the valid indentation levels listed above.

parinfer should be consistent about how it does or doesn't remove whitespace

we currently don't remove whitespace here, represented as an underscore:

• input: (def b_)
• output: (def b)_

but we do remove whitespace between close-parens:

• input: (def b []_)
• output: (def b []) <-- whitespace removed

Need to be consistent about this

Dim trailing delims

this will signal to the user that these are not in stone, and can be reposition by indented lines

will have to implement a new clojure mode in CodeMirror to highlight them differently

keeping exprs intact

There seems to a necessity for a separate mode when performing certain edits. Given this:

(defn foo [a b]
  (+ a b))

Prepending the first line with an ignore form macro in infer-mode, gives this:

#_(defn foo [a b])
  (+ a b)

whereas, we want this instead:

#_(defn foo [a b]
    (+ a b))

I think this is the job for the prep function during live editing. We will have to extend the prep function to accept a cursor position and cursor delta (maybe just previous cursor position to compute delta). The delta is required so we can preserve relative indentation of expressions after the cursor (we can just add the cursor delta to the indent-delta line data at the cursor). This could get tricky for selection-based editing. Just keep it simple with cursor for now.

highlight diffs in prep mode 'how' section

we can use js diff, tried here:

unblock when cursor is in a comment

bugs

This:

((a ) ;

Produces:

((a  ;))

Add a sockets API?

I'd be interested in making a Parinfer bundle for Howl, but Howl's written in MoonScript and can't interface easily with Clojure. Would it be possible to have a sockets-based API to allow other editors to access Parinfer?

Paren Mode format-text-change

Inserting a quote can cause loss of information

If I have the following string:

"]"

And I insert a quote at the beginning of this line, I end up with this:

"""

This happens because it considers ] to be code now. And since we disallow unmatched closing delims, it is erased.

I am not sure how to solve this problem:

• we can't use the mechanics of indentation to auto-wrap/splice like we can with parens
• even if we could, placing a quote before another completely changes the meaning of the latter, which can erase previously fine characters
• we could restrict quotes to be inserted in pairs
  • would have to restrict deleting in pairs too
  • would have to auto-correct pasted code

introducing this extra complexity may result in the antithesis of my objective for an approachable lisp.

throw away quick-look anim for something better

don't use the editor for the quick look. it doesn't communicate the essence of the idea. we need to show the parens actually moving with css transitions. We can do that by scripting a simple css animation.

proposals for better quote-handling

I waited too long to write these down and it's not fresh on my mind anymore, but here are some suggestions from clojure conj folks:

• @SVMBrown suggests a new two-step process. Upon loading a file, fix unclosed quotes in all comments. And then enforce all subsequent additions of quotes in comments are balanced or escaped.
• @jteneycke suggests explicitly treating the jsdoc comments as special and untouchable (since escaping quotes would result in invalid jsdocs), while correcting all other comments.

But I'm now recalling a goal which seems to prevent these from working: we want to preserve reversibility. Imagine commenting and uncommenting a line of code. In my mind, the resulting line shouldn't be different (i.e. 1 and 4 below should be the same):

1. We have a line with a quote:

   "foo

2. We comment it out:

   ; "foo

3. Quote fixer escapes it

   ; \"foo

4. Uncomment

   \"foo

We also talked about how the source of the problem is that quotes are non-directional, and how their meaning is dependent on the number of quotes before it, allowing strings to be turned inside out when parity changes. svmbrown proposed that we imagine how chaotic this would be if parens were non-directional.

new plugin: emacs-parinfer

Hi Shaun, I've decided this will be a great way to work on my elisp chops so I'm working on emacs-parinfer which is planned to be a minor mode for emacs. Depending on how much work I get done over the coming holiday I might be at a point of making code public in a week or so. Thanks for all the work you have done so far!
Roger

multi-line indent is broken (codemirror)

just a code mirror integration problem, not the algorithm

Support idiomatic Clojure style

In the first example [1], change the :div to a :span:

(defn component []
  (html)
  [:span {:style {:background "#FFF"
                   :color "#000"}}]
  [:h1 "title"])

It is now difficult to get the keywords to line up in the map literal.

[1] https://shaunlebron.github.io/parinfer/

new plugin for Visual Studio Code

Looks like someone's working on an extension for Visual Studio Code. I'm not quite sure how to make it work yet, though.

strip trailing delimiters on current line after cursor

It's a bit strange to dedent this line and not see the change until leaving the line:

 <[:div]
   [:h1 "title"]

Since we only prevent stripping delimiters on the current line to allow us to insert tokens after them, I think we only need to prevent stripping ones appearing after the cursor.

unified change api for both modes

We can describe all text changes with this model: https://github.com/shaunlebron/parinfer/blob/master/test-cases/text-change.md

but right now, the parinfer api requires us to specify changes differently depending on the mode:

• indent mode - line numbers to replace, new lines to insert
• paren mode - specify a cursor-dx position indicating how far the cursor has moved

this requires a lot of manual work on the api user, so if we simply accept a change object described by the above link (which I think most editors expose on change events), we can offload this work to the modes themselves.

This should be done before #24 (creating the standalone parinfer-lib)

backslash in comment will cause prep to delete everything before it

• https://github.com/clojure/clojurescript/blob/85d41cfbaf46f6b7081003576b91dae368c483a8/src/main/clojure/cljs/js_deps.cljc#L261
• https://github.com/clojure/clojurescript/blob/85d41cfbaf46f6b7081003576b91dae368c483a8/src/main/cljs/cljs/pprint.cljs#L2784

new plugin: neovim

I put together a plugin for neovim, it's super early but I'd love to get feedback or have some people try it out. This is an amazing tool, thank you!

https://github.com/snoe/nvim-parinfer.js

fix publish script to deploy advanced

the publish script somehow pushed out a development build

return only changed lines?

each format function returns the full text, but we may want to return only an index of lines that have changed

indent/dedent should keep selection

faster cutoff for detecting imbalanced strings

right now, process-text-change will process until the end of the file if an imbalanced quote is inserted. we could probably track the number of unescaped quotes (in both code and comment) per line to cutoff processing. not sure though

Move right-parens at start of next line to end of current line

this allows us to type matching right-parens at the start of a line without them disappearing

Emacs

Plugin for Light Table

This would be fantastic for Light Table and all of the Lispy languages our users use.

Light Table uses Code Mirror internally so maybe this isn't too hard.

Add Visual Studio Code extension to the plugins list

Looks like someone from Microsoft is working on a Parinfer extension for Visual Studio Code.

Need to add this to the Parinfer webpage.

process-text-change should handle invalid states better

right now, I think the state returned by process-text-change is unstable if the input is invalid (i.e. unclosed quote).

handling auto-indent

There are two extra features related to auto-indentation that Parinfer users are starting to expect. Parinfer may grow in scope to handle these cases somehow, or we will have to detail how this can be handled external to it:

1. position of cursor after pressing enter (auto-indentation of newlines)
2. auto-indenting a region of text according to specific rules (realigning according to conventions)

Specifically, @snoe and @SevereOverfl0w expect the following from item 1: when pressing enter at the end of the last line:

(defn foo [a b]
  (let [x (+ a b)]
    x))|

They expect the cursor to be at the following position:

(defn foo [a b]
  (let [x (+ a b)]
    x))
   |

This behavior is handled in vim by using a mode called autoindent (seen here), which trivially copies the indentation of the previous line.

The behavior for item 2 can be handled by vim-clojure-static, which defines configurable rules for how to indent forms.

These two modes conflict with each other inside vim, so a solution to this problem must combine them when appropriate.