clj-commons / rewrite-clj

Rewrite Clojure code and edn

A library that reads and writes Clojure, ClojureScript and EDN from Clojure and ClojureScript in a whitespace and comment preserving way.

Status

Rewrite-clj v1 has been successfully and widely adopted.

See project page for current priorities.

Docs

• User Guide

• Developer Guide

• Design

  • Merging rewrite-clj and rewrite-cljs

• FAQ

Used In…​

Some projects using rewrite-clj v1

• ancient-clj 🐥 - Version Metadata Retrieval for Maven Artifacts

• antq 🐥 - Point out your outdated dependencies

• babashka 📍 - Native, fast starting Clojure interpreter for scripting

• carve 🐥 - Carve out the essentials of your Clojure app

• classpath 📍 - Classpath/classloader/deps.edn related utilities

• clerk 🐥 - Local-First Notebooks for Clojure

• cljfmt 🐥 - A tool for formatting Clojure code

• cljstyle 🐥 - A tool for formatting Clojure code

• clojure-lsp 🐥 - Language Server (LSP) for Clojure

• duck-repled 📍 - Transform your REPL interation into Pathom queries

• kit 📍 - Lightweight, modular framework for scalable web development in Clojure

• kusonga 🐥 - Renaming and moving namespaces in Clojure(script)

• refactor-nrepl 🐥 - nREPL middleware to support refactorings in an editor agnostic way

• reval 🐥 -reproduceable eval - namespace eval and storage with browser ui

• rewrite-edn 🐥 - Utility lib on top of rewrite-clj with common operations to update EDN while preserving whitespace and comments

• rich-comment-tests 🐥 - Turns rich comment forms into tests

• test-doc-blocks 🐥 - Test AsciiDoc and CommonMark code blocks found in articles and docstrings

• umschreiben-clj 🐥 - Rewrite utilities for refactoring clojure files

• zprint 🐥 - Executables, uberjar, and library to beautifully format Clojure and Clojurescript source code and s-expressions

Some projects using rewrite-clj v0 and/or rewrite-cljs

• depot 🐥 🩹 - Find newer versions of your dependencies in your deps.edn file

• confair 📍 - A configuration library for Clojure

• gen-fn 📍 - Generate Datomic function literals from regular Clojure namespaces

• kibit 🐥 - There’s a function for that!

• lein-ancient 🐥 - Check your Projects for outdated Dependencies

• mranderson 🐥 - Dependency inlining and shadowing

• mutant 🐥 - Mutation testing for Clojure

• repl-tooling 📍 - a base package for Clojure’s editor tooling

• update-leiningen-dependencies-skill 📍 - Track project.clj dependencies across different projects

Have an update? Let us know!

🐥 canary tested against rewrite-clj v1 lib test suite
🩹 source required minor change to work with rewrite-clj v1
📍 no easy-peasy way to run automated unit tests found for this project

Versioning

Rewrite-clj versioning scheme is: major.minor.release-test-qualifier

• major increments when a non alpha release API has been broken - something, as a rule, we’d like to avoid.

• minor increments to convey significant new features have been added.

• release indicates small changes or bug fixes - starting with v1.1, it is the rewrite-clj release count over the life of rewrite-clj.

• test-qualifier is absent for stable releases. Can be alpha, beta, rc1, etc.

People

A big thank you to all the people who have contributed to rewrite-clj!

Contributors

Founders

Current maintainers

Changes

Licences

We honor the original MIT license from rewrite-clj v0.

Code has been merged/adapted from:

• rewrite-cljs which has an MIT license

• clojure zip which is covered by Eclipse Public License 1.0

rewrite-clj's Issues

Naming of functions

z/append-space and z/append-newline work differently from z/append-child - the first two would be better called z/insert-space and z/insert-newline.

Accessing quoted symbols with zipper

Currently:

(z/sexpr (z/next (z/next (z/of-string "(println 'a)"))))
;(quote a)

There is no way of navigating directly to the symbol "a" with the zipper using next (is there another function I have missed that can do this?).

While it is possible to do something like
(second (z/sexpr (z/next (z/next (z/of-string "(println 'a)")))))
;a

The zipper can never point to "a" so it isn't possible to replace the "a" with another symbol using the zipper. You have to replace the who "(quote a)" and if this is done, you have lost the detail on whether this was represented in the original as either "'a" or (quote a)".

The zipper seems to be trouble when parsing a `ns` form in emacs

I'm developing in emacs:

if I evaluate the form:

(z/of-string "(ns)") => it is fine

(z/of-string "(ns a.name.space)") => it tosses an exception

'cannot find the namespace z'

This is only the case in emacs... it works fine in the repl

Porting the code to clojurescript

How difficult will it be to port the code to clojurescript?

Getting a zipper of an all-comment string returns nil

This is likely an fairly niche case, but it cropped up in some of my unit tests for cljfmt.

(z/edn (p/parse-string-all ";foo"))
=> nil

How to do a prewalk of all nodes?

Hi,

great library, I'm having fun playing with it.

I'm struggling though with prewalk/postwalk. I would have expected it to traverse all nodes of a given zipper, but it actually only traverses the first node.

e.g. in https://github.com/xsc/rewrite-clj/blob/f515767e4f07f27389ee2885cb459bd87d903887/test/rewrite_clj/zip/walk_test.clj , if I make this change:

I'd expect each test case there to change the [5 6 7] to [6 7 8]. But that doesn't happen, because [5 6 7] is not the first node.

I can make it work by adding:

...but that's cheating - I want a generalized walk fn, that doesn't need fixed knowledge about the input - just like clojure.walk.

Is that possible?

Thanks - Victor

Unused node types/dead code

All of the node types in indent.clj are never produced by the parser, and the constructors aren't imported or exposed through the rewrite-clj.node API. Should I remove them?

Also, integer-node constructor is never called by the parser, although this one is part of the public API. I'd love to remove it. I suppose it is possible someone is inserting it into the parse tree and that it works for positive or base-ten numbers. If we want to be conservative, then I think we should deprecate it instead of remove it. (But ugh. :/)

Thoughts?

Feature request: Provide start-end col/line for nodes

It seems currently only row and col are available ? Am I right.

Even better would be a feature to find node (of type) by position
Something like:

(-> root z/down (z/find-by-pos z/next z/list? {:line 15 :col 37}))

similarily find-next-by-pos etc :)

Adding reader information to parsed tokens

I remember raising this with you before. I think I found a solution to this that just want to run it by you.

The 'parse-next' method is a multimethod, so we could write a function that sticks the row and column numbers as metadata in each element that is being parsed.

Instead of using parse-next, the calls are made to a new function 'parse-next-meta' that puts the line information after a call to parse-next.

If you are find with that, I'll start making the changes.

rewrite-clj.zip/sexpr throws on form that begins with reader macro #_

user=> (rewrite-clj.zip/sexpr (rewrite-clj.zip/of-string "#_foo"))

UnsupportedOperationException   rewrite-clj.node.uneval.UnevalNode (uneval.clj:6)

I would expect it to return nil.

Atom integrations?

Hi, and sorry for asking this here ― are there any known Atom integrations for what they call "refactorings" in the Java/IDE world?

Stackoverflow using postwalk on clojurescript/analyzer.cljc and clojurescript/core.cljc

This seems to be purely a function of the size of the file, or the number of forms and not anything in the file in particular as I can remove either the first ~2000lines or the last ~2000lines (at a form boundary) and there's no exception.

Grab the file:

$ curl https://raw.githubusercontent.com/clojure/clojurescript/master/src/main/clojure/cljs/analyzer.cljc -o analyzer.cljc

In the repl:

(-> (slurp "analyzer.cljc")
      (z/of-string)
      (z/up)
      (z/postwalk identity))

Exception:

#error {
 :cause nil
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message "java.lang.StackOverflowError, compiling:(clojure_lsp/parser.clj:729:3)"
   :at [clojure.lang.Compiler load "Compiler.java" 7526]}
  {:type java.lang.StackOverflowError
   :message nil
   :at [clojure.lang.ASeq more "ASeq.java" 131]}]
 :trace
 [[clojure.lang.ASeq more "ASeq.java" 131]
  [clojure.lang.RT more "RT.java" 715]
  [clojure.core$rest__5110 invokeStatic "core.clj" 73]
  [clojure.core$take_while$fn__5638 invoke "core.clj" 2898]
  [clojure.lang.LazySeq sval "LazySeq.java" 40]
  [clojure.lang.LazySeq seq "LazySeq.java" 49]
  [clojure.lang.RT seq "RT.java" 528]
  [clojure.core$seq__5124 invokeStatic "core.clj" 137]
  [clojure.core$take_while$fn__5638 invoke "core.clj" 2896]
  [clojure.lang.LazySeq sval "LazySeq.java" 40]
  [clojure.lang.LazySeq seq "LazySeq.java" 49]
  [clojure.lang.RT seq "RT.java" 528]
  [clojure.core$seq__5124 invokeStatic "core.clj" 137]
  [clojure.core$drop_while$step__5661 invoke "core.clj" 2964]
  [clojure.core$drop_while$fn__5664 invoke "core.clj" 2969]
  [clojure.lang.LazySeq sval "LazySeq.java" 40]
  [clojure.lang.LazySeq seq "LazySeq.java" 49]
  [clojure.lang.LazySeq first "LazySeq.java" 71]
  [clojure.lang.RT first "RT.java" 685]
  [clojure.core$first__5106 invokeStatic "core.clj" 55]
  [clojure.core$first__5106 invoke "core.clj" 55]
  [rewrite_clj.zip.whitespace$skip invokeStatic "whitespace.clj" 28]
  [rewrite_clj.zip.whitespace$skip invoke "whitespace.clj" 21]
  [rewrite_clj.zip.whitespace$skip_whitespace invokeStatic "whitespace.clj" 34]
  [rewrite_clj.zip.whitespace$skip_whitespace invoke "whitespace.clj" 30]
  [rewrite_clj.zip.move$next invokeStatic "move.clj" 32]
  [rewrite_clj.zip.move$next invoke "move.clj" 26]
  [rewrite_clj.zip.walk$postwalk_subtree invokeStatic "walk.clj" 29]
  [rewrite_clj.zip.walk$postwalk_subtree invoke "walk.clj" 27]
  [rewrite_clj.zip.walk$postwalk_subtree invokeStatic "walk.clj" 32]
  [rewrite_clj.zip.walk$postwalk_subtree invoke "walk.clj" 27]
  [rewrite_clj.zip.walk$postwalk_subtree invokeStatic "walk.clj" 32]
  [rewrite_clj.zip.walk$postwalk_subtree invoke "walk.clj" 27]
...

*... last two entries repeat for a few hundreds of lines

Fix parsing of reader conditionals

#?(:clj hello) is currently parsed as :reader-macro ?. That's fine, but the splicing form #?@(:clj hello) gets parsed as :reader-macro ? (:deref ...), which is the wrong meaning.

I think ideally both #? and #?@ would be different nodes altogether, but at the very least #?@ shouldn't be a deref.

z/sexpr on maps with splicing reader conditionals

(rzip/sexpr (rzip/of-string "{:foo :bar #?@(:cljs [:key :val])}"))
This fails with No value supplied for key: (read-string "#?@(:cljs [:key :val])") at this line

Not sure if this is even a bug, or if it is, what the behavior should be. Maybe the solution is don't do that! 😜

Can it find sexps satisfying a predicate?

Hey there!

Not sure if my need is very related to the goal of rewrite-clj, but there is some chance that it can be trivially solved with it.

Given a sexpr, like:

(defn a []
  [[[[[[[(case 2 2 2)]]]] (case 1 1 1)]]])

I'd like to take all the nodes that are a case statement.

e.g. (mystery-fn '(defn a [] [[[[[[[(case 2 2 2)]]]] (case 1 1 1)]]])) gives me ['(case 2 2 2), '(case 1 1 1)]

Can this be easily achieved with rewrite-clj?

Cheers - Victor

The double colon ::keyword throws an error

The double colon keyword is not recognised by the parser.
I ran into the problem and had to avoid parsing it by turning it into a string.

An example of the code that i had to escape is here:

https://github.com/zcaudate/ribol/blob/master/test/midje_doc/ribol_implementation.clj#L100

zip/remove missing an `zip/up` movement somewhere

So this piece of code should return (hello) but instead is returning hello:

 (-> (z/of-string "(hello) (world)")
      (z/right)
      (z/remove)
      (z/sexpr))
=> 'hello

whitespace-aware insertion problems

(-> (z/of-string "1 ")
    z/rightmost*
    (z/insert-right 2)
    z/root-string) ;=> "1  2")

Note the two spaces instead of one in the result. This is a simpler case of trying to insert with proper indentation: I append a newline, append (dec col) spaces, then insert the object, and it's off by one. This seems wrong.

Symbolic Value support

Hi,

Apparently, this should parse (z/of-string "##NaN") but I get java.lang.Exception: :reader-macro node expects 2 values. [at line 1, column 6]

In the wild here: https://github.com/clojure/clojurescript/blob/master/src/main/cljs/cljs/test.cljs#L376

https://clojure.org/guides/weird_characters#__code_code_symbolic_values

Looks like it was added in 1.9.0-alpha20

uneval not handling space between tag and form

I believe #_ (cat) and #_(cat) should be equivalent but currently only the latter currently works.

Multiple metadata tokens don't work

The following test case fails on the latest 2 examples that uses token for metadata. Only the :private metadata is on the sexp.

(tabular
  (fact "about parsing multi metadata"
    (let [s (str ?s " s")
          n (p/parse-string s)
          [mta ws sym] (node/children n)]
      (node/tag n)          => ?t
      (node/string n)       => s
      (node/sexpr n)        => 's
      (meta (node/sexpr n)) => {:private true :amazing true}
      (node/tag mta)        => ?mt
      (node/tag ws)         => :whitespace
      (node/tag sym)        => :token
      (node/sexpr sym)      => 's))
  ?s                   ?t     ?mt
  "^:private ^:amazing"          :meta  :token
  "^{:private true :amazing true}"   :meta  :map
  "#^:private #^:amazing"         :meta* :token
  "#^{:private true :amazing true}"  :meta* :map)

Feature request: zip-to

Hi

The function (zip/zip-to zipper line column), which takes you to whatever is at the coordinates (line, column), in the text consumed by zipper, would be hugely helpful.

If there's nothing there, e.g. the coordinates are to whitespace, or beyond the file content, I'd prefer nil to anything else.

Obviously I'm not married to the name, I'd be equally happy with navigate-to, goto, or a random sequence of letters, as long as it does the job :)

zip/right does not skip over uneval nodes

rewrite-clj.zip/right skips over whitespace, comments, commas, newlines, but does not skip over uneval nodes.

I thought this was quite surprising. Is this intentional?

zip/next never returns nil

Hi

I wanted to generate all nodes in a depth first manner so I did:

(take-while (complement nil?) (iterate zip/next zipper))

but this stream never ends!

After checking the source I realized I had to do:

(take-while (complement zip/end?) (iterate zip/next zipper))

I think it would be nice if zip/next returned nil to indicate that there is no next element so it's consistent with zip/left, zip/right etc which do return nil.

Infinite loop when replacing a symbol with a form that contains that symbol

Here's a minimal demonstration:

(let [r (z/of-string "(str x)")]
  (z/sexpr
    (z/prewalk r
      (comp '#{x} z/sexpr)
      #(z/replace % '(y 1))))) ;; => (str (y 1))

(let [r (z/of-string "(str x)")]
  (z/sexpr
    (z/prewalk r
      (comp '#{x} z/sexpr)
      #(z/replace % '(x 1))))) ;; => Infinite Loop

Exception when trying to parse \\

To reproduce on 0.4.2:

(require '[rewrite-clj.parser :as p])
(p/parse-string "(str \\\\)")

I believe this is a problem with rewrite-clj.parser.token/parse-token and how it uses rewrite-clj.reader/boundary?.

Records are coerced into HashMaps instead of themselves

Hi all,

I have the following defrecord type:

(defrecord Foo [a])

Which is serializable obviously as an EDN tagged literal:

#user/Foo "blah"

Unfortunately coerce coerces IPersistentMap into hash maps, which looses the important type information that this is a #user/Foo. This can be problematic as walking over a tree with a zipper can loose information from the source form.

It might be better if coerce would coerce records into themselves.

<U+2028> causes exception

Somehow had got into my code (using mac notes I think) this caused cljfmt to throw
Exception in thread "main" java.lang.AssertionError: Assert failed: (re-matches #"(\s|,)+" s) and it took me ages to work out why. @weavejester advised me that this is a rewrite-clj issue, see weavejester/cljfmt#51

Proposal for paredit-like functionality

I've been thinking about how to best do source code transforms. It is very difficult without structural editing and so I'm proposing new paredit
Ike features for rewrite-clj

• Auto-aligning sexps on new lines.
• Recalculate line/column positioning
• conversion of comments to code and back again
• support for paredit-like behaviour
  • kill rest
  • slurp and barf
  • wrap parens
  • change depth
  • transpose
  • support for movement to end of line
  • autodetermination of how much space to put in when a new line is created
  • auto new line a partially commented sexp
  • More here http://www.emacswiki.org/emacs/PareditCheatsheet

Multi-line regular expressions are not correctly parsed

Reported in weavejester/cljfmt#76.

user=> (read-string "#\"a\nb\"")
#"a
b"
user=> (require '[rewrite-clj.parser :as p])
nil
user=> (p/parse-string "#\"a\nb\"")
<token: #"a">

radix support not working

So
(parser/parse-string "(def x 2r1101)")
yields:
[:list [:token def] [:whitespace " "] [:token x] [:whitespace " "] [:token 13]]

Which is not quite what you would expect if you were to then write that back into a file.

Parse problems with : inside of symbols

It was pointed out to me in weavejester/cljfmt#21 that the end two sentences of the reader docs on symbols appear to allow :s inside of symbols:

Symbols beginning or ending with ':' are reserved by Clojure. A symbol can contain one or more non-repeating ':'s.

Which means that technically foo:bar is a valid symbol name. Currently rewrite-clj treats that as a symbol followed by a keyword, which is incorrect.

Overflow on new tools.reader

This doesn't happen with tools.reader directly, so I presume it must be something in how this library works.

This happens on tools.reader 1.3.0-alpha3, which is used by clojurescript, hence it gets pulled in. This also happens with 1.0.5 (discovered by accident).

This only happens with z/of-file, not with z/of-string, so I worked around it by using slurp. I don't expect this to be a problem for my use-case.

The file in question is https://github.com/robert-stuttaford/bridge/blob/7c4e0f18b1c64d93f2f56145b8ec16e11295cb71/deps.edn if I comment out some of the :deps keys, it works fine.

Exception in thread "main" java.io.IOException: Pushback buffer overflow
        at java.io.PushbackReader.unread(PushbackReader.java:155)
        at clojure.tools.reader.reader_types$fn__726.invokeStatic(reader_types.clj:170)
        at clojure.tools.reader.reader_types$fn__726.invoke(reader_types.clj:155)
        at clojure.tools.reader.reader_types$fn__643$G__638__650.invoke(reader_types.clj:30)
        at clojure.tools.reader.reader_types.IndexingPushbackReader.unread(reader_types.clj:141)
        at rewrite_clj.parser.keyword$parse_keyword.invokeStatic(keyword.clj:17)
        at rewrite_clj.parser.keyword$parse_keyword.invoke(keyword.clj:8)
        at rewrite_clj.parser.core$eval1934$fn__1935.invoke(core.clj:95)
        at clojure.lang.MultiFn.invoke(MultiFn.java:229)
        at rewrite_clj.reader$read_with_meta.invokeStatic(reader.clj:132)
        at rewrite_clj.reader$read_with_meta.invoke(reader.clj:128)
        at rewrite_clj.parser.core$parse_next.invokeStatic(core.clj:34)
        at rewrite_clj.parser.core$parse_next.invoke(core.clj:32)
        at rewrite_clj.parser.core$parse_delim$fn__1902.invoke(core.clj:42)
        at rewrite_clj.reader$read_repeatedly$fn__1006.invoke(reader.clj:141)
        at clojure.core$repeatedly$fn__6176.invoke(core.clj:5089)
        at clojure.lang.LazySeq.sval(LazySeq.java:40)
        at clojure.lang.LazySeq.seq(LazySeq.java:49)
        at clojure.lang.RT.seq(RT.java:528)
        at clojure.core$seq__5124.invokeStatic(core.clj:137)
        at clojure.core$take_while$fn__5638.invoke(core.clj:2896)
        at clojure.lang.LazySeq.sval(LazySeq.java:40)
        at clojure.lang.LazySeq.seq(LazySeq.java:49)
        at clojure.lang.RT.seq(RT.java:528)
        at clojure.core$seq__5124.invokeStatic(core.clj:137)
        at clojure.core$dorun.invokeStatic(core.clj:3125)
        at clojure.core$doall.invokeStatic(core.clj:3140)
        at clojure.core$doall.invoke(core.clj:3140)
        at rewrite_clj.reader$read_repeatedly.invokeStatic(reader.clj:143)
        at rewrite_clj.reader$read_repeatedly.invoke(reader.clj:137)
        at rewrite_clj.parser.core$parse_delim.invokeStatic(core.clj:43)
        at rewrite_clj.parser.core$parse_delim.invoke(core.clj:38)
        at rewrite_clj.parser.core$eval1978$fn__1979.invoke(core.clj:178)
        at clojure.lang.MultiFn.invoke(MultiFn.java:229)
        at rewrite_clj.reader$read_with_meta.invokeStatic(reader.clj:132)
        at rewrite_clj.reader$read_with_meta.invoke(reader.clj:128)
        at rewrite_clj.parser.core$parse_next.invokeStatic(core.clj:34)
        at rewrite_clj.parser.core$parse_next.invoke(core.clj:32)
        at rewrite_clj.parser$parse.invokeStatic(parser.clj:13)
        at rewrite_clj.parser$parse.invoke(parser.clj:10)
        at rewrite_clj.parser$parse_all$fn__1987.invoke(parser.clj:18)
        at clojure.core$repeatedly$fn__6176.invoke(core.clj:5089)
        at clojure.lang.LazySeq.sval(LazySeq.java:40)
        at clojure.lang.LazySeq.seq(LazySeq.java:49)
        at clojure.lang.RT.seq(RT.java:528)
        at clojure.core$seq__5124.invokeStatic(core.clj:137)
        at clojure.core$take_while$fn__5638.invoke(core.clj:2896)
        at clojure.lang.LazySeq.sval(LazySeq.java:40)
        at clojure.lang.LazySeq.seq(LazySeq.java:49)
        at clojure.lang.RT.seq(RT.java:528)
        at clojure.core$seq__5124.invokeStatic(core.clj:137)
        at clojure.core$dorun.invokeStatic(core.clj:3125)
        at clojure.core$doall.invokeStatic(core.clj:3140)
        at clojure.core$doall.invoke(core.clj:3140)
        at rewrite_clj.parser$parse_all.invokeStatic(parser.clj:20)
        at rewrite_clj.parser$parse_all.invoke(parser.clj:15)
        at rewrite_clj.parser$parse_file_all.invokeStatic(parser.clj:49)
        at rewrite_clj.parser$parse_file_all.invoke(parser.clj:44)
        at rewrite_clj.zip.base$of_file.invokeStatic(base.clj:79)
        at rewrite_clj.zip.base$of_file.invoke(base.clj:75)
        at mach.pack.alpha.inject$_main.invokeStatic(inject.clj:97)
        at mach.pack.alpha.inject$_main.doInvoke(inject.clj:94)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.lang.Var.applyTo(Var.java:702)
        at clojure.core$apply.invokeStatic(core.clj:657)
        at clojure.main$main_opt.invokeStatic(main.clj:317)
        at clojure.main$main_opt.invoke(main.clj:313)
        at clojure.main$main.invokeStatic(main.clj:424)
        at clojure.main$main.doInvoke(main.clj:387)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.lang.Var.applyTo(Var.java:702)
        at clojure.main.main(main.java:37)

Reducing the number of deps

As some development tools might end up depending on rewrite-clj (e.g. cljfmt) it makes sense to keep its deps to the bare minimum (if feasible). I wonder if the dependencies to fast-zip and potemkin are really necessary?

See clojure-emacs/cider-nrepl#148 for some background

namespaced maps in Clojure 1.9 alpha don't work

I'm trying to use rewrite-clj to parse code from Clojure 1.9 alpha, specifically namespaced maps.

The support is described here:

https://groups.google.com/forum/#!topic/clojure-dev/8tHLCm8LpyU

Doesn't seem to work for at least one case, and another case is ... questionable.

user=> (p/parse-string "#::{:a :b :c :d}")

ExceptionInfo Invalid token: :  clojure.core/ex-info (core.clj:4617)

user=> (p/parse-string "#:x{:a :b :c :d}")
<reader-macro: #:x{:a :b :c :d}>

The exception appears to be coming from clojure.tools.reader.

The latest version of clojure.tools.reader says:

Release 1.0.0-beta3 on 25 Jun 2016
Initial implementation of new namespaced maps support
Removed further boxed math
Droped clojure 1.4.0 support

Even if that were to work, I'm not sure that the #:x should really be a reader conditional? Perhaps it should be a map? Still, if these all end up as reader conditionals, I can certainly handle that fine. What I really need is for it to parse the namespaced map into something that I can at least work with and not throw an exception.

When I try this with explicitly loading the 1.0.0-beta3 version of clojure.tools.reader along with
rewrite-clj 0.6.0, it doesn't change anything. The output of lein deps-tree suggests that rewrite-clj is using the later clojure.tools.reader instead of the version it was built with, but I can't say that for sure. I can say it still fails.

I can understand your not wanting to link with a -beta3 version of a library, but it would be nice if it worked if I linked with the -beta3 version of the library. I can't tell if the problem is that I'm not actually getting rewrite-clj to use the new library, or whether it is using the new library but something else is wrong.

Thanks for any help or suggestions that you can provide!

Assertion error for single semicolon comment followed by newline (or nothing)

These two expressions,

(rewrite-clj.parser/parse-string-all "(def a 1);\n")
(rewrite-clj.parser/parse-string-all "(def a 1);")

result in an assertion error AssertionError Assert failed: (re-matches #"[^\r\n]+[\r\n]?" s) rewrite-clj.node.comment/comment-node (comment.clj:25) (the latter expression threw an EOF exception prior to version 0.4.5).

Whereas these,

(rewrite-clj.parser/parse-string-all "(def a 1); ")
(rewrite-clj.parser/parse-string-all "(def a 1); \n")

are parsed successfully.

The problematic expressions are handled by the Clojure reader, so I'm assuming that they're legal and parse-string-all should handle them?

Auto-indentation

Seems like even for my first, small use-case, I would easily benefit from an autoindent "mode" or "interface" (I've manually computed indentation in three different ways, for two different contexts.)

We have this spec:
https://github.com/clojure-emacs/cider/blob/master/doc/Indent-Spec.md

I'm not sure exactly how this would work: I can think of a few possibilities, but the one that seems the most plausible to me at the moment.

1. Add an options map to the whitespace-aware zipper functions.
2. Add options to a) force a linebreak and b) linebreak if the line is longer than some limit instead of inserting a single space.

Weird edge condition with uneval forms

This is a strange one. If you have a uneval form with a newline in the middle of it, then trying to insert a value via a zipper produces a strange result:

(-> "#_(b c)" p/parse-string z/edn z/down z/down (z/insert-left 'a) z/root)
=> <forms: #_[:tag :list][:format-string "(%s)"][:wrap-length 2][:seq-fn #<seq$list_node$fn__14876 rewrite_clj.node.seq$list_node$fn__14876@a798224>][:children (<token: a> <whitespace: " "> <token: b> <whitespace: " "> <token: c>)]>

Note that an eval form works fine:

(-> "#=(b c)" p/parse-string z/edn z/down z/down (z/insert-left 'a) z/root)
=> <forms: #=(a b c)>

My guess is that there's some typo or small error in the uneval form zipper setup that's causing this problem.

Prewalk may miss walking the last element of the list

I'm not sure about this but just wanted to check with you why the prewalk function using my ?-left function does not work as expected.

I should expect one more result: [{:row 4, :col 18} x]
Instead of just 3 [{:row 4, :col 12} 2] [{:row 4, :col 14} 3] [{:row 4, :col 16} 4]

https://github.com/zcaudate/juy/blob/master/src/juy/core.clj#L50-L69

?-right works fine - giving 4 results.

Two-colon keywords cause exceptions

Keywords like ::xyz/foo cause exceptions when read by rewrite-clj:

(p/parse-string-all "(require '[clojure.set :as xyz])\n(prn ::xyz/foo)")

Because xyz isn't a valid namespace alias in the namespace doing the parsing, rewrite-clj will throw an exception when it tries to read in the keyword ::xyz/foo.

You might need to add a specific keyword node, rather than relying on clojure.lang.Keyword.

EOF Exception when parsing file that ends in a comment

A EOF exception is generated when trying to parse a Clojure file that ends in a comment. e.g.

(require '[rewrite-clj.parser :as p])

(p/parse-string-all ";foo")             ;; exception
(p/parse-string-all "(def x 1)\n;foo")  ;; exception
(p/parse-string-all ";foo\n(def x 1)")  ;; ok
(p/parse-string-all ";foo\n")           ;; ok

If the file ends in anything else, including a newline, everything is okay. Perhaps the parser is looking for the end of a comment by searching for the next newline, but doesn't account for an EOF.

inconsistent handling of fn literal v.s. quotes

It would be more uniform if fn literals were handled like

Right now #(+ a b) -> [:fn [:token +] [:token a] [:token b]]

I suggest: [:fn [:list [:token +] [:token a] [:token b]]]

So that it is more consistent with quote forms.

e.g. '(+ a b) -> [:quote [:list [:token +] [:token a] [:token b]]]

Symbols with single quotes at the end aren't read correctly

The Clojure reader docs indicate that a single quote is not character that's allowed in a symbol, however, there's a standard function clojure.core/+' that breaks this rule.

rewrite-clj assumes that a single quote can never appear at the end of a symbol, which leads to valid Clojure code (as defined by the Clojure interpreter) being parsed incorrectly:

(-> (p/parse-string "(+' 1 2)") n/children)
=> (<token: +> <quote: ' 1> <whitespace: " "> <token: 2>)

And an error occurring if such a symbol is the last in a form:

(p/parse-string "(juxt +' -')")
=> Exception expects 1 value. [at line 1, column 13]  rewrite-clj.reader/throw-reader (reader.clj:18)

cannot assoc on childless map

Looks like an empty map needs at least some whitespace for assoc to work:

(def a (z/of-string "{}"))
(z/assoc a :hi "world")
;; => NullPointerException   fast-zip.core/rightmost

(def a (z/of-string "{ }")) ;; (space inserted between curly brackets
(z/assoc a :hi "world")
;; => succeeds

Remove Recursion from Parser

Currently the parser is implemented in a recursive way (for example, parse-next for lists calls parse-delimited which in turn relies on parse-next, passing the end-delimiter as parameter). This can obviously cause problems when parsing deeply nested structures and should thus be replaced by a more stable approach.

Next does not work with empty vector

It looks like next is not progressing past empty vectors correctly:

(z/node (z/next (z/next (z/next (z/next (z/of-string "(defn func []
(println 1))"))))))
;nil

(z/sexpr (z/next (z/next (z/next (z/next (z/of-string "(defn func []
(println 1))"))))))
;Exception: Not a vector: ...

;Works ok if we add another "next":
(z/sexpr (z/next (z/next (z/next (z/next (z/next (z/of-string "(defn
func [] (println 1))")))))))
;(println 1)

Also there is no problem if the vector is not empty

Release to fix z/end? behavior?

I've run into problems with z/end? in z/find which I note is fixed in the master branch in April, but not yet released. Is it possible to cut a "0.3.10" release of rewrite-clj sometime soon?

Incorrect leftmost behavior

The rewrite-clj.zip/leftmost function doesn't appear to work correctly. This may be a bug in fastzip, or in the way rewrite-clj constructs the zipper.

user=> (-> (z/of-string "(merge foo bar)") z/down z/leftmost z/node)
[:token merge]
user=> (-> (z/of-string "(merge foo bar)") z/down z/right z/leftmost z/node)
[:token foo]

These two expressions should return the same value. I've tested this with rewrite-clj 0.3.10, and I've tried updating fastzip to 0.5.1 without success.

Should node/coerce also be implemented for nil?

If I call (node/coerce nil) I get an error: (I've added line-breaks in the error message)

IllegalArgumentException No implementation of method: :coerce of protocol: 
#'rewrite-clj.node.protocols/NodeCoerceable found for class: nil  
clojure.core/-cache-protocol-fn (core_deftype.clj:568)

However, calling (parser/parse-string "nil") returns a node <token: nil>

Is this behavior by design? Would it conflict with anything to extend the NodeCoerceable protocol to nil-values?

Can't read ##Inf, ##-Inf and ##NaN

This works:

(clojure.tools.reader/read-string "##Inf")

but this doesn't:

(rewrite-clj.parser/parse-string "##Inf")

It fails with:

1. Unhandled java.lang.Exception
   :reader-macro node expects 2 values. [at line 1, column 6]

                reader.clj:   18  rewrite-clj.reader/throw-reader
                reader.clj:   11  rewrite-clj.reader/throw-reader
               RestFn.java:  490  clojure.lang.RestFn/invoke
                reader.clj:  157  rewrite-clj.reader/read-n
                reader.clj:  145  rewrite-clj.reader/read-n