vugufmt - make formatter for .vugu files about vugu

[erinpentecost]

I started the process to get the change into Go's package, which is going to be a learning experience for me: golang/go#31312

I have some ideas on getting indentation working, and I might tackle it tonight or tomorrow.

[erinpentecost]

I'm going to give this one a shot.

[erinpentecost]

Here's some WIP: https://github.com/erinpentecost/vugufmt

[bradleypeabody]

@erinpentecost Very cool!

A few quick notes:

• Ideally, I think this should be available as a struct+method call, as well as a command line tool. As an example of this, vugugen (https://github.com/vugu/vugu/blob/master/vugugen/vugugen.go) is a command line tool that performs code generation, but it is just a thin wrapper around ParserGoPkg (https://github.com/vugu/vugu/blob/master/parser-go-pkg.go). It's really useful to be able to call it in-process or from the command line depending on the use case.

• I like that you have HTML and CSS formatting in there - I think that would be really useful as it's very common to be pasting around things in HTML files and have the indentation get messed up. There will be some interesting edge cases to sort out but that will be all part of the fun (for example, if you forget to close a tag, if the HTML parser you're using closes it automatically, should it do this? Would that be expected behavior? And it should it be optional?)

• Bear in mind that the .vugu file format is not set in stone. One thing that might be added soon is a top level <script type="application/javascript"> tag for JS to be included on the page. Probably not a huge deal to just add later to vugufmt, but thought I'd mention it.

Definitely let me know if you have questions or want to discuss the design, etc. further.

[erinpentecost]

I finished a big rewrite. The vugufmt package exposes this function as its API:

// RunFmt runs vugufmt on input, and sends a pretty version of
// if out to output. If there is an error, throw away output!
func RunFmt(input io.Reader, output io.Writer, simplify bool) error 

There's a runnable main package in there, too, which mimics most of the arguments of gofmt.

The HTTP parser assumes everything in a vugu file is in the portion of an HTML file. If you add stuff in a , it will currently be thrown away. That's probably not what you want. If you have malformed HTML that the parser can't figure out, it just won't attempt to gofmt the file.

Let me know what you think, and if there are specific changes you want to see.

[bradleypeabody]

Looks like it's coming along nicely!

Here's my feedback:

• Thanks for splitting it up so it's callable from go and a separate command line package. I don't see any reason what you currently have in vugufmt.go, htmlfmt.go and gofmt.go wouldn't live directly in the primary vugu dir as vugufmt.go. I don't want to put too much in that package but what you have so far is fairly compact and with the only exposed API then being vugu.RunFmt, I think it could work nicely. Let me know what you think on that.

• Likewise, the "vugufmt" package that builds to the command could live at github.com/vugu/vugu/cmd/vugufmt. Right now "vugugen" is not in a "cmd" subfolder, but I think it should be moved there.

• If feasible, it should be an option to use [goimports](https://godoc.org/golang.org/x/tools/cmd/goimports). It is a drop-in replacement for gofmt and is really handy when you're quickly editing stuff and commenting things in and out to test, etc.

• Rather than having a single option for simplify, I think there needs to be a FmtOpts struct with some booleans and whatever else on it. This way more options can be added without breaking the public API and it can remain vugu.RunFmt(in,out,opts) err

• Looking through this again, I'm noticing that vugufmt the command has a bunch of stuff in it related to processing multiple files in a directory. Is there a way to move that functionality out to the vugu.RunFmt layer of things? Maybe to a vugu.RunFmtDir...? (Although I notice Go's gofmt does it more like how you have it now. I wonder if there is a specific reason. I still like the idea of having as much as possible accessible directly from Go - I don't see a down side to it.)

Regarding "If you add stuff in ... be thrown away" - if we just error in those cases with something like "XYZ block not allowed/support", that should be fine for the time being.

That said, this is great work and I think will really help the project along. (I took a look at what it would take to make a VSCode plugin and it looks easier than I thought - having vugufmt would be super useful when called from VSCode automatically.)

[erinpentecost]

I like all those suggestions, and I'll submit a PR later to the main project. My only concern with it right now is that the next step is to fork golang.org/x/net/html so I can get actual line and column numbers in the error message, which is not going to be compact.

[bradleypeabody]

Oh, I see, hm...

I definitely I agree the line numbers would be very useful to have for several reasons. (I want to make use of that functionality in the code generator too so it can make more useful and better commented .go files from .vugu).

I was thinking maybe the \ns could be counted inside of text/comment nodes and in attributes but it doesn't handle this common case properly:

<div
  id="blah">

Another approach to consider is: What if you make a proxy for io.Reader that counts lines (just using the number of \n bytes encountered) as it is read from. If you could use the tokenizer, this should allow you to get the line number of each element as you go (I do not believe it reads ahead, although I'm not sure). But I don't know how much additional work is involved if using Tokenizer instead of just html.Parse/ParseFragment.

If there's absolutely no other way then I guess a fork of the HTML parser could be done. In that case my thought is it should live in it's own package be noted that's not intended to be used except by vugu internals (as above the code generator would probably use this if it existed), and some attempt should be made to add line number functionality generically and see if it can be PR'ed back into the original package. It is unfortunate such a simple and useful piece of information is so difficult to get. I imagine the author of the package would be happy to see the additional functionality contributed to it.

But yeah if my suggestion above doesn't work or seems impractical to you, a fork would be the way to go for now so things can move forward.

[erinpentecost]

I started on a fork of the html package, since I wanted attr values to use single quotes instead of double quotes in order to minimize the amount of escaping. Since I'm in there, I might as well modify the Tokenizer itself to track lines and columns.

The API for the package has been updated to use the functional options pattern. Example:

// gofmt
formatter := vugufmt.NewFormatter(vugufmt.UseGoFmt(*simplifyAST))
// goimports
formatter := vugufmt.NewFormatter(vugufmt.UseGoImports)
formatter.Format(filename, bytes.NewReader(src), &resBuff)

This lets us use an arbitrary formatter for different languages inside script blocks, and sets us up for CSS formatting in the future.

[bradleypeabody]

Makes sense on the single quotes, that is an excellent point.

This is looking great. I think it's going to be awesome!

[erinpentecost]

Ok, I got the error line offset working.

I've been messing around with the HTML parser, and it looks like it will silently fail and attempt recovery when you pass in malformed HTML. Then, when I go to render the DOM back to the file, there's potential for the HTML to get even more mangled. I'd rather that the error be reported and for it to be picky than risk this, so that's my next step. I doubt that change will get pulled all the way in to the official repo for the HTML parser, so it's probably best if we moved the forked packages under /internal/ somewhere and leave it at that.

[bradleypeabody]

@erinpentecost Thanks and I 💯% agree. Makes sense to be careful about writing mangled nodes and preferring error rather than senseless output. And yup, sounds like an internal package.

[bradleypeabody]

I think this one can be closed now as the basic functionality is now in place. A new issue can and should be made for further improvements or fixes.

@erinpentecost Let me know if you'll have more time to work on this (e.g. HTML indentation or CSS formatting) and want to discuss anything on it. Thanks again!