Unbreak submatching about one-more-re-nightmare

[no-defun-allowed]

I think dab9f14 correctly implements submatching now. It also gave me an excuse to separate code generation and DFA generation, which is nice.

[no-defun-allowed]

We used to break with «a*» but I fixed that. But we still struggle with the nth derivative «a*>* growing in size proportional to 2^n, which shouldn't happen.

[no-defun-allowed]

The almost-last change was to rewrite a + [blah]a → a (or generally any a + b where (remove-tags a) = (remove-tags b)) to prevent the latter RE from growing in size pointlessly, but we have a problem where we optimise too eagerly now!

CL-USER> (one-more-re-nightmare:all-string-matches "«a»|«a»" "aaa")
("a" "a" "a")
(#("a") #("a") #("a"))

While a nice gesture to the programmer to save them some consing from redundant registers, they were probably expecting to have both registers. The parser already maintains a count of registers, which we could use to specify the layout of a correct register vector.

[no-defun-allowed]

At this rate, we probably should encapsulate regular expressions in an object with the information we want, as well as an optimised RE. This would also solve the problem where the group constructor requires an index, as we can't assign indexes automatically. But then that also suggests we should have a second representation which is more literal; we could throw in a S-expression input syntax like cl-ppcre does.

[no-defun-allowed]

Normal matching is broken too! Try to scan for (ab)* on ababaand it will fail, despite that it should match the abab prefix. Baumann also provided a better way to implement grep and what you want instead of backtracking in the paper, so I will go to implement that.

[no-defun-allowed]

I am slowly getting close to having useful machines. The current idea is to use a alpha wrapper expression to record history (recall we don't backtrack with a DFA), and then slam that into a grep machine which clones it for each character. We use tags to record the start and end of a match. After some fiddling, we get a sufficiently small DFA with few statements and assignments:

Some assignments don't look right, but I'm happy to get something this neat at this point.

[no-defun-allowed]

More impressive is the DFA for ab* of course:

If one traces out abc on this machine, one will see the final state failed but remembered the last winning end position.

[no-defun-allowed]

As of 111eb8f the assignments appear correct:

[no-defun-allowed]

The assignments for ab* required more shaking out, but I think we got there in the end?

Perhaps I need a real Graphviz interface somewhere rather than using one-more-re-nightmare::print-dfa and making up a digraph file and fixing up escaping...

[no-defun-allowed]

I got an interpreter working properly for «ab»* which I regard as a big win. So all that is left is to write a code generator.

[no-defun-allowed]

The DFA for a grep for ca«ab»* looks about right. I decided to make graphs using cl-dot which eliminated some hair-pulling to get nice-looking output.

[no-defun-allowed]

The new compiler seems to work, I just have to make it work with the protocol functions.

It would also be useful to generate type-splitted scanners at runtime, as cl-ppcre does. Then we could also lint the RE, detecting pointless expressions and unmatchable subgroups and that sort of thing.

[no-defun-allowed]

All done methinks. As of 8a1b5d2 the compile time is bearable. Might as well merge the new compiler now.