Use go/format.

Hi, we are using golex in prometheus instead of the previous hand-written parser and the performance benefit has been amazing.

But we need to support null-bytes \0 in the text being parsed. For example, we need to be able to parse <string>{<string>="<string-with-null-byte>"}

But because we use \0 for EOF, we are not able to do it. For example: <string>{<string>=" results in an infinite loop as we added \0 as a special character for that state here.

Do you have any ideas as to how we can handle null-byte better?

/cc @fabxc

On my 64-bit Intel Linux box, I get the following error messages while attempting to install golex
**$ go get github.com/cznic/golex

github.com/cznic/golex

../mygo/src/github.com/cznic/golex/render.go:33:6: l.YYM undefined (type *lex.L has no field or method YYM)
../mygo/src/github.com/cznic/golex/render.go:234:10: l.YYM undefined (type *lex.L has no field or method YYM)
../mygo/src/github.com/cznic/golex/render.go:255:13: l.YYM undefined (type lex.L has no field or method YYM)
../mygo/src/github.com/cznic/golex/render.go:257:23: l.YYM undefined (type lex.L has no field or method YYM)

Can anyone provide guidance? Am I doing something wrong?
-Pete

golex fails if regex contains '-', which is escaped.

golex test.l
golex: runtime error: invalid memory address or nil pointer dereference

And test.l looks like the following. Note '-' is escaped in xcstart, but apparently golex interprets as character class range from '+' to '*', which is invalid.

%{
package main

func (l *lexer) Lex(lval *yySymType) int {

%}

%x xc

xcstart [\+\-\*]
xcstop \*+\/

other .

%%
{xcstart}
    BEGIN(xc)

<xc>{xcstop}
    BEGIN(INITIAL)

other

%%
}

If you move '-' to the beginning of bracket, if passes.

I think this feature is very usefull.

If %yym is not used, it defaults to "yym" and renders as such. It should not.

Hi,

could you add a small c code example print the AST?

Thanks,
-Gerald

hello,

How set input of golex & goyacc as runes Data type?

lexer.l :

%{
package main
import (
    "bufio"
    "log"
    "strconv"
    "fmt"
)
type yylexer struct{
    src     *bufio.Reader
    buf     []byte
    empty   bool
    current byte
}
func newLexer(src *bufio.Reader) (y *yylexer) {
    fmt.Println(src)
    y = &yylexer{src: src}
    if b, err := src.ReadByte(); err == nil {
        y.current = b
    }
    return
}
func (y *yylexer) getc() byte {
    if y.current != 0 {
        y.buf = append(y.buf, y.current)
    }
    y.current = 0
    if b, err := y.src.ReadByte(); err == nil {
        y.current = b
    }
    return y.current
}
func (y yylexer) Error(e string) {
    log.Fatal(e)
}
func (y *yylexer) Lex(lval *yySymType) int {
    var err error
    c := y.current
    if y.empty {
        c, y.empty = y.getc(), false
    }
%}
%yyc c
%yyn c = y.getc()
D  [0-9]+
E  [eE][-+]?{D}
F  {D}"."{D}?{E}?|{D}{E}?|"."{D}{E}?
%%
    y.buf = y.buf[:0]
[ \t\r]+
{F}
    if lval.value, err = strconv.ParseFloat(string(y.buf), 64); err != nil {
        log.Fatal(err)
    }

    return NUM

%%
    y.empty = true
    return int(c)
}

parser.y file:

%{
package main
import (
    "fmt"
    "math"
)
%}
%union{
    value float64
}
%token	NUM
%left	'-' '+'
%left	'*' '/'
%left	NEG     /* negation--unary minus */
%right	'^'     /* exponentiation */
%type	<value>	NUM, exp
%% /* The grammar follows.  */
input:    /* empty */
        | input line
;
line:     '\n'
        | exp '\n'  { fmt.Printf("\t%.10g\n", $1) }
;

exp:      NUM                { $$ = $1          }
        | exp '+' exp        { $$ = $1 + $3     }
        | exp '-' exp        { $$ = $1 - $3     }
        | exp '*' exp        { $$ = $1 * $3     }
        | exp '/' exp        { $$ = $1 / $3     }
        | '-' exp  %prec NEG { $$ = -$2         }
        | exp '^' exp        { $$ = math.Pow($1, $3) }
        | '(' exp ')'        { $$ = $2;         }
;
%%

main.go file:

package main
import (
    "bufio"
    "os"
)func main() {
	yyParse(newLexer(bufio.NewReader(os.Stdin)))
    os.Exit(0)
}

speed for me in important.
i want use a good data type, like []byte or []rune!
then also can get from file, or inline input.

how fix?
may write a good sample ? or help me?

Hi, Nice to meet you, I am student at University of San Luis Potosi, actually i am programmer in language GOLANG but i have a problem when i try to compile a projet file github.com/cznic/golex/calc because i cant to compile with the command

calc.go:calc.y

double gofmt due to it not being idempotent

go tool yacc -o /dev/stdout $< | gofmt | gofmt > $@

I try with other options to compile but no generate a executable file.

You can to explainme how to other options to compile this file please.

Thanks for you comprension.
Regards

I get this error from shell using the makefile:

golex -t example.l | gofmt > example.go
/bin/sh: 1: golex: not found

Is it possible to get golex/goyacc to parse a document character by character? For context, I am trying to get my parser to ignore whitespace unless we have a newline after a valid statement. For example, assume we have the following file:

a = b

a = 
b

If we parse this by character, we should be able to check after each character if the statement is valid (perhaps if the stack is empty). A hack would be to try parsing more and more lines until we get a valid statement, but this duplicates a lot of parsing. Any ideas?