Idiomatic way to distinguish presence of optional nonterminal in AST? about textmapper

[inspirer]

We need an IsValid() method.

Give me a week, and I'll rewrite the templates to produce ready-to-use ASTs. I don't like the current structure with a mixture of interfaces and structs. The traversal code should look like this:

func traverse(n ast.Node/*struct*/) {
   // Value semantics: n == ast.Node{} =>  n.Text() == "" AND n.Child(...) == ast.Node{}, etc.
   if c := n.Child(tm.Package).Child(tm.Name); c.IsValid() {   // such call chains never fail
      report(c.Text())
   }
}

Also, using pointers instead of packing everything into a slice is a bit faster for most compiler-like use-cases.

[mewmew]

Give me a week, and I'll rewrite the templates to produce ready-to-use ASTs.

I would be glad to hear your thoughts during this process, and perhaps have the possibility of giving some input. What will the AST look like (roughly) if it will not look like the current AST?

Will there still be accessor methods on the AST nodes? E.g. X and Y on AddInsts?

https://github.com/mewspring/foo/blob/master/none/ll/ast/ast.go#L1577

func (n AddInst) X() TypeValue
func (n AddInst) Y() Value

This approach has been quite pleasant to use, as a user of TextMapper.

I don't like the current structure with a mixture of interfaces and structs.

The main purpose for interfaces seem to be to support sum types (i.e. union types).

[inspirer]

Type() == NoType should indicate that a node is zero value (see f228765)

[inspirer]

I won't change the wrappers much but the underlying node representation will always be a pointer to the node struct, supporting nils as normal values (for method chains).

[mewmew]

Using NoType it was possible to implement parsing of optional calling conventions. The solution was not as pretty as I would have hoped, so adding it here for future consideration.

The solution looked as follows.

if n := old.Child(selector.CallingConv); n.Type() != ll.NoType {
	i.CallingConv = irCallingConv(ast.ToLlvmNode(n).(ast.CallingConv))
}

This works. I would have liked to write something along the lines of:

// In this case, the call to `irCallingConv` would check if the CallingConv node
// is valid using `IsValid`
i.CallingConv = irCallingConv(old.CallingConv())

Or perhaps:

if oldcc := old.CallingConv(); oldcc.IsValid() {
	i.CallingConv = irCallingConv(oldcc)
}

Edit: Note, both of the latter to examples panic with unknown node type NONE. Which may be caused by #23.

[inspirer]

Please have a look at the ast_experiments branch. If you like it, I'll clean it up and update the templates. You will automatically get two more files (parser.go an tree.go) in the ast/ directory for all parsers that want an AST.

[mewmew]

Please have a look at the ast_experiments branch. If you like it, I'll clean it up and update the templates. You will automatically get two more files (parser.go an tree.go) in the ast/ directory for all parsers that want an AST.

Thanks. That's great. I'll take a look this week. Currently busy with exams :)

[inspirer]

The change is backwards compatible and is now merged into master. To enable AST generation, use eventAST = true option. Also, consider setting fileNode = "File" if you want comments in the AST (see below). This is still work in progress, so any feedback is very welcome.

Consider the following grammar

language a(go);

reportTokens = [comment]
eventBased = true
eventFields = true
eventAST = true

:: lexer
space: /[\n\r\t ]+/
comment: /#.*/
a : /a/

:: parser
input -> Input : a+ ;

Executed on the following input:

## comment
a

The output parsing tree will contain two top-level nodes, since the comment cannot be put under the input node, which covers just a. ASTs with multiple top-level nodes are ugly to work with. With fileNode = "Input", the generated parser will not report Input when reducing input: a+ but at the very end if parsing succeeded, which means the resulting AST will contain exactly one node.

[mewmew]

I have not been able to use IsValid.

The following version works, when checking for nil instead of invoking IsValid.

// irOptAddrSpace returns the IR address space corresponding to the given
// optional AST address space.
func irOptAddrSpace(n *ast.AddrSpace) types.AddrSpace {
	if n == nil {
		return 0
	}
	x := uintLit(n.N())
	return types.AddrSpace(x)
}

While the following version gives the panic presented below.

// irOptAddrSpace returns the IR address space corresponding to the given
// optional AST address space.
func irOptAddrSpace(n *ast.AddrSpace) types.AddrSpace {
	if n.IsValid() {
		return 0
	}
	x := uintLit(n.N())
	return types.AddrSpace(x)
}

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x504512]

goroutine 1 [running]:
github.com/mewmew/l-tm/asm.irOptAddrSpace(0x0, 0x0)

[mewmew]

I updated the LLVM IR repo to use the latest version of TM, and set eventAST = true. This works great in general! Thanks a lot for the work done. I could remove both the parser.go and tree.go that I had originally copied from the TM repo.

For the most part, I've been able to handle optional nonterminals by checking for nil. I've not really been able to get the IsValid method to work, as mentioned above. However, checking n == nil is just fine and can be used instead of having n.IsValid().

The only issue I've really run into, is when using this approach for a combination of nodes with the shared type ast.LlvmNode. Specifically, this is how linkage is modelled in the grammar (I can elaborate on the reasons for this, but basically it boils down to trying to avoid a shift/reduce conflict).

The code for translating optional ExternLinkage and Linkage into corresponding Go enums is as follows.

// irOptLinkage returns the IR linkage corresponding to the given optional AST
// linkage.
func irOptLinkage(n ast.LlvmNode) enum.Linkage {
	if n == nil {
		return enum.LinkageNone
	}
	switch n := n.(type) {
	case *ast.ExternLinkage:
		if n == nil {
			return enum.LinkageNone
		}
	case *ast.Linkage:
		if n == nil {
			return enum.LinkageNone
		}
	}
	return asmenum.LinkageFromString(n.LlvmNode().Text())
}

Notice, that the type switch is required to handle the case where the interface n ast.LlvmNode is non-nil, but it's value is nil (essentially https://golang.org/doc/faq#nil_error).

If I only have the if n == nil check, and skip the type switch, I get the following panic (essentially related to https://golang.org/doc/faq#nil_error).

panic: value method github.com/mewmew/l-tm/asm/ll/ast.ExternLinkage.LlvmNode called using nil *ExternLinkage pointer

goroutine 1 [running]:
github.com/mewmew/l-tm/asm/ll/ast.(*ExternLinkage).LlvmNode(0x0, 0x0)
	<autogenerated>:1 +0x55
github.com/mewmew/l-tm/asm.irOptLinkage(0x5cb4c0, 0x0, 0xc000bd76c8)

If I try to use n.IsValid(), the following panic is presented.

// irOptLinkage returns the IR linkage corresponding to the given optional AST
// linkage.
func irOptLinkage(n ast.LlvmNode) enum.Linkage {
	if n == nil {
		return enum.LinkageNone
	}
	if n.LlvmNode() == nil {
		return enum.LinkageNone
	}
	if n.LlvmNode().IsValid() {
		return enum.LinkageNone
	}
	return asmenum.LinkageFromString(n.LlvmNode().Text())
}

panic: value method github.com/mewmew/l-tm/asm/ll/ast.ExternLinkage.LlvmNode called using nil *ExternLinkage pointer

goroutine 1 [running]:
github.com/mewmew/l-tm/asm/ll/ast.(*ExternLinkage).LlvmNode(0x0, 0x0)
	<autogenerated>:1 +0x55
github.com/mewmew/l-tm/asm.irOptLinkage(0x5cb4e0, 0x0, 0x0)

[inspirer]

Yeah, what you stumbled into is a typical Go pitfall. Somewhere on the way into irOptLinkage() a nil pointer to a struct gets converted to an interface (ast.LlvmNode), so you get a typed nil. It is very likely that you pass a non-interface variable into this function, and it happens when you enter the function. So the answer is: either avoid typed nils at all costs or make them useful.

I have two options for you:
a) Avoid implicit type conversions that might take nil pointer as an input (this is not hard, actually).
b) We change the philosophy of typed wrappers generated by Textmapper and teach them to accept nils (as protocol buffers and the Node struct do).

Textmapper produces two types of ASTs: typed wrappers (LlvmNode) and untyped underlying storage (Node). The latter is useful for unstructured traversals, such as find all methods or identifiers in a file. The Node struct treats nils as non-existing "empty" nodes without a type, and so we can use method chains to traverse the tree nicely.

// This will work fine if there is no IdentifierExpression under n.
if text := n.Child(IdentifierExpression).Child(Identifier).Text(); text != "" {
   // do something.. 
}

Typed wrappers return an interface (untyped) nil when node is not found. This does not work with method chaining. If we want to make it work, methods will have to return typed nils, so you will have to use IsValid() everywhere for node presence instead of nil checks. This is not as intuitive but is much more maintainable in the long term, so I think we should do it.

In practice this means adding one more typed node struct: NilNode, which will be returned instead of the nil value when the type of the returned value is not known.

elseStmt := ifStatement.ElseBranch().Statement()
// post-invariant: elseStmt != nil

In this example, for an absent else branch, ElseBranch() will return &ElseBranch{nil}, and Statement() will return &NilNode{} since there is more than one Statement node that can be present here.

[mewmew]

Yeah, what you stumbled into is a typical Go pitfall. Somewhere on the way into irOptLinkage() a nil pointer to a struct gets converted to an interface (ast.LlvmNode), so you get a typed nil.

Indeed, this is a typical Go pitfall (enough so to warrant a place in the FAQ).

However, I seem to get the panics also for concrete AST struct types, e.g. calling n.IsValid on n *ast.AddrSpace: #19 (comment). Is this the expected behaviour?

The interface issue only occurs at a single position in my code base thus far, so not to worry. However, I'd like to figure out how to use IsValid for the concrete AST types.

[inspirer]

Since d2e9e4d, AST wrappers never return nil and you should check for IsValid() instead. Please have a look and let me know if this works for you. You will need to update your traversal code.

[mewmew]

I get the following error for ll.tm using the latest version of Textmapper.

lalr: 0.236s, text: 0.457s, parser: 2221 states, 228KB
# github.com/llir/ll/ast
ast/factory.go:731:3: cannot use nilInstance (type *nilNode) as type LlvmNode in return argument:
	*nilNode does not implement LlvmNode (missing LlvmNode method)
make: *** [Makefile:9: gen] Error 2

[mewmew]

Given the grammar:

%interface CallingConv;

CallingConv -> CallingConv
	: CallingConvEnum
	| CallingConvInt
;

I've been unable to figure out how to check for IsValid on optional calling conventions.

Attempt 1

Using the following leads to ast.CallingConv has no method IsValid error on compile time.

if n := old.CallingConv(); n.IsValid() {
	new.CallingConv = irCallingConv(n)
}

Attempt 2

Using the following leads to panic: support for calling convention type *ast.nilNode not yet implemented

switch n := n.(type) {
case *ast.CallingConvEnum:
case *ast.CallingConvInt:
default:
	panic(fmt.Errorf("support for calling convention type %T not yet implemented", n))
}

--- FAIL: TestParseFile (0.00s)
panic: support for calling convention type *ast.nilNode not yet implemented [recovered]
	panic: support for calling convention type *ast.nilNode not yet implemented

Attempt 3

Using the following leads to panic: LlvmNode invoked on nil node

if !n.LlvmNode().IsValid() {
	return enum.CallingConvNone
}

--- FAIL: TestParseFile (0.00s)
panic: LlvmNode invoked on nil node [recovered]
	panic: LlvmNode invoked on nil node

[mewmew]

Note, the change to use IsValid instead of nil to check for optional nonterminals was made in rev llir/llvm@8a077fc in the isvalid branch.

Edit: also, without knowing the implementation of nilNode, I added the following to the generated AST of the parser, to fix the compile time error reported in #19 (comment).

+func (nilNode) LlvmNode() *Node {
+       panic("LlvmNode invoked on nil node")
+}

[inspirer]

Sorry, my bad, this is fixed now in 973a0cd.

LlvmNode() should return nil for NilNode. I also made NilNode exported so that you case use it in type switches:

case *ast.NilNode: /*do nothing */
default:
    panic("....")

[mewmew]

Sorry, my bad, this is fixed now in 973a0cd.

No worries.

LlvmNode() should return nil for NilNode. I also made NilNode exported so that you case use it in type switches:

Thanks! I found another solution that I prefer.

	if !n.LlvmNode().IsValid() {
		return enum.CallingConvNone
	}

Now I've updated the llir/llvm package to make use of the latest revision of Textmapper (rev 973a0cd). It works great!

As there is now an idiomatic way to distinguish optional nonterminal, using IsValid, I'll go ahead and close this issue. Thanks for working on it.

[mewmew]

P.S. I have no need of NilNode being exported, as n.LlvmNode().IsValid() may be used for the same effect. So, unless you see a reason to keep it exported, then feel free to unexport it again.