mmcloughlin / avo Goto Github PK

Add golint to list of enabled linters.

We are reliant primarily on the examples for testing at this point. It would be good to "stress" the allocator because I would be (pleasantly) surprised if it's bug-free.

• More extensive unit testing
• Integration test under tests/ that (for example) uses the max number of registers of a given kind and confirms the register allocator doesn't fall over
• More specifically, handling of "casting" is a little ugly. It would be good to have a test that stresses this specifically. Perhaps, something that allocates 16 virtual general purpose registers and then accesses all sub-registers. This is clearly a possible allocation, but I could imagine the allocator messing up.
• I'm sure other tests will come to mind

Unit tests right now are trivial:

Creating an issue to record thoughts on moving to Intel XED for the instruction database (started in #22).

Support compiler directives such as go:noescape.

https://groups.google.com/forum/#!topic/golang-nuts/_gKfyReaPmY
https://golang.org/cmd/compile/

TEXT and GLOBL symbols are currently too restrictive. I think we do not correctly support:

• Static text symbols
• Symbols with package references

See:

Related to #37

Summary

Symbols have:

• Name
• Package qualification: what package does it belong to? . is a convenience for current package
• Static flag: is it visible outside this file?
• For arcane reasons . and / are replaced with unicode in asm source code

Name is required. The others are optional and we see all 4 possibilities in the standard library:

no pkg, no static: p256SubInternal
pkg, no static: sync∕atomic·LoadInt32
no pkg, static: _expand_key_192b<>
pkg, static: runtime∕internal∕atomic·kernelcas<>

Examples

The following TEXT symbol names do occur in the standard library:

_expand_key_192b<> (static, no dot)
cmpbody<> (static, no dot)
runtime·sigprofNonGoWrapper<> (static, package qualified)
runtime∕internal∕atomic·kernelcas<> (static, package qualified)
sync∕atomic·LoadInt32 (package qualified)
p256SubInternal (no dot)

Static TEXT Symbols

https://github.com/golang/go/blob/fd752d5ede482cdf52a920c75486677cbcb441b0/src/crypto/aes/asm_amd64.s#L243
https://github.com/golang/go/blob/master/src/internal/bytealg/compare_amd64.s#L30

Almost always these do not include the dot symbol, but there are some weird exceptions.

https://github.com/golang/go/blob/14560da7e469aff46a6f1270ce84204bbd6ffdb3/src/runtime/sys_linux_ppc64x.s#L420
https://github.com/golang/go/blob/14560da7e469aff46a6f1270ce84204bbd6ffdb3/src/runtime/internal/atomic/sys_linux_arm.s#L34

Package-Qualified TEXT Symbols

https://github.com/golang/go/blob/14560da7e469aff46a6f1270ce84204bbd6ffdb3/src/runtime/race_amd64.s#L204

Note this is the unicode "division slash U+2215"

Missing a dot

https://github.com/golang/go/blob/14560da7e469aff46a6f1270ce84204bbd6ffdb3/src/crypto/elliptic/p256_asm_amd64.s#L1313

Processing Code

Assembler lexer:

https://github.com/golang/go/blob/fd752d5ede482cdf52a920c75486677cbcb441b0/src/cmd/asm/internal/lex/lex.go#L104-L114
https://github.com/golang/go/blob/fd752d5ede482cdf52a920c75486677cbcb441b0/src/cmd/asm/internal/lex/tokenizer.go#L52-L67

• The unicode characters are mapped directly to standard . and /
• Leading unicode dot is mapped to "".

Assembler:

https://github.com/golang/go/blob/fd752d5ede482cdf52a920c75486677cbcb441b0/src/cmd/asm/internal/asm/asm.go#L79-L90
https://github.com/golang/go/blob/fd752d5ede482cdf52a920c75486677cbcb441b0/src/cmd/internal/obj/plist.go#L80-L82

• Must be NAME_EXTERN or NAME_STATIC

asmdecl pass

https://github.com/golang/tools/blob/3ef68632349c4eab68426d81d981d131b625cafc/go/analysis/passes/asmdecl/asmdecl.go#L277-L279

• Function names not containing <> must have Go declarations

For complexity reasons AVX-512 was not initially considered. We should add support.

I don't think the ast.go types make much sense in the root avo directory. Consider moving them to a sub-package github.com/mmcloughlin/avo/ir.

Preference for ir (intermediate representation) since these types contain much more than a representation of the syntax, for example the results of liveness analysis or register allocation. Moreover I have a half-baked plan to write a parser for a superset of Go assembly into avo formats, so in that world ast may make more sense as a target for the parser (which is then transformed to ir).

• (root)
• build
• buildtags
• gotypes
• operand
• pass
• printer
• reg
• src
• x86
• internal/cmd/avogen
• internal/gen
• internal/inst
• internal/load
• internal/opcodescsv
• internal/opcodesxml
• internal/prnt
• internal/stack
• internal/test
• package-level doc comments

Once done, enable public doc linting. Unfortunately this will require a workaround for golangci/golangci-lint#21.

In a previous life, I did a z80 disassembler that generated almost correct c for the "Small C" compiler. The combination could round-trip with a little human help, and I found it far clearer than disassembling to raw assembler notation.
This language might be an elegant disassembler target.

GLOBL should not construct a static symbol.

Replace for example:

The go assembler supports both of the following instructions but avo does not.

• ANDQ imm64, r64
• MOVD r32, xmm

I see that x86/zctors.go is autogenerated, but I'm a bit lost on how to update the generator to resolve this. To make sure that valid output would be generated I modified it by hand and the generated code was properly assembled:

diff --git a/x86/zctors.go b/x86/zctors.go
index 6d03480..f52ba55 100644
--- a/x86/zctors.go
+++ b/x86/zctors.go
@@ -1280,6 +1280,7 @@ func ANDPS(mx, x operand.Op) (*intrep.Instruction, error) {
 // 	ANDQ imm32 rax
 // 	ANDQ imm8  r64
 // 	ANDQ imm32 r64
+// 	ANDQ imm64 r64
 // 	ANDQ r64   r64
 // 	ANDQ m64   r64
 // 	ANDQ imm8  m64
@@ -1308,6 +1309,13 @@ func ANDQ(imr, mr operand.Op) (*intrep.Instruction, error) {
 			Inputs:   []operand.Op{mr},
 			Outputs:  []operand.Op{mr},
 		}, nil
+	case operand.IsIMM64(imr) && operand.IsR64(mr):
+		return &intrep.Instruction{
+			Opcode:   "ANDQ",
+			Operands: []operand.Op{imr, mr},
+			Inputs:   []operand.Op{mr},
+			Outputs:  []operand.Op{mr},
+		}, nil
 	case operand.IsR64(imr) && operand.IsR64(mr):
 		return &intrep.Instruction{
 			Opcode:   "ANDQ",
@@ -8450,6 +8458,7 @@ func MOVBWZX(mr, r operand.Op) (*intrep.Instruction, error) {
 // 	MOVD imm32 m64
 // 	MOVD r64   m64
 // 	MOVD xmm   r64
+// 	MOVD r32   xmm
 // 	MOVD r64   xmm
 // 	MOVD xmm   xmm
 // 	MOVD m64   xmm
@@ -8505,6 +8514,13 @@ func MOVD(imrx, mrx operand.Op) (*intrep.Instruction, error) {
 			Inputs:   []operand.Op{imrx},
 			Outputs:  []operand.Op{mrx},
 		}, nil
+	case operand.IsR32(imrx) && operand.IsXMM(mrx):
+		return &intrep.Instruction{
+			Opcode:   "MOVD",
+			Operands: []operand.Op{imrx, mrx},
+			Inputs:   []operand.Op{imrx},
+			Outputs:  []operand.Op{mrx},
+		}, nil
 	case operand.IsR64(imrx) && operand.IsXMM(mrx):
 		return &intrep.Instruction{
 			Opcode:   "MOVD",

Run the asmdecl analysis pass on all generated assembly to ensure we are referencing parameters correctly.

https://godoc.org/golang.org/x/tools/go/analysis/passes/asmdecl

klauspost/asmfmt is the de facto standard for Go assembly formatting. It would be good to produce output that conforms to asmfmt.

I have a preference for avoiding non-Go dependencies (stdlib and sub-repos only). Therefore:

• If it is possible to produce conforming output without actually depending on asmfmt explicitly, that would be preferred. This may actually be possible since (at the time of writing) many of the asmfmt rules simply wouldn't apply to avo output.
• If the rules enforced by asmfmt are too complicated, then we can accept the additional dependency.

Either way, it would be good to have a check in CI to confirm that all generated files are formatted correctly. Something like find . -name '*.s' | xargs asmfmt -w and check the git repo is clean.

The v in GP64v() and others is ugly. I no longer remember why this was necessary. Consider removing it.

Consider porting existing PeachPy Go projects to avo. At a minimum this would be really valuable feedback and system-level testing for avo. These could be committed to the avo examples directory or potentially committed back to the original repos if their maintainers are interested.

Our linter is passing but we're getting errors on goreportcard at the moment:

https://goreportcard.com/report/github.com/mmcloughlin/avo

These are all golint errors:

$ golint ./...
examples/args/args.go:3:6: exported type Struct should have comment or be unexported
examples/args/args.go:18:6: exported type Sub should have comment or be unexported
examples/returns/returns.go:3:6: exported type Struct should have comment or be unexported
examples/sha1/sha1.go:8:2: exported const Size should have comment (or a comment on this block) or be unexported
examples/sha1/sha1.go:12:1: exported function Sum should have comment or be unexported
examples/stadtx/stadtx.go:30:1: exported function SeedState should have comment or be unexported
examples/stadtx/stadtx.go:64:6: exported type State should have comment or be unexported

It appears that golangci-lint excludes examples directories by default.

https://github.com/golangci/golangci-lint/blob/3345c7136f58f042f8fa952a58eec0d8ff4f02c5/pkg/packages/skip.go#L22

I would like to enable linting of examples directory and fix any issues.

We currently always include textflag.h

We should be able to write a simple pass which uses ContainsTextFlags() to determine if a given file needs the include or not.

Change this to Label and make it match the corresponding function in Context.

Related to #33 (we should keep the context and package-level functions in sync)

It would be great to support a "debug" print mode. This would display all metadata from the AST, including the results of passes. See PeachPy format_instructions for example.

Use function/data attributes #2 in a new or existing example.

Currently the build package consists of a Context struct with methods for incrementally building a program. Then for convenience, we have a global Context object and package-level functions operating on it. The two have to be kept in sync manually. Ideally this would be automated.

The following gist contains a start of what this could look like

https://gist.github.com/mmcloughlin/2f6ff496978ef57efce13eab84c69cd5

@myitcv suggested that godoc could be used to get most of the way:

I'd say writing a code generator in that case would be overkill. Use 'go doc` on the package to get the methods (via grep) then sed the output to generate the functions.

https://gophers.slack.com/archives/C0VPK4Z5E/p1544342455029800

Some instructions require fixed registers for certain operands. We need to incorporate this into register allocation.

See PeachPy implementation:

https://github.com/Maratyszcza/PeachPy/blob/01d15157a973a4ae16b8046313ddab371ea582db/peachpy/x86_64/function.py#L728-L748

I think it would be better to return something that implements error. Some "multi error" type wrapper similar to https://golang.org/pkg/go/scanner/#ErrorList.

avo errors are currently very hard to understand due to lack of file:line references. We should fix this.

At the same time, it may be good to consider how to make position information more widely available to other components (#6 for example).

(Reuse go/token.Position?)

Remove leftover comments from porting the Stadtx example

#50 showed that some instructions may be missing from the avo database (besides #20).

It would be good to compare our database with the assembler optab:

https://github.com/golang/go/blob/4e8aaf6b22557a485cd3af874bf5be2722b07835/src/cmd/internal/obj/x86/asm6.go#L922

We should support TEXT and GLOBL attributes as defined in textflag.h.

• define attribute type
• add attribute to Function and Global
• define builder interface
• integration test that confirms our attributes are in sync with textflag.h?

Using shippable while the repo is private. Prefer to switch to Travis CI once public.

Add support for referencing symbols in other packages. Include an example of calling an external function or (potentially easier) reading a variable from another package.

Related: #39

Is there a golangci-lint configuration option or another linter that will ensure all packages have package-level doc?

Related to #9

Provide READMEs for examples:

• add
• sum
• args
• returns
• complex
• data
• fnv1a
• dot
• geohash
• sha1
• stadtx
• root examples/ folder: links and summary for each example (maybe auto-generated)

It would be good to support comments in the output. At the moment it is natural to have comments in the generator:

This could just as well be replaced with Comment("Load initial hash") and it would serve to document the generator and the output.

We should provide an example of using build tags (support added in #3).

As part of this it would be good to gather a collection of the sorts of common use cases for build tags (stdlib and popular packages using Go asm). I am not yet clear that the interface introduced in #3 is sufficient.

Liveness analysis should process instructions in topological sort order. Performance is acceptable for now, but this should be addressed at some point.

Measuring unit test coverage is trivial. We should do this anyway.

A lot of the testing at the moment is integration testing (see examples and tests directories). Can we measure the coverage of this testing too?

Some common use cases are a bit ugly right now. Consider adding helpers to make this nicer.

Loading Pointers

Perhaps offer LoadPointer that wraps in a Mem operand for you.

Loading Slices

Perhaps offer a LoadSlice that returns a Slice object with registers in it for base, len, cap? But, then what about the common case where cap is ignored? Seems you might need some mechanism for lazy loading the parameters, depending on which of the subcomponents are actually accessed? Needs more thought.

Working with Labels

It's more annoying that it should be

• For labels defined and then used later (like "loop" above): if Label returned a LabelRef you would not need to repeat the string literal "loop"
• For labels used and defined later (like "done" above)... not sure yet

Block Register Allocations

Allocating an array of registers is fairly common.

In cases like these it could be nice to have a convenience to block allocate registers. So this would be something like GP32s(5).

We depend on Maratyszcza/Opcodes for our instruction database, and indeed include a copy in this repo. Need to check the license and include a copy if necessary.

In the past I have used preprocessor #defines to effectively name registers. See for example:

https://github.com/mmcloughlin/meow/blob/871e50784daff1c9ec5ef66fe3d8fda6bbcd42cf/block_amd64.s#L10-L11

I am wondering if the same could be done in avo with virtual registers (or even physical registers too). This would help make the output more comprehensible, and ease debugging #6.

It should be possible to add build tags to generated files. Questions/thoughts:

• Do we need to support different tags for stubs files?
• Do we want to support some kind of and/or helpers, or just allow raw text?
• Automatically add amd64?

Related: Maratyszcza/PeachPy#75

We should use go modules for dependencies.

Implement a dot product example. This is both an excellent example and helps text feature parity with the same example in PeachPy.

Depends on #1

It would be good to generate a massive test case that covers all the instruction builder functions.

The asmtest generator works by building a huge assembly function with one line for every instruction form.

We could do the same but for the avo builders instead of going straight to assembly.

Following #27 it would be good to track coverage results over time.

Perhaps tackle this at the same time as #26 .

Provide access to interface arguments. See this asmdecl code for the memory layout and expected parameter naming.

https://github.com/golang/tools/blob/d00ac6d27372a4273825635281f2dc360d4be563/go/analysis/passes/asmdecl/asmdecl.go#L449-L455

Should also extend examples/args to demonstrate how to use it.

For example, it is sometimes necessary to access the X sub-register of a Y register. PeachPy supports this through the as_xmm property (see dot product example). avo should have support for this feature.

Need a full README with all the usual sections.

• Title, tagline
• Logo
• Purchase logo icon (https://thenounproject.com/term/avocado/956339)
• Warning about experimental status
• Description
• Installation
• Usage/Quick start: add example
• "Tour" of features: with more links to examples
• Contributing
• License

Other test suites likely provide useful edge cases:

• asmjit tests particularly asmjit_test_x86_cc.cpp
• peachpy x86_64 tests

We currently have U8, U16, ... as well as Imm. I am concerned that Imm is confusing, so perhaps should be removed.

Instruction builders do not have documentation.

We should autogenerate it (same as for constructors).