I understand that gb tries to replace go build and go test and replace or avoid go get. But what about the other tools at golang.org/x/tools? godoc, oracle, goimports, callgraph, eg, gomvpkg, gorename, etc. can be very useful and rely on a GOPATH being set. Thanks!

Sorry if it's dumb question, but based on this line from readme:

A project is conceptually a $GOPATH workspace dedicated to your project's code.

Do you see gb vendor as a one time operation to grab dependencies; then use that libs DVCS native update function for the infrequent times when the library needs changes from upstream?

Could gb vendor receive add functionality similar to what go get has? Does it need them?

Is there anything special about the $GOPATH/vendor directory? Could it store non-Go 3rd party libraries? I'm thinking about javascript and css libraries.

Related to #31

gb currently ignores any package it thinks is part of the standard library; it just assumes they are up to date.

This may not be correct in a cross compilation world, where package resolution and isStale calculation could reach all the way back to runtime

Can will need to be taken that package resolution always goes, $GOROOT, $PROJECT/src, $PROJECT/vendor/src-- it should never be possible for$PROJECT/src/errorsfor example, to override$GOROOT/src/errors`.

However, when it comes to package compilation time, we want to cache the results of compiling source from $GOROOT/src in $PROJECT/pkg and this will be tricky.

I want to use Sourcegraph for gb code search, browsing, and usage examples. Can an admin enable Sourcegraph for this repository? Just go to https://sourcegraph.com/github.com/constabulary/gb. (It should only take 30 seconds.)

Thank you!

Following up on https://twitter.com/dimisec/status/595895255834206208 exchange:

dimisec: how can we cover the use case when the vendored code is in different repo, ex. http://3dparty.mycorp.com? Must have a config then?

davecheney: all that gb cares about is the source is in the correct path in vendor/src/
davecheney: I can't answer this in a tweet, could you log an issue and I'll do my best to explain there, with pictures.

Here's the use case I am thinking about. Suppose that your company is using GHE (GitHub Enterprise) and all the third-party / mirrored opensource code lives in a dedicated GHE instance "opensource.git.corp.company.com" separate from the "git.corp.company.com" which is used for all the internal code. Therefore you don't have one single source code tree where you could put vendor/src close to the root. What people want is to use the "natural" package names in the imports such as github.com/robertkrimen/otto, avoid the nuisance of continuous import path rewriting and rely on a building tool that would be able to "magically" map "github.com/abc/xyz" to "[email protected]:abc/xyz.git " so that you could use your central internal mirror (which is reliable and persistent in case the upstream repository disappears).

I wonder if it would be possible to "templatize" the vendor/src location so that the default would be like it is now (for everyone who is happy with gb's status quo) but you would also be able to specify with some optional configuration (for example, ~/.gbconfig file) that instead of vendor/src, the dependencies code should be checked out from [email protected]:[org]/[repo](and maybe added to _vendor/src subdirectory of the currently compiled project for building).

Another part of the equation is reproducible builds and freezing the versions of the dependencies - being able to nail them down without having to duplicate the code of N dependencies for M projects N_M times which seems somewhat space-wasteful and, probably much worse, means that all those N_M mini-mirrors need to be created and updated/managed (yikes! - a very real pain point). If a [light-weight] configuration file would be required for that, I think it's a fair price to pay (especially if we can have tooling to help creating and updating such configuration).

One more wish that I see commonly expressed: to be able to fetch the dependencies with a certain tag. I sorely miss the ability to "go get -tag [sometag]" which would grab the code with the given tag if and where it's available (like Go does with the magical but rather useless "go1" tag) - so that I would not need to do scripting voodoo of: "go get -d [pkg] && cd $GOPATH/pkg && git checkout [sometag] && go get -d ./..." (and then somehow do it recursively for all the deps). I even tried to request such an extension but of course it was quickly rejected 😄

I know it should not build a test file, but it should also not produce this output. It should returning something simple or nothing.

File: example3_test.go

sh gb build -q | done: 124.45275ms ]
FATAL command "build" failed: [/Users/bill/go/pkg/tool/darwin_amd64/6g -p github.com/ArdanStudios/gotraining/09-testing/01-testing/example3 -pack -o /var/folders/8q/d2pfdk_x4qd4__l6gypvzsw40000gn/T/gb966307503/github.com/ArdanStudios/gotraining/09-testing/01-testing/example3.a -I /var/folders/8q/d2pfdk_x4qd4__l6gypvzsw40000gn/T/gb966307503 -I /Users/bill/Spaces/Go/Projects/pkg/darwin/amd64 -complete]: exit status 1
usage: 6g [options] file.go...
-+ compiling runtime
-% debug non-static initializers
-A for bootstrapping, allow 'any' type
-B disable bounds checking
-D path
<cut remaining display of 6g usage>

Hi!

I started to convert a bigger project of mine but I tripped over one of it's dependencies.

This is what I got when I made a new gb project for the dependency:

[cryptix@higgs ~] mkdir devel/cznicZappy
[cryptix@higgs ~] cd devel/cznicZappy 
[cryptix@higgs ~/devel/cznicZappy] mkdir -p src/github.com/cznic/zappy
[cryptix@higgs ~/devel/cznicZappy] git clone -1 https://github.com/cznic/zappy src/github.com/cznic/zappy 
[cryptix@higgs ~/devel/cznicZappy] gb build all
2015/04/29 13:38:59 INFO project root "/home/cryptix/devel/cznicZappy"
2015/04/29 13:38:59 INFO compile github.com/cznic/zappy [decode.go encode.go zappy.go]
2015/04/29 13:38:59 INFO pack [[compile {zappy github.com/cznic/zappy /home/cryptix/devel/cznicZappy/src/github.com/cznic/zappy}]]
2015/04/29 13:38:59 INFO build duration: 4.985918ms map[compile:3.364623ms pack:712.983µs]
2015/04/29 13:38:59 command "build" failed: run: [/home/cryptix/gosrc/pkg/tool/linux_amd64/pack r /tmp/gb213012010/github.com/cznic/zappy.a]: [/home/cryptix/gosrc/pkg/tool/linux_amd64/pack r /tmp/gb213012010/github.com/cznic/zappy.a]: exit status 2
[cryptix@higgs ~/devel/cznicZappy] 

Running the last command by hand gave me this:

[cryptix@higgs /tmp/gb213012010] /home/cryptix/gosrc/pkg/tool/linux_amd64/pack r /tmp/gb213012010/github.com/cznic/zappy.a 
Usage: pack op file.a [name....]
Where op is one of cprtx optionally followed by v for verbose output.
For compatibility with old Go build environments the op string grc is
accepted as a synonym for c.

For more information, run
    godoc cmd/pack
[cryptix@higgs /tmp/gb213012010] 

[cryptix@higgs /tmp/gb213012010] file /tmp/gb213012010/github.com/cznic/zappy.a
/tmp/gb213012010/github.com/cznic/zappy.a: current ar archive

I've added a Debugf to runOut to get the stderr output and pasted the verbose run here.

i really like the idea and approach you have taken. I hope it takes off. Sadly, I missed your talk even though I was in the Berlin at the time...

Just tried to add -R to override the project root and ran into the fact that the flags are not evaluated till after the plugins.

After adding a bit of debugging *goroot, *goos, *goarch will only ever get set to the default values.

https://github.com/constabulary/gb/blob/master/cmd/gb/main.go#L77

vs flags evaluated on

https://github.com/constabulary/gb/blob/master/cmd/gb/main.go#L100

I am disappointed that you are reusing the name of my build tool without letting me know :(

In #50 @ChrisHines pointed out some ambiguity in the project detection rules.

Clarify them, and add tests. Also.come up with a good story for what happens when accidentally building inside vendor/

When performing a normal build all I care about is seeing compiler messages. Everything else is noise.

% gb build
animals.go:17: syntax error: unexpected EOF, expecting } (edited)

Anything else should be provided only with a switch

%db build -v
2015/05/05 18:33:32 command "build" failed: [/Users/bill/go/pkg/tool/darwin_amd64/6g -p animals -pack -o /var/folders/8q/d2pfdk_x4qd4__l6gypvzsw40000gn/T/gb782637558/animals.a -I /var/folders/8q/d2pfdk_x4qd4__l6gypvzsw40000gn/T/gb782637558 -I /Users/bill/gb/test/pkg/darwin/amd64 -complete animals.go]: exit status 1
animals.go:17: syntax error: unexpected EOF, expecting }

Thank you for doing this groundwork, Dave. Having a sane story around dependency management and handling during the build is important to the language and uptake, and gb plays into that.

lucky(~/devel/sftp/src/github.com/pkg) % gb build ./sftp/examples/gsftp                                                                              
2015/04/29 17:06:41 INFO project root "/home/dfc/devel/sftp"
2015/04/29 17:06:41 INFO compile github.com/pkg/sftp/examples/gsftp [main.go]
2015/04/29 17:06:42 INFO link /tmp/gb198604988/github.com/pkg/sftp/examples/main [/tmp/gb198604988/github.com/pkg/sftp/examples/gsftp.a]
2015/04/29 17:06:43 INFO build duration: 1.221536005s map[compile:204.161508ms link:1.009184124s]

Two problems,

1. the temporary shouldn't be called main, it should be the name of the directory, not the name of the package
2. the temporary, if successful, should be moved into $PROJECT_ROOT/bin/

Hi Dave,

I read your slides and gb sounds like a really great project.

But: I'm a little unclear on how the basic vendoring and GOPATH stuff is supposed to work.

Could you outline the theory of operation in the documentation? A getting started guide would be a great help too.

Thanks!

Jason

I know of the persistent advice of "use go install" when "go build" doesn't do what people expect... But "go build" does what I expect, or in certain cases GOBIN=something go install does it better.

I was hoping that gb build would do the right thing (in my opinion) by placing the build artifact in the current directory, so that I can tar it up etc as part of a build process. However in actuality it ends up in ../../../../bin, which is an annoying place to go looking for build artifacts.

Consider this layout

$PROJECT
    src/
        main.go

main.go will be ignored. Similar to #45, this is a bug, but probably not helpful to the user as the command would be installed as $PROJECT/bin/src

Shouldn't gb vendor support a tag / commit option? Otherwise it just does exactly the same thing as go get.

Is there some kind of gb test?

Reading through README.md and getting-started.md, I got a bit confused whether projects are meant to be checked into version control, and how the different directory structure would interact with making a project go getable.

So I thought that since gb is proposing the new structure, and it seems to fit the definition of a project (i.e. it has a package main, and isn't designed to be consumed by other code), it would make sense for the gb repo to be structured as a project itself.

I'm not sure how to go about this though. If I understand correctly, this would only entail moving the code to gb/src. This creates a chicken-and-egg problem, though, since cmd/path.go and cmd/gb/main.go both import github.com/constabulary/gb. When I build with go build, this would now need to be import "github.com/constabulary/gb/src", and building with gb would probably expect the code to live under src/github.com/constabulary/gb, right?

Providing functionality to support library authors (vendoring deps) and library users (building with those deps) would be very useful, and would cover part of the use-case that tools/godep does without the import path re-writing.

• A user can go get github.com/you/yourpackage and pull down the master.
• They can then gb build to use the vendored packages under /vendor instead of their global GOPATH
• Else they can fall back to go build and go install and build off their global GOPATH. This (obviously) may not work, but we can't change the behaviour of go get. This is no different from how things work now with go get and pulling deps from head.
• There may also be benefit in storing the commits/tags in some kind of .gbvendors file (the name is unimportant right now!) as well that makes it clear what revisions were pulled in. Avoiding submodules/subtrees is great (IMO) but knowing what revision the vendored dependency is pinned at still has value.

My library would then look like:

github.com/me/mylib
\
 |- auth.go
 |- auth_test.go
 |- doc.go
 |- vendor/
 \
  |-- .gbvendors
  |-- github.com/someother/somepkg/
  |-- github.com/another/anotherpkg

The -toolchain option in cmd/gb seems to be unused at the moment. Is this intentional?

toolchain = fs.String("toolchain", "gc", "choose go compiler toolchain")

Continuing the conversation from Twitter:

• https://twitter.com/reiver/status/596004239324749824
• https://twitter.com/davecheney/status/596030215924494336

The go command has both go build and go run.

The gb command currently only has gb build but not gb run

Are there any plans of making a run command, as in gb run?

2015/05/06 08:01:07 DEBUG removing work directory: /tmp/gb216867794
--- FAIL: TestBuild (0.22s)
        build_test.go:32: ctx.ResolvePackage(cmd): want <nil>, got no buildable Go source files in /home/dfc/go/src/cmd

If you have the following project structure

$PROJECT
    src/
        cmd/
            main.go

src/cmd/ will be ignored. This is a legacy of reusing the go/build package resolution code where all, std, and cmd are reserved words.

However, you probably don't want to do this, because even if it did work, your command would be installed as $PROJECT/bin/cmd

Would something like the following be useful?

gb ./path/to/src/my/package

or perhaps

gb build -c ./path/to/src/my/package # -c change working dir

I ran into this because I have a project with multiple languages at the root and the go projects are a sub directory, I want to invoke gb from a top level build script.
It may also be nice to specify an alternative output directory.

I am happy to work around the issue if you don't think it is worth it.

I have been experimenting with using gb inside Docker. I am starting from the base Dockerfile that is used by the ManagedVMs Google App Engine SDK, with modifications to retrieve gb (noted below).

During the execution of the final command inside the docker instance, the gb build all exits with the following error, indicating that fork/exec is missing from the Go installation:

2015/05/01 14:11:54 INFO build duration: 10.724911ms map[compile:4.194256ms]
2015/05/01 14:11:54 command "build" failed: run: [/usr/lib/google-golang/pkg/tool/linux_amd64/6g -p github.com/constabulary/gb/cmd/gb -pack -o /tmp/gb172786071/github.com/constabulary/gb/cmd/gb/main.a -I /tmp/gb172786071 -I /go/pkg/linux/amd64 -complete build.go main.go plugin.go test.go]: [/usr/lib/google-golang/pkg/tool/linux_amd64/6g -p github.com/constabulary/gb/cmd/gb -pack -o /tmp/gb172786071/github.com/constabulary/gb/cmd/gb/main.a -I /tmp/gb172786071 -I /go/pkg/linux/amd64 -complete build.go main.go plugin.go test.go]: fork/exec /usr/lib/google-golang/pkg/tool/linux_amd64/6g: no such file or directory

Steps to reproduce

Create a Dockrfile with the following contents in the top level project directory (the directory containing the src, bin, pkg, vendor directories) containing the source for a trivial Go app:

FROM gcr.io/google_appengine/base

RUN apt-get update \
    && apt-get install -y \
       curl gcc git libc6-dev make \
       --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

ENV GOLANG_VERSION 1.4.2
ENV PATH /go/bin:/usr/src/go/bin:$PATH
ENV GOPATH /go

# Download, build and install the specified version of Go:
RUN curl -sSL https://golang.org/dl/go$GOLANG_VERSION.src.tar.gz \
    | tar -v -C /usr/src -xz
RUN cd /usr/src/go/src && ./make.bash --no-clean 2>&1
RUN mkdir -p /go/src/app /go/bin && chmod -R 777 /go

# Customizations on this line and below:
WORKDIR /go
RUN go get github.com/constabulary/gb/...
COPY . /go
RUN gb build all

Next, run the command to build the docker image:

% docker build --tag='gbtest:1' . 2>&1 1>build.log
time="2015-05-01T07:11:54-07:00" level="info" msg="The command [/bin/sh -c gb build all] returned a non-zero code: 1" 

Inspection of the build.log does not indicate any errors that occurred during the build/install of the go tools.

The docker version information used in this test:

$ docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef

Just working through a couple of user scenarios, and trying to get gb working on a windows workstation.

I built gb.exe successfully, and then created the following directory structure:

C:\Users\jason_000\mymysqltestproj> dir /a-d /b /s
C:\Users\jason_000\mymysqltestproj\bin\gb-env.exe
C:\Users\jason_000\mymysqltestproj\bin\gb-vendor.exe
C:\Users\jason_000\mymysqltestproj\bin\gb.exe
C:\Users\jason_000\mymysqltestproj\pkg\windows_amd64\github.com\constabulary\gb.a
C:\Users\jason_000\mymysqltestproj\pkg\windows_amd64\github.com\constabulary\gb\cmd.a
C:\Users\jason_000\mymysqltestproj\src\mytestproj\main.go

The only GO-named environment variable set is GOPATH:

C:\Users\jason_000\mymysqltestproj> set GO
GOPATH="C:\Users\jason_000\mymysqltestproj"

I can run the program using go run as follows:

C:\Users\jason_000\mymysqltestproj> go run src\mytestproj\main.go
Hello, world.

But trying to build the application using gb fails:

C:\Users\jason_000\mymysqltestproj> bin\gb.exe build -v all
2015/05/02 14:54:14 INFO project root "C:\\Users\\jason_000\\mymysqltestproj"
2015/05/02 14:54:14 DEBUG matchPackages: [C:\Users\jason_000\mymysqltestproj\src] all
2015/05/02 14:54:14 DEBUG args: [mytestproj]
2015/05/02 14:54:14 DEBUG loadPackage: mytestproj
2015/05/02 14:54:14 INFO build duration: 0 map[]
2015/05/02 14:54:14 command "build" failed: failed to resolve package "mytestproj": cannot find package "mytestproj" in any of:
        c:\go\src\mytestproj (from $GOROOT)
        C:\Users\jason_000\mymysqltestproj:C:\Users\jason_000\mymysqltestproj\vendor\src\mytestproj (from $GOPATH)

I believe the problem is due o gb not appending src to the project root when it looks for matching package/path names. In the last line of output, notice that the search path is:

C:\Users\jason_000\mymysqltestproj:C:\Users\jason_000\mymysqltestproj\vendor\src\mytestproj

Instead of:

C:\Users\jason_000\mymysqltestproj\src:C:\Users\jason_000\mymysqltestproj\vendor\src\mytestproj

After running a successful gb build -v all on a hello world app, I tried to run the same command again immediately:

$ gb build -v all
DEBUG project root "c:\\Users\\jason_000\\gbtest"
DEBUG matchPackages: [c:\Users\jason_000\gbtest\src] all
DEBUG args: [github.com/constabulary/gb github.com/constabulary/gb/cmd github.com/constabulary/gb/cmd/gb github.com/constabulary/gb/cmd/gb-env mysqltest]
DEBUG loadPackage: github.com/constabulary/gb false (c:\Users\jason_000\gbtest\src\github.com\constabulary\gb)
DEBUG loadPackage: github.com/constabulary/gb/cmd false (c:\Users\jason_000\gbtest\src\github.com\constabulary\gb\cmd)
DEBUG loadPackage: github.com/constabulary/gb/cmd/gb true (c:\Users\jason_000\gbtest\src\github.com\constabulary\gb\cmd\gb)
DEBUG loadPackage: github.com/constabulary/gb/cmd/gb-env true (c:\Users\jason_000\gbtest\src\github.com\constabulary\gb\cmd\gb-env)
DEBUG loadPackage: mysqltest true (c:\Users\jason_000\gbtest\src\mysqltest)
DEBUG buildPackage: github.com/constabulary/gb
DEBUG buildPackage: github.com/constabulary/gb/cmd
DEBUG buildPackage: github.com/constabulary/gb/cmd/gb
DEBUG {main github.com/constabulary/gb/cmd/gb c:\Users\jason_000\gbtest\src\github.com\constabulary\gb\cmd\gb} is a main package, not installing
DEBUG buildPackage: github.com/constabulary/gb/cmd/gb-env
DEBUG {main github.com/constabulary/gb/cmd/gb-env c:\Users\jason_000\gbtest\src\github.com\constabulary\gb\cmd\gb-env} is a main package, not installing
DEBUG buildPackage: mysqltest
DEBUG {main mysqltest c:\Users\jason_000\gbtest\src\mysqltest} is a main package, not installing
INFO github.com/constabulary/gb/cmd/gb
DEBUG gc:gc github.com/constabulary/gb/cmd/gb c:\Users\jason_000\gbtest\src\github.com\constabulary\gb\cmd\gb C:\Users\JASON_~1\AppData\Local\Temp\gb238271103\github.com\constabulary\gb\cmd\gb\main.a [build.go main.go plugin.go test.go]
DEBUG cd c:\Users\jason_000\gbtest\src\github.com\constabulary\gb\cmd\gb; [C:\Go\pkg\tool\windows_amd64\6g -p github.com/constabulary/gb/cmd/gb -pack -o C:\Users\JASON_~1\AppData\Local\Temp\gb238271103\github.com\constabulary\gb\cmd\gb\main.a -I C:\Users\JASON_~1\AppData\Local\Temp\gb238271103 -I c:\Users\jason_000\gbtest\pkg\windows\amd64 -complete build.go main.go plugin.go test.go]
INFO github.com/constabulary/gb/cmd/gb-env
DEBUG gc:gc github.com/constabulary/gb/cmd/gb-env c:\Users\jason_000\gbtest\src\github.com\constabulary\gb\cmd\gb-env C:\Users\JASON_~1\AppData\Local\Temp\gb238271103\github.com\constabulary\gb\cmd\gb-env\main.a [main.go]
DEBUG cd c:\Users\jason_000\gbtest\src\github.com\constabulary\gb\cmd\gb-env; [C:\Go\pkg\tool\windows_amd64\6g -p github.com/constabulary/gb/cmd/gb-env -pack -o C:\Users\JASON_~1\AppData\Local\Temp\gb238271103\github.com\constabulary\gb\cmd\gb-env\main.a -I C:\Users\JASON_~1\AppData\Local\Temp\gb238271103 -I c:\Users\jason_000\gbtest\pkg\windows\amd64 -complete main.go]
INFO mysqltest
DEBUG gc:gc mysqltest c:\Users\jason_000\gbtest\src\mysqltest C:\Users\JASON_~1\AppData\Local\Temp\gb238271103\mysqltest\main.a [main.go]
DEBUG cd c:\Users\jason_000\gbtest\src\mysqltest; [C:\Go\pkg\tool\windows_amd64\6g -p mysqltest -pack -o C:\Users\JASON_~1\AppData\Local\Temp\gb238271103\mysqltest\main.a -I C:\Users\JASON_~1\AppData\Local\Temp\gb238271103 -I c:\Users\jason_000\gbtest\pkg\windows\amd64 -complete main.go]
DEBUG cd .; [C:\Go\pkg\tool\windows_amd64\6l -o c:\Users\jason_000\gbtest\bin\gb-env -L C:\Users\JASON_~1\AppData\Local\Temp\gb238271103 -L c:\Users\jason_000\gbtest\pkg\windows\amd64 C:\Users\JASON_~1\AppData\Local\Temp\gb238271103\github.com\constabulary\gb\cmd\gb-env\main.a]
DEBUG cd .; [C:\Go\pkg\tool\windows_amd64\6l -o c:\Users\jason_000\gbtest\bin\mysqltest -L C:\Users\JASON_~1\AppData\Local\Temp\gb238271103 -L c:\Users\jason_000\gbtest\pkg\windows\amd64 C:\Users\JASON_~1\AppData\Local\Temp\gb238271103\mysqltest\main.a]
DEBUG cd .; [C:\Go\pkg\tool\windows_amd64\6l -o c:\Users\jason_000\gbtest\bin\gb -L C:\Users\JASON_~1\AppData\Local\Temp\gb238271103 -L c:\Users\jason_000\gbtest\pkg\windows\amd64 C:\Users\JASON_~1\AppData\Local\Temp\gb238271103\github.com\constabulary\gb\cmd\gb\main.a]
DEBUG build duration: 79.0402ms map[compile:111.0999ms link:16.003ms]
FATAL command "build" failed: [C:\Go\pkg\tool\windows_amd64\6l -o c:\Users\jason_000\gbtest\bin\gb -L C:\Users\JASON_~1\AppData\Local\Temp\gb238271103 -L c:\Users\jason_000\gbtest\pkg\windows\amd64 C:\Users\JASON_~1\AppData\Local\Temp\gb238271103\github.com\constabulary\gb\cmd\gb\main.a]: exit status 1
cannot create c:\Users\jason_000\gbtest\bin\gb: Permission denied

It looks like this is the result of gb build trying to rebuild the gb.exe file which is busy (because it is being executed). Shouldn't the gb build command be smart enough to know that it doesn't need to rebuild gb.exe?

Or should the documentation instruct users to remove the gb source package after the tool has been built?

This may have been asked and answered before; if so, forgive the noise.

Given gb is a greenfield implementation of the build tooling, it's not bound to conventions, like that all Go code must live in a directory with an ancestor called src. When looking at gb project structures, vendor/src/ feels stuttery, incongruent with src/, somehow unnecessary. Can vendored code live directly in vendor/, e.g. vendor/github.com/foo/bar/?

Input:

$ gb

Output:

2015/04/27 11:00:39 no command supplied

Expected output:

Usage:
  gb [flags] build [package] - builds a package
  gb [flags] test [package]  - tests a package

Flags:
  -goarch="amd64": override GOARCH
  -goos="darwin": override GOOS
  -goroot="/Users/peter/src/go.googlesource.com/go": override GOROOT
  -q=false: suppress log messages below ERROR level
  -toolchain="gc": choose go compiler toolchain
  -v=false: enable log levels below INFO level

This is a blanket issue to cover questions on the go-pm mailing list and other issues.

Supporting gb projects that don't have a main package

Currently a gb project is defined as a collection of packages and main packages (commands), along with supporting source code in vendor/src/.

At the GDG presentation in Berlin in April I said that gb was not suitable for libraries (projects consisting of non main packages, to use an imprecise definition), after reflecting on this for a few days I think this restriction may have been premature and probably motivated by trying to simplify the story for the GDG audience.

Specifically there is nothing preventing a library author using gb, and in fact many of the reasons why a library author would want to take advantage of the reproducibility guarantees that a gb project attempts to offer.

Distinguishing between projects and library projects

In principal a gb project, and a gb library project (need to find a better description) are the same, they both contain some code in $PROJECT/src/, they both may contain some supporting dependencies in vendor/src/. The only remaining point of differentiation is gb build executed in the former may produce some output in $PROJECT/bin/, while the latter most likely will not.

In short, this prohibition was short sighted and unnecessary.

However, with this prohibition relaxed, a new question arises: how can one gb project consume another gb project as a dependency ? It turns out that this isn't a very difficult proposition, but it is regrettably manual at the moment. Consider this scenario, mongodb/ is the project root of some library, twitter is the project root of a gb project that consumes mongodb's packages as a dependency. The names are completely arbitrary.

mongodb/
    src/
        client/
            client.go
    vendor/
        src/
            bson/
                bson.go
            gocheck/
                tests.go

twitter/
    src/
        cmd/
            twitter/
                main.go
    vendor/
        src/

mongodb and twitter are developed separately, they live in different respositories. The mongodb client has a dependency on bson, and includes its source in its' vendor/src/ directory. mongodb also has a dependency on gocheck for tests.

The process for vendoring mongodb into twitter/vendor/src/ would be to copy the contents of mongodb/src/* into twitter/vendor/src/ and also selectively copying mongodb/vendor/src/* into twitter/vendor/src/. The project owner of twitter may choose to omit the gocheck dependency of mongodb, of they may choose to include it for completeness.

Open issues

The process described above is not difficult, but it does require the package owner to be cognoscente of their actions when importing new dependencies into their project. Automating this initial import with tools is difficult at the moment due to a lack of version information in Go code in general (see #20 for a distinction between version and revision). Automating the orderly upgrade of a set of vendored dependencies is more difficult again.

Tackling these issues are outside the scope of this specific bug, and arguably outside the code mandate of gb -- it just provides the mechanism to work with a "project" of Go code, it is mute on how that source is assembled -- but feels like ample fuel for a plugin that tackles these complicated issues.

go list is a very useful command and gb could benefit from a similar tool to introspect a project.

gb list should probably be integrated into gb, but I would like to see it initially developed as a plugin to road test the plugin API (such that it is).

gb list should work the same as go list; you supply a text/template which is executed in an environment that has access to *gb.Project, *gp.Context and maybe a []*gb.Package from the resolved package set.

This template can be supplied on the command line, similar to go list -f, but I'd also like to see it passed into the command via stdin, eg echo "{{ .Context }}" | go list

Just setting up a practice project, with the following structure:

$ find .
.
./bin
./bin/gb-env.exe
./bin/gb-vendor.exe
./bin/gb.exe
./bin/main
./pkg
./pkg/windows_amd64
./pkg/windows_amd64/github.com
./pkg/windows_amd64/github.com/constabulary
./pkg/windows_amd64/github.com/constabulary/gb
./pkg/windows_amd64/github.com/constabulary/gb/cmd.a
./pkg/windows_amd64/github.com/constabulary/gb.a
./src
./src/mytestproj
./src/mytestproj/main
./src/mytestproj/main/main.go
./vendor
./vendor/src

GOPATH, GOROOT are both unset. Go is installed in /home/jbuberel/go:

jbuberel@ubuntu:/media/sf_mymysqltestproj$ gb build -v -goroot="/home/jbuberel/go" all
2015/05/02 18:31:50 INFO project root "/media/sf_mymysqltestproj"
2015/05/02 18:31:50 DEBUG matchPackages: [/media/sf_mymysqltestproj/src] all
2015/05/02 18:31:50 DEBUG args: [mytestproj/main]
2015/05/02 18:31:50 DEBUG loadPackage: mytestproj/main
2015/05/02 18:31:50 DEBUG buildPackage: mytestproj/main
2015/05/02 18:31:50 DEBUG {main mytestproj/main /media/sf_mymysqltestproj/src/mytestproj/main} is a main package, not installing
2015/05/02 18:31:50 INFO compile mytestproj/main [main.go]
2015/05/02 18:31:50 DEBUG gc:gc mytestproj/main /media/sf_mymysqltestproj/src/mytestproj/main /tmp/gb086080072/mytestproj/main/main.a [main.go]
2015/05/02 18:31:50 DEBUG cd /media/sf_mymysqltestproj/src/mytestproj/main; [/home/jbuberel/go/pkg/tool/linux_amd64/6g -p mytestproj/main -pack -o /tmp/gb086080072/mytestproj/main/main.a -I /tmp/gb086080072 -I /media/sf_mymysqltestproj/pkg/linux/amd64 -complete main.go]
2015/05/02 18:31:50 INFO build duration: 5.121073ms map[compile:3.065923ms]
2015/05/02 18:31:50 command "build" failed: [/home/jbuberel/go/pkg/tool/linux_amd64/6g -p mytestproj/main -pack -o /tmp/gb086080072/mytestproj/main/main.a -I /tmp/gb086080072 -I /media/sf_mymysqltestproj/pkg/linux/amd64 -complete main.go]: exit status 1
main.go:4: can't find import: "fmt"

So I then set GOROOT, and remove the command-line option. The build works:

jbuberel@ubuntu:/media/sf_mymysqltestproj$ export GOROOT=/home/jbuberel/go
jbuberel@ubuntu:/media/sf_mymysqltestproj$ gb build -v all
2015/05/02 18:32:51 INFO project root "/media/sf_mymysqltestproj"
2015/05/02 18:32:51 DEBUG matchPackages: [/media/sf_mymysqltestproj/src] all
2015/05/02 18:32:51 DEBUG args: [mytestproj/main]
2015/05/02 18:32:51 DEBUG loadPackage: mytestproj/main
2015/05/02 18:32:51 DEBUG buildPackage: mytestproj/main
2015/05/02 18:32:51 DEBUG {main mytestproj/main /media/sf_mymysqltestproj/src/mytestproj/main} is a main package, not installing
2015/05/02 18:32:51 INFO compile mytestproj/main [main.go]
2015/05/02 18:32:51 DEBUG gc:gc mytestproj/main /media/sf_mymysqltestproj/src/mytestproj/main /tmp/gb614142370/mytestproj/main/main.a [main.go]
2015/05/02 18:32:51 DEBUG cd /media/sf_mymysqltestproj/src/mytestproj/main; [/home/jbuberel/go/pkg/tool/linux_amd64/6g -p mytestproj/main -pack -o /tmp/gb614142370/mytestproj/main/main.a -I /tmp/gb614142370 -I /media/sf_mymysqltestproj/pkg/linux/amd64 -complete main.go]
2015/05/02 18:32:51 INFO link /media/sf_mymysqltestproj/bin/main [/tmp/gb614142370/mytestproj/main/main.a]
2015/05/02 18:32:51 DEBUG cd .; [/home/jbuberel/go/pkg/tool/linux_amd64/6l -o /media/sf_mymysqltestproj/bin/main -L /tmp/gb614142370 -L /media/sf_mymysqltestproj/pkg/linux/amd64 /tmp/gb614142370/mytestproj/main/main.a]
2015/05/02 18:32:51 DEBUG removing work directory: /tmp/gb614142370
2015/05/02 18:32:51 INFO build duration: 120.359956ms map[compile:5.178768ms link:110.870078ms]

Is this an intended behavior?

Unfortunately I can't put the actual repo url in here, but this is hanging when trying to vendor one of our (Uber's) repositories:

gb vendor <host>/<path>/<repo>.git
2015/04/29 15:00:53 INFO project root "<project root>"

...and just stays there until I Ctrl-C.

I set up a simple project:
src/main.go
vendor/src/go-ini/ini.go

When I run gb build all nothing happens, no files created, no error messages.

I ran gb build -v=true all and got the following:
DEBUG project root "t:\2015\PJT\Golang\CWI Automation"
DEBUG matchPackages: [t:\2015\PJT\Golang\CWI Automation\src] all
DEBUG args: []
DEBUG removing work directory: C:\Users\Laci\AppData\Local\Temp\gb086314939
DEBUG build duration: 976.6us map[]

I would be happy to debug this but don't even know where to start.

Looks like the plugin name is being treated as an import path.

$ mkdir foo/src/bar
$ cd foo/src
$ gb env
GB_PROJECT_DIR=/Users/rb/go/foo
$ cd bar
$ gb env
FATAL command "env" failed: plugin: unable to locate "gb-bar/env": exec: "gb-bar/env": stat gb-bar/env: no such file or directory

While I really like the direction gb is taking, the main obstacle to adoption for me is the fact that it breaks my tool chain, more specifically goimports which kicks in every time is save a go file inside vim.

Currently it's only showing the filename. An example error message:

color ❯ gb build -q
2015/05/09 00:43:44 command "build" failed: [/usr/local/go/pkg/tool/darwin_amd64/6g 
-p github.com/fatih/color 
-pack -o /var/folders/mc/hxkh39y93dn__458mq8xrbcm0000gn/T/gb360767389/github.com/fatih/color.a 
-I /var/folders/mc/hxkh39y93dn__458mq8xrbcm0000gn/T/gb360767389 
-I /Users/fatih/Code/color/pkg/darwin/amd64 -complete color.go doc.go]: exit status 1
color.go:78: c evaluated but not used

Would be better if it's changes too (relative path):

src/github.com/fatih/color.go:78: c evaluated but not used

Or even an absolute path. As I know the go tool outputs the file names as relative paths. This is important so an editor can capture and jump to it (example: vim's quickfix window).

Of course a much better approach would be having a editor compatible output (such as we have on in Oracle). It would make parsing easier and we could pipe the result directly to the editor's error parsing function.

Specifically, -X main.Version v1.0.47 and similar to create properly version tagged binaries.

(Or how do you solve this?)

dcc4f81 removed the -toolchain flag.

This bug is to track adding support for gccgo (at least)

Could you add a summary of the differences with the standard go tool?

Here is the two main differences I noted:

1. The GOPATH is automatically determined depending on the current working directory by searching directories src or vendor/src.
2. The tool makes sure that deleted packages are not used anymore, even in their compiled form, unlike the standard go tool that still uses remaining .a files.

Regarding the first difference, which is the main selling point of gb, I'm wondering why it couldn't be implemented as a wrapper around the standard tool. This is what godep does, in some way. But I'm probably missing something :)

Even if it cannot be implemented as a wrapper, it would be great to reuse as much code as possible from the standard library, instead of reimplementing the whole tool?

Regarding the second difference, is there a reason for not fixing the behavior directly in the standard go tool?

Instead of fetching dependencies into vendor/src, should it put them in internal/src or internal/vendor/src to comply with the proposed "official recommendation" for vendoring?

https://groups.google.com/d/msg/golang-dev/nMWoEAG55v8/iJGgur7W_SEJ

1. officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.

One of the design goals of gb is to handle tags efficiently; any change to the build tags for an build/test invocation should result in a rebuild of the relevant cached packages.

A subtler twist on this is $GOOS and $GOARCH are effectively tags, although they are handled more explicitly; eg $PROJECT/pkg/$GOOS/$GOARCH. Taking this to it's logical extent the value of $GOROOT, and the toolchain used also affect the caching and reuse of cached packages.

I propose that we follow the lead of the go tool and use a hash to identify packages constructed from a combination of toolchains, goroots, arches, tags, etc. Unlike the go tool, which is embedding this value inside the .a file, I propose we put it in a prefix of the package path, as the isStale computation uses file paths to assert staleness. Specifically I'm thinking the shorthash of a SHA1 of a set of hashable values.

So, where we currently have, $PROJECT/pkg/$GOOS/$GOACH/importpath.a, this would become $PROJECT/pkg/$HASH/importpath.a.

A hash would be computed from a gb.Context, and be stable for the use of that Context, in theory, the same Context constructed

Some (probably not all) the properties of the build that would be:

• $GOROOT
• toolchain
• tags
• $GOOS / $GOARCH

The downside of this approach is aggressive use of tags, cross compilation and toolchains would bloat $PROJECT/pkg with multiple cached file. I don't see this as a serious concern, this directory is only a cache and can be removed without consequence and rebuilt as needed.

A separate question of how to handle multiple final binaries in $PROJECT/bin will have to be addressed in another issue.

When building from inside the vendor directory gb would view vendor as the project root which will likely surprise many people.

External tests are currently ignored by gb test, this needs to be fixed.

Related to #9 and #26 cross compilation with gb should always just work.

Right now it does not and there are a few parts to solving this

• The first is is #28 because not only does the toolchain change between go 1.4 and go 1.5, but so does the procedure for creating a cross compilation capable toolchain.
• The second is related to the first. Specifically in Go 1.5 we removed the requirement to do the ./make.bash --no-clean dance by letting the go tool rebuild all the way back to runtime if necessary.

For gb, I never intended it to be a tool that would compile the Go standard lib itself, so any package which is considered to be the standard library is assumed to be up to date and never recompiled, with Go 1.5, this won't be sufficient.

Actions needed are at least

• icky interaction between gb.gcToolchain and the package selection logic to determine if it needs to rebuild a std lib package
• decide on #28
• enhancements to gb to build, but not install into the toolchains' $GOROOT, packages.

If main package is placed in src folder, gb build does not recognise the package and builds nothing.
Should it be considered as a bug?

At the moment gb assumes go 1.4 as a minimum requirement, we do allow you to change the toolchain with the -goroot flag, but this was mainly for convenience.

When Go 1.5 ships, there will be a significant change to the toolchain, which I would prefer not to have to sniff at runtime.

Some options

• base the gc toolchain on the version of the toolchain that built gb using // +build go1.5 or something.
• raise the minimum requirement to Go 1.5

This also impacts the cross compilation story