feature request: support alternate project layouts about godep

[kr]

Can you provide the actual error message you see?
I don't immediately see why this wouldn't work.

What happens when you run go list -json instead of godep save?

[kr]

(My apologies for the delay. I was on out of the country
with limited computer access.)

[flaub]

I have a similar (same?) issue. Our source tree is pretty structurally equivalent to @insasho.

The source tree layout is rooted at /home/user/src/company/branch:

/
/ansible
/cpp
/go
/go/src/company/proj1
/go/src/github.com/user/repo

Our GOPATH points to /home/user/src/company/branch/go.

go list -json from /home/user/src/company/branch/go/src/company/proj1 does work.

Output of godep save from /home/user/src/company/branch/go/src/company/proj1

godep: directory "/home/user/src/company/branch/go" is not using a known version control system
godep: error loading packages

[jdx]

I believe I have a standard gopath as well, and I can use godep with most paps, but it seems to break with martini. Here's a very simple application that causes the same error: https://github.com/dickeyxxx/martini-godep

[insasho]

@kr If this is something you're willing to consider supporting, I'd love it, and I know other projects would too.

To answer your question, here's an example of our tree:

go/.               <--- subdirectory of root of git repo
   ├── bin
   ├── pkg
   │   └── linux_amd64
   ├── src
   │   ├── code.google.com   <--- external deps are not checked in (but we'd like to)
   │   ├── github.com
   │   ├── launchpad.net
   │   └── corpcorpcorp.com  <--- herein lies many of our projects
   ├── Makefile
   └── README.md

I'd like to be able to run bin/godep save corpcorpcorp.com/... from the go/ directory so that we can have consistent dependencies across our entire codebase. I don't want to maintain per-build-target Godeps files. We have company-wide dependencies and would like to stay consistent between developers as much as possible.

go list -json returns what you'd expect (failure):

$ go list -json
can't load package: package .: no buildable Go source files in /home/insasho/src/go

However, export GOPATH=$(pwd) && go list -json corpcorpcorp.com/... does return a list of dependencies.

[rliebling]

I did a little hacking to get this to work and identified two barriers to supporting this alternate layout.

1. in dep.go (from this package) the calls to VCSFromDir pass the src directory as the toplevel directory at which to stop searching for version control info (eg a .git/ dir). That is, it invokes:

_, reporoot, err := VCSFromDir(p.Dir, filepath.Join(p.Root, "src"))

where p is a package. In my hacking i changed the second arg to VCSFromDir to just '/'. But, this may be bad for various reasons. (I didn't really think about it; just wanted to see if i could make things work)
2. After making that change, there was a second problem. In code.google.com/p/go/go.tools/go/vcs/vcs.go the FromDir method executes:

    if len(dir) <= len(srcRoot) || dir[len(srcRoot)] != filepath.Separator {
        return nil, "", fmt.Errorf("directory %q is outside source root %q", dir, srcRoot)
    }

But, this fails when srcRoot is just '/'. Not sure why they check it this way, but replacing it with (what I think is better):

    if strings.HasPrefix(dir, srcRoot) {
        return nil, "", fmt.Errorf("directory %q is outside source root %q", dir, srcRoot)
    }

resolves that issue. But, this code is not directly under the control of godeps...

Anyway, i'm not sure what the best solution is here. But, as i did hack my way through to find the issues, i wanted to share them here. I would like to see this resolved.

[kr]

It seems like you are trying to recreate some of the functionality of godep by hand. Why not just use godep?

[kr]

go list -json returns what you'd expect (failure)

Godep assumes that you're in a standard Go workspace. That's a fundamental assumption and not likely to change. In particular, if the go tool itself doesn't work in your environment, then godep definitely won't work.

[insasho]

My understanding of http://golang.org/doc/code.html is that this is indeed a standard Go environment; in fact, it is basically the example in http://golang.org/doc/code.html#Command. Can you clarify what Godep's version of a "standard Go workspace" is? What does it expect GOPATH to be? I can run export GOPATH=$(pwd) && go build corpcorpcorp.com/... from the top-level directory (where bin, pkg, and src are siblings and children of the $GOPATH) and it works just fine.

I don't mind if godep is in philosophical disagreement with the idea of pinning an entire company's external dependencies to fixed versions, but pinning an entire project's dependencies with a single Godeps seems like a standard use case. If there is a better way to deal with this that I'm missing, I'd love to hear it!

Thanks!

[rliebling]

@kr I'm not sure i understand your comment about it not being a standard go workspace. I think what we're talking about is just where the root of the version control repo lies (eg - where does .git live). The issue arises when the VCS root lies outside/above the src directory. I don't really think that the "standard go workspace" really does or should care about that. The only time it matters, i think, to the go tool chain is whether the code in this particular repo is go get-able.

In my particular case, the go code i'm dealing with is but one part of a bigger repo where i have reasons to want the go code versioned together with the rest of the repo. (and, don't want to deal with git submodules or such alternatives). Within this repo i'd like to be able to build the go tool by just changing to the appropriate subdirectory, setting export GOPATH=$PWD, and running godep go install github.com/foo/bar

Anyway, i understand if you don't want to support such situations, although I think it would be a good addition.

[kr]

As you say, the go tool commands other than 'go get' don't care where you put the .git directory, and this is because they don't interact with git (or any other VCS) at all. However, godep interacts with git quite a lot, even more than go get, and I've chosen to support the most common convention, the same one that 'go get' encourages: repos that live inside subdirectories (typically domain names) of src.

I don't see why you wouldn't be able to follow this convention too.

Try something like this, mashing up both of your examples:

/home/user
    src
        company
            branch
                .git
                ops
                ansible
                cpp
                go/proj1
                go/proj2
                Makefile
                README
                whatever

In your shell .profile, set GOPATH=/home/user.

Then you can cd ~/src/company/branch and run godep save ./... to freeze all your dependencies for the entire organization in one repo, alongside non-go code. A typical import path for one of your go projects will be company/branch/go/proj1. Optionally, omit the 'go' directory level and make your import paths company/branch/proj1. Also optionally, instead of 'company/branch' you can use a valid domain name and path for your organization, and then the repo will be go-gettable.

I'm not sure i understand your comment about it not being a standard go workspace.

All I said was that if 'go list' doesn't work, then certainly godep won't work. But I don't think you meant to focus on the example you gave where that's the case.

[kr]

@dickeyxxx I wasn't able to reproduce the error you describe.

Here's what I get:

ume:~ kr$ go get github.com/dickeyxxx/martini-godep
ume:~ kr$ cd src/github.com/dickeyxxx/martini-godep
ume:martini-godep kr$ godep save
ume:martini-godep kr$ 

[kr]

Ok well, it's true that godep interacts with the VCS a lot, but this particular case was a little excessive. Does #107 solve this issue?

[kr]

After more thought, I have what I think is a better approach in #108.

I'd love to know if that solves the issue here.

[rliebling]

@kr Thanks! It sounds like it should, at least to the extent i understand it. However, i have checked out your branch and tried it and still get errors. But, I may well not be doing it right :)

I've pushed a trivial example demonstrating my basic setup; please see https://github.com/rliebling/godep_example

The readme explains what i've tried.

[kr]

Thanks for the minimal test case. This still isn't working because godep tries to keep track of which packages are subdirectories of other packages by the import path, but the import path of the root looks like _/home/rich/dev/godep_example/src and therefore github.com/rliebling/main looks like it's not contained in the same tree.

I think we might be able to do this in terms of filepaths instead. I will try.

[kr]

I'm continuing to work on this, but I should reiterate that I strongly recommend you set up a single GOPATH for your computer and put the git repos inside it. It will make your life easier.

https://twitter.com/bketelsen/status/456036577010270208

[kr]

Please note the example in http://golang.org/doc/code.html#Workspaces depicts repo roots inside subdirectories of src. That's not an accident.

[rliebling]

@kr - thanks for the suggestions on proper workspace layout and links to the twitter status with several examples. I understand and accept that my layout is not the intended way a go project should be organized. And, I will certainly continue to think about possible changes. To my thinking, though, the issue is that I think of my repo not as a go project with other files, but as a non-Go project, one small piece of which is a go program. In fact, at the subdir I set as my GOPATH, i have a Vagrantfile and Dockerfile to build (and crosscompile) the go project for windows, osx and linux. (The Vagrantfile allows any team member to do this whichever platform they are using.) The point is, most of these team members won't even have a Go compiler installed on their dev machine (outside of the docker image that automatically does the work for them). The setup is so they can git clone the project and simply build the current version of the Go project, for whichever platform, and use that tool to do their work in the larger project. The recommended layout seems appropriate only when Go is the primary focus of the project.

Anyway, all that aside, i do appreciate your efforts to make this work, especially when we all acknowledge this is to support a non-standard, non-recommended layout.

Back to my example demonstrating the problem, the point is that godep needs to figure out what to do with the github.com/rliebling/dep package. I understand that to handle the dep package properly godep will need to determine that it's in the same repo as the main package. With that in mind, I can propose a few possibilities (or, what seem to me like possibilities)

1. you don't find a VCS root for it, so you omit it from the godeps.json file, possibly only acting this way if some command line argument enables this behavior
2. a command line argument adds other, external, directories to be considered as VCS roots. So, in my example i'd specify '/home/rich/dev/godep_example' and godep would, after vcs.FromDir() returns an error, VCSFromDir would try this other VCS root dir.
3. You allow the search for VCS roots to go above the src/ dir, perhaps enabled by having a command line flag to specify a top level directory beyond which it should not search.
4. After not finding a VCS root for it, you compare directories and notice that as you recurse up the directory tree searching for the VCS root you end up in the same directory as you do searching for the VCS root of the rootset (destImportPath). You conclude that this package is already maintained in sync with the rootset pkg (for which you are generating the Godeps.json) - whether by being in the same repo or just neither being version-controlled.

I must admit that I'm only thinking about the godep save scenario and don't know the godep code very well at all. So, i may be overlooking things that should be obvious.

If one of those approaches sounds both acceptable and promising to you, i'd be happy to take a stab at a PR.

[kr]

It still seems like you should be able to accomplish all you describe, including the docker and vagrant stuff, with the example layout I proposed above. I think you can include docker stuff to help out people with the go bits of a non-go project while the repo layout fits inside a standard go workspace. Then people will have the option of using either standard go tooling or the special container tooling you describe.

If your developers ever have other go projects they want to work on, then the layout with a src directory inside the repo will cause them trouble—they'll have to start switching their GOPATH depending on whether they're working on this particular project (or else they'll have to use the special Makefile or Dockerfile or whatever build procedure in this project instead of using the standard Go tools). But perhaps that is YAGNI.

Regardless, it seems like you are creating extra work for yourself here. Many people use docker or vagrant for a standardized build environment for Go and other languages, but usually the developers end up cloning the project to work on from inside the container, or the container does this automatically. Once you're inside the container, you can do whatever's best for whatever language you're working with at the moment, since it's so easy to set up for new people. They can use standard C makefiles, standard go tooling, a uniform JVM version, whatever.

[rliebling]

@kr - while I greatly appreciate you help on this and advice on directory layout, the bottom line is I know I could change my project structure to adapt to the current limitations. For whatever reasons, good or bad, I don't think that makes sense in my case. For lack of a better analogy, retructuring the project layout around the tool I'm writing in go feels like having a house where when you enter through the front door you are taken immediately into the laundry room. It's functional, but it sure feels wrong.

For Go in general, i think the only real limitation (at least the only one I'm aware of) regarding VCS is for a package to be go get-able. If you don't care that your package can be retrieved automatically by go get then Go really does not care where the root of your VCS is. So, Go is perfectly ok with having its workspace live in a subdirectory of a VCS project. I understand godep must understand more about the VCS structure and so this issue arises.

For me, personally, i'd be perfectly happy with a flag for godep save to have godep merely warn about errors finding VCS roots and omit those packages from the Godeps.json file. This is a trivial fix, but a bit more involved to report which packages were skipped because dep.Load() would have to return more data in its errors.

[insasho]

@rliebling - This golang-nuts post describing @robfig's glock might be germane: https://groups.google.com/forum/#!topic/golang-nuts/dc0hIrbWX_U

[kr]

@rliebling that's totally fair. :)

It's possible my next change (mentioned above) will make this work for you. If not, my apologies.

[freeformz]

#108 is merged and may fix this.