Support for Bzlmod in a monorepo about bazel-gazelle

[tyler-french]

@tyler-french What do you think, are there any footguns in allowing multiple go.mod files per module? Proper go.work support would of course be more involved.

I want to better understand the reason behind having the multiple go.mod files.

Are the different projects built with different resolved versions of the different modules? In this case, supporting multiple go.mod files would not have the desired effect, as they would deterministically resolve to a single set of dependencies.

Are there any exclude or replace directives in these go.mod files? If so, allowing multiple go.mod files is actually a bit dangerous, because it might make the resolved versions from MVS non-deterministic (or dependent on the ordering of tags)

I'm wondering if it might be more practical to have one MODULE.bazel per go.mod file, each as a separate "module", and then you could have a root module /MODULE.bazel that imports the others (this should work, right @fmeum?), does resolution, and applies overrides. This would also be more "correct", in terms of following the duality of a go.mod module and a Bazel module.

[chandraaditya]

I'm wondering if it might be more practical to have one MODULE.bazel per go.mod file, each as a separate "module", and then you could have a root module /MODULE.bazel that imports the others (this should work, right @fmeum?), does resolution, and applies overrides. This would also be more "correct", in terms of following the duality of a go.mod module and a Bazel module.

Actually I think yes, this would be a better question to ask as there's no good documentation on the best practices with gazelle and Bzlmod anywhere(specifically when it comes to monorepos).

[fmeum]

A true monorepo would only have a single MODULE.bazel and a single go.mod file. If you would like to still have multiple go.mod files, you might be interested in go.work support (see #1731).

Replace directives mentioning local paths are waiting for Bazel features that would allow us to overlay a local repo with generated build files. It's also actively being worked on.

[fmeum]

@tyler-french What do you think, are there any footguns in allowing multiple go.mod files per module? Proper go.work support would of course be more involved.

[opicaud]

Hello,

I am currently using a monorepo with some multiple go modules inside, managed via bzlmod.
I can share my approach, not optimal but at least it seems working..
see here: https://github.com/opicaud/monorepo
FYI I don't use any exclude or replace directives, but i must duplicate a lot of configuration, specifically extensions go_deps and directive bazel_deps
I have also those kind of warnings, which are "normal" root module are not using any kind of go modules..

WARNING: /__w/monorepo/monorepo/MODULE.bazel:112:24: The module extension go_deps defined in @bazel_gazelle//:extensions.bzl reported incorrect imports of repositories via use_repo():

Imported, but reported as indirect dependencies by the extension:
    com_github_beorn7_floats, com_github_cucumber_godog, com_github_davecgh_go_spew, com_github_google_uuid, com_github_grpc_ecosystem_grpc_gateway_v2, com_github_hashicorp_logutils, com_github_pact_foundation_pact_go_v2, com_github_smarty_assertions, com_github_spf13_pflag, com_github_spf13_viper, com_github_stretchr_testify, com_github_urfave_cli_v3, org_golang_google_grpc, org_golang_x_exp, org_golang_x_mod

 ** You can use the following buildozer command to fix these issues:

buildozer 'use_repo_remove @bazel_gazelle//:extensions.bzl go_deps com_github_beorn7_floats com_github_cucumber_godog com_github_davecgh_go_spew com_github_google_uuid com_github_grpc_ecosystem_grpc_gateway_v2 com_github_hashicorp_logutils com_github_pact_foundation_pact_go_v2 com_github_smarty_assertions com_github_spf13_pflag com_github_spf13_viper com_github_stretchr_testify com_github_urfave_cli_v3 org_golang_google_grpc org_golang_x_exp org_golang_x_mod' //MODULE.bazel:all

Hope it can help

[aran]

I'm also trying to understand how rules_go is supposed to work with bzlmod + multiple interconnected go modules and I ended up here.

(as well as relatedly #1715)

Concretely -

• I am looking for a clear answer about whether I "should" have one MODULE.bazel at my repository root alongside WORKSPACE, or many MODULE.bazel files, one per go.mod, and perhaps need to introduce tooling to interconnect them. @fmeum says "one" here: #1715 (comment) and @tyler-french says "many" here: #1658 (comment). Or maybe it is supposed to be only one MODULE.bazel + one go.mod for a whole monorepo?
• I'm trying to understand the blockers / plan for the unsupported feature documented here: https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md ("go.mod replace directives referencing local files")

Here is a setup that is not pleasant but it works (without bzlmod). I can't see how to port it to bzlmod.

• WORKSPACE with the following stanza:

# gazelle:repository_macro go_repositories.bzl%go_repositories
go_repositories()`

• Many go modules representing different modules of functionality. Each go module has its own subfolder, a go.mod file and a BUILD.bazel file. To add an external dep, run go get (to get transitive dependencies), then bazel run //:gazelle to move those to BUILD.bazel.
• Internally-developed go modules use replace statements to point to each other via their paths in the monorepo, to allow go get to work.
• Use the github.com/airyhq/airy/tools/update-deps tool to sync all go.mod files to a consistent, repo-wide version.
• Some gazelle:prefix and gazelle:proto_strip_import_prefix fixes to manage the names/go module paths of shared .proto files that aren't in the same part of the tree as most of the go code.

[aran]

My current state is multiple go.mod files with no go.work. I'm experimenting with cutting down to one go.mod file for the whole repo based on your suggestion. Are you aware of any published examples of multi-language monorepos that follow this design? It would help me a lot, and I'm sure others. I think it would be very helpful to have some expert documentation or example about the recommended workflows.

To satisfy both go mod and gazelle, it looks like I need to

1. Colocate all go code into one go code directory that has the single go.mod.
2. Create a shadow directory structure in the go code directory, matching the directory structure of the shared repo-wide protobuf directory, and put a dummy .go file in each.

Does that sound right? The protobuf shadow directory makes me suspicious I'm still missing some best practices.