isabella232 / rules_proto-1

Modern bazel build rules for protobuf / gRPC

rules_proto

Bazel skylark rules for building protocol buffers +/- gRPC ✨.

Bazel	rules_proto	gRPC

Rules

Status	Lang	Rule	Description
	android	android_proto_compile	Generates android protobuf artifacts (example)
	android	android_grpc_compile	Generates android protobuf+gRPC artifacts (example)
	android	android_proto_library	Generates android protobuf library (example)
	android	android_grpc_library	Generates android protobuf+gRPC library (example)
	closure	closure_proto_compile	Generates closure *.js protobuf+gRPC files (example)
	closure	closure_proto_library	Generates a closure_library with compiled protobuf *.js files (example)
	cpp	cpp_proto_compile	Generates .h,.cc protobuf artifacts (example)
	cpp	cpp_grpc_compile	Generates .h,.cc protobuf+gRPC artifacts (example)
	cpp	cpp_proto_library	Generates .h,.cc protobuf library (example)
	cpp	cpp_grpc_library	Generates .h,.cc protobuf+gRPC library (example)
	csharp	csharp_proto_compile	Generates csharp protobuf artifacts (example)
	csharp	csharp_grpc_compile	Generates csharp protobuf+gRPC artifacts (example)
	csharp	csharp_proto_library	Generates csharp protobuf library (example)
	csharp	csharp_grpc_library	Generates csharp protobuf+gRPC library (example)
	d	d_proto_compile	Generates d protobuf artifacts (example)
	dart	dart_proto_compile	Generates dart protobuf artifacts (example)
	dart	dart_grpc_compile	Generates dart protobuf+gRPC artifacts (example)
	dart	dart_proto_library	Generates dart protobuf library (example)
	dart	dart_grpc_library	Generates dart protobuf+gRPC library (example)
	go	go_proto_compile	Generates *.go protobuf artifacts (example)
	go	go_grpc_compile	Generates *.go protobuf+gRPC artifacts (example)
	go	go_proto_library	Generates *.go protobuf library (example)
	go	go_grpc_library	Generates *.go protobuf+gRPC library (example)
	java	java_proto_compile	Generates a srcjar with protobuf *.java files (example)
	java	java_grpc_compile	Generates a srcjar with protobuf+gRPC *.java files (example)
	java	java_proto_library	Generates a jar with compiled protobuf *.class files (example)
	java	java_grpc_library	Generates a jar with compiled protobuf+gRPC *.class files (example)
	node	node_proto_compile	Generates node *.js protobuf artifacts (example)
	node	node_grpc_compile	Generates node *.js protobuf+gRPC artifacts (example)
	node	node_proto_library	Generates node *.js protobuf library (example)
	node	node_grpc_library	Generates node *.js protobuf+gRPC library (example)
	objc	objc_proto_compile	Generates objc protobuf artifacts (example)
	objc	objc_grpc_compile	Generates objc protobuf+gRPC artifacts (example)
	php	php_proto_compile	Generates php protobuf artifacts (example)
	php	php_grpc_compile	Generates php protobuf+gRPC artifacts (example)
	python	python_proto_compile	Generates *.py protobuf artifacts (example)
	python	python_grpc_compile	Generates *.py protobuf+gRPC artifacts (example)
	python	python_proto_library	Generates *.py protobuf library (example)
	python	python_grpc_library	Generates *.py protobuf+gRPC library (example)
	ruby	ruby_proto_compile	Generates *.ruby protobuf artifacts (example)
	ruby	ruby_grpc_compile	Generates *.ruby protobuf+gRPC artifacts (example)
	ruby	ruby_proto_library	Generates *.rb protobuf library (example)
	ruby	ruby_grpc_library	Generates *.rb protobuf+gRPC library (example)
	rust	rust_proto_compile	Generates rust protobuf artifacts (example)
experimental	rust	rust_grpc_compile	Generates rust protobuf+gRPC artifacts (example)
	rust	rust_proto_library	Generates rust protobuf library (example)
experimental	rust	rust_grpc_library	Generates rust protobuf+gRPC library (example)
	scala	scala_proto_compile	Generates *.scala protobuf artifacts (example)
	scala	scala_grpc_compile	Generates *.scala protobuf+gRPC artifacts (example)
	scala	scala_proto_library	Generates *.scala protobuf library (example)
	scala	scala_grpc_library	Generates *.scala protobuf+gRPC library (example)
experimental	swift	swift_proto_compile	Generates swift protobuf artifacts (example)
experimental	swift	swift_grpc_compile	Generates swift protobuf+gRPC artifacts (example)
experimental	swift	swift_proto_library	Generates swift protobuf library (example)
experimental	swift	swift_grpc_library	Generates swift protobuf+gRPC library (example)
	gogo	gogo_proto_compile	Generates gogo protobuf artifacts (example)
	gogo	gogo_grpc_compile	Generates gogo protobuf+gRPC artifacts (example)
	gogo	gogo_proto_library	Generates gogo protobuf library (example)
	gogo	gogo_grpc_library	Generates gogo protobuf+gRPC library (example)
	gogo	gogofast_proto_compile	Generates gogofast protobuf artifacts (example)
	gogo	gogofast_grpc_compile	Generates gogofast protobuf+gRPC artifacts (example)
	gogo	gogofast_proto_library	Generates gogofast protobuf library (example)
	gogo	gogofast_grpc_library	Generates gogofast protobuf+gRPC library (example)
	gogo	gogofaster_proto_compile	Generates gogofaster protobuf artifacts (example)
	gogo	gogofaster_grpc_compile	Generates gogofaster protobuf+gRPC artifacts (example)
	gogo	gogofaster_proto_library	Generates gogofaster protobuf library (example)
	gogo	gogofaster_grpc_library	Generates gogofaster protobuf+gRPC library (example)
	grpc-gateway	gateway_grpc_compile	Generates grpc-gateway *.go files (example)
	grpc-gateway	gateway_swagger_compile	Generates grpc-gateway swagger *.json files (example)
	grpc-gateway	gateway_grpc_library	Generates grpc-gateway library files (example)
	grpc.js	closure_grpc_compile	Generates protobuf closure grpc *.js files (example)
	grpc.js	closure_grpc_library	Generates protobuf closure library *.js files (example)
	grpc-web	closure_grpc_compile	Generates a closure *.js protobuf+gRPC files (example)
	grpc-web	commonjs_grpc_compile	Generates a commonjs *.js protobuf+gRPC files (example)
	grpc-web	commonjs_dts_grpc_compile	Generates a commonjs_dts *.js protobuf+gRPC files (example)
	grpc-web	ts_grpc_compile	Generates a commonjs *.ts protobuf+gRPC files (example)
	grpc-web	closure_grpc_library	Generates protobuf closure library *.js files (example)

Overview

These rules are the successor to https://github.com/pubref/rules_protobuf and are in a pre-release status. The primary goals are:

1. Interoperate with the native proto_library rules and other proto support in the bazel ecosystem as much as possible.
2. Provide a proto_plugin rule to support custom protoc plugins.
3. Minimal dependency loading. Proto rules should not pull in more dependencies than they absolutely need.

NOTE: in general, try to use the native proto library rules when possible to minimize external dependencies in your project. Add rules_proto when you have more complex proto requirements such as when dealing with multiple output languages, gRPC, unsupported (native) language support, or custom proto plugins.

Installation

Add rules_proto your WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "build_stack_rules_proto",
    urls = ["https://github.com/stackb/rules_proto/archive/{GIT_COMMIT_ID}.tar.gz"],
    sha256 = "{ARCHIVE_TAR_GZ_SHA256}",
    strip_prefix = "rules_proto-{GIT_COMMIT_ID}",
)

Important: Follow instructions in the language-specific README.md for additional workspace dependencies that may be required.

Usage

Step 1: write a protocol buffer file (example: thing.proto):

syntax = "proto3";

package example;

import "google/protobuf/any.proto";

message Thing {
    string name = 1;
    google.protobuf.Any payload = 2;
}

Step 2: write a BAZEL.build file with a native proto_library rule:

proto_library(
    name = "thing_proto",
    srcs = ["thing.proto"],
    deps = ["@com_google_protobuf//:any_proto"],
)

In this example we have a dependency on a well-known type any.proto hance the proto_library to proto_library dependency.

Step 3: add a cpp_proto_compile rule (substitute cpp_ for the language of your choice).

NOTE: In this example thing.proto does not include service definitions (gRPC). For protos with services, use the cpp_grpc_compile rule instead.

# BUILD.bazel
load("@build_stack_rules_proto//cpp:cpp_proto_compile.bzl", "cpp_proto_compile")

cpp_proto_compile(
    name = "cpp_thing_proto",
    deps = [":thing_proto"],
)

But wait, before we can build this, we need to load the dependencies necessary for this rule (from cpp/README.md):

Step 4: load the workspace macro corresponding to the build rule.

# WORKSPACE
load("@build_stack_rules_proto//cpp:deps.bzl", "cpp_proto_compile")

cpp_proto_compile()

Note that the workspace macro has the same name of the build rule. Refer to the cpp/deps.bzl for details on what other dependencies are loaded.

We're now ready to build the rule:

Step 5: build it.

$ bazel build //example/proto:cpp_thing_proto
Target //example/proto:cpp_thing_proto up-to-date:
  bazel-genfiles/example/proto/cpp_thing_proto/example/proto/thing.pb.h
  bazel-genfiles/example/proto/cpp_thing_proto/example/proto/thing.pb.cc

If we were only interested for the generated file artifacts, the cpp_grpc_compile rule would be fine. However, for convenience we'd rather have the outputs compiled into an *.so file. To do that, let's change the rule from cpp_proto_compile to cpp_proto_library:

# WORKSPACE
load("@build_stack_rules_proto//cpp:deps.bzl", "cpp_proto_library")

cpp_proto_library()

# BUILD.bazel
load("@build_stack_rules_proto//cpp:cpp_proto_library.bzl", "cpp_proto_library")

cpp_proto_library(
    name = "cpp_thing_proto",
    deps = [":thing_proto"],
)

$ bazel build //example/proto:cpp_thing_proto
Target //example/proto:cpp_thing_proto up-to-date:
  bazel-bin/example/proto/libcpp_thing_proto.a
  bazel-bin/example/proto/libcpp_thing_proto.so  bazel-genfiles/example/proto/cpp_thing_proto/example/proto/thing.pb.h
  bazel-genfiles/example/proto/cpp_thing_proto/example/proto/thing.pb.cc

This way, we can use //example/proto:cpp_thing_proto as a dependency of any other cc_library or cc_binary rule as per normal.

NOTE: the cpp_proto_library implicitly calls cpp_proto_compile, and we can access that rule by adding _pb at the end of the rule name, like bazel build //example/proto:cpp_thing_proto_pb.

Summary

• There are generally four rule flavors for any language {lang}: {lang}_proto_compile, {lang}_proto_library, {lang}_grpc_compile, and {lang}_grpc_library.

• If you are solely interested in the source code artifacts, use the {lang}_{proto|grpc}_compile rule.

• If your proto file has services, use the {lang}_grpc_{compile|library} rule instead.

• Load any external dependencies needed for the rule via the load("@build_stack_rules_proto//{lang}:deps.bzl", "{lang}_{proto|grpc}_{compile|library}").

Developers

Code Layout

Each language {lang} has a top-level subdirectory that contains:

1. {lang}/README.md: generated documentation for the rule(s).

2. {lang}/deps.bzl: contains macro functions that declare repository rule dependencies for that language. The name of the macro corresponds to the name of the build rule you'd like to use.

3. {lang}/{rule}.bzl: rule implementation(s) of the form {lang}_{kind}_{type}, where kind is one of proto|grpc and type is one of compile|library.

4. {lang}/BUILD.bazel: contains proto_plugin() declarations for the available plugins for that language.

5. {lang}/example/{rule}/: contains a generated WORKSPACE and BUILD.bazel demonstrating standalone usage.

6. {lang}/example/routeguide/: routeguide example implementation, if possible.

The root directory contains the base rule defintions:

• plugin.bzl: A build rule that defines the name, tool binary, and options for a particular proto plugin.

• compile.bzl: A build rule that contains the proto_compile rule. This rule calls protoc with a given list of plugins and generates output files.

Additional protoc plugins and their rules are scoped to the github repository name where the plugin resides. For example, there are 3 grpc-web implementations in this repo: `github.com/improbable-eng/grpc-web, github.com/grpc/grpc-web, and github.com/stackb/grpc.js.

Rule Generation

To help maintain consistency of the rule implementations and documentation, all of the rule implementations are generated by the tool //tools/rulegen. Changes in the main README.md should be placed in tools/rulegen/README.header.md or tools/rulegen/README.footer.md. Changes to generated rules should be put in the source files (example: tools/rulegen/java.go).

Transitivity

Briefly, here's how the rules work:

1. Using the proto_library graph, collect all the *.proto files directly and transitively required for the protobuf compilation.

2. Copy the *.proto files into a "staging" area in the bazel sandbox such that a single -I. will satisfy all imports.

3. Call protoc OPTIONS FILELIST and generate outputs.

The concept of transitivity (as defined here) affects which files in the set *.proto files are named in the FILELIST. If we only list direct dependencies then we say transitive = False. If all files are named, then transitive = True. The set of files that can be included or excluded from the FILELIST are called transitivity rules, which can be defined on a per-rule or per-plugin basis. Please grep the codebase for examples of their usage.

Developing Custom Plugins

Follow the pattern seen in the multiple examples in this repository. The basic idea is:

1. Load the plugin rule: load("@build_stack_rules_proto//:plugin.bzl", "proto_plugin").
2. Define the rule, giving it a name, options (not mandatory), tool, and outputs.
3. tool is a label that refers to the binary executable for the plugin itself.
4. outputs is a list of strings that predicts the pattern of files generated by the plugin. Specifying outputs is the only attribute that requires much mental effort.

Contributing

Contributions welcome; please create Issues or GitHub pull requests.

ProtoInfo provider

A few notes about the 'proto' provider. Bazel initially implemented providers using simple strings. For example, one could write a rule that depends on proto_library rules as follows:

my_custom_rule = rule(
    implementation = my_custom_rule_impl,
    attrs = {
        "deps": attr.label_list(
            doc = "proto_library dependencies",
            mandatory = True,
            providers = ["proto"],
        ),
    }
) 

We can then collect a list of the provided "info" objects as follows:

def my_custom_rule_impl(ctx):
    # list<ProtoInfo> A list of ProtoInfo values
    deps = [ dep.proto for dep in ctx.attr.deps ]

This style of provider usage is now considered "legacy". As of bazel 0.22.0 (Jan 2019) 4817df, the ProtoInfo provider was introduced. Rules using this provider should be changed to:

my_custom_rule = rule(
    implementation = my_custom_rule_impl,
    attrs = {
        "deps": attr.label_list(
            doc = "proto_library dependencies",
            mandatory = True,
            providers = [ProtoInfo],
        ),
 

def my_custom_rule_impl(ctx):
    # <list<ProtoInfo>> A list of ProtoInfo
    deps = [ dep[ProtoInfo] for dep in ctx.attr.deps ]

A flag to toggle availability of the "legacy" provider was introduced in e3f4ce as --incompatible_disable_legacy_proto_provider. Therefore, if you still want the legacy behavior specify --incompatible_disable_legacy_proto_provider=false.

Additional reading:

• bazelbuild/bazel#3701
• bazelbuild/bazel#6901