googleapis / api-linter

A linter for APIs defined in protocol buffers.

Google APIs

This repository contains the original interface definitions of public Google APIs that support both REST and gRPC protocols. Reading the original interface definitions can provide a better understanding of Google APIs and help you to utilize them more efficiently. You can also use these definitions with open source tools to generate client libraries, documentation, and other artifacts.

Building

Bazel

The recommended way to build the API client libraries is through Bazel >= 4.2.2.

First, install bazel.

To build all libraries:

bazel build //...

To test all libraries:

bazel test //...

To build one library in all languages:

bazel build //google/example/library/v1/...

To build the Java package for one library:

bazel build //google/example/library/v1:google-cloud-example-library-v1-java

Bazel packages exist in all the libraries for Java, Go, Python, Ruby, Node.js, PHP and C#.

Overview

Google APIs are typically deployed as API services that are hosted under different DNS names. One API service may implement multiple APIs and multiple versions of the same API.

Google APIs use Protocol Buffers version 3 (proto3) as their Interface Definition Language (IDL) to define the API interface and the structure of the payload messages. The same interface definition is used for both REST and RPC versions of the API, which can be accessed over different wire protocols.

There are several ways of accessing Google APIs:

1. JSON over HTTP: You can access all Google APIs directly using JSON over HTTP, using Google API client library or third-party API client libraries.

2. Protocol Buffers over gRPC: You can access Google APIs published in this repository through GRPC, which is a high-performance binary RPC protocol over HTTP/2. It offers many useful features, including request/response multiplex and full-duplex streaming.

3. Google Cloud Client Libraries: You can use these libraries to access Google Cloud APIs. They are based on gRPC for better performance and provide idiomatic client surface for better developer experience.

Discussions

This repo contains copies of Google API definitions and related files. For discussions or to raise issues about Google API client libraries, GRPC or Google Cloud Client Libraries please refer to the repos associated with each area.

Repository Structure

This repository uses a directory hierarchy that reflects the Google API product structure. In general, every API has its own root directory, and each major version of the API has its own subdirectory. The proto package names exactly match the directory: this makes it easy to locate the proto definitions and ensures that the generated client libraries have idiomatic namespaces in most programming languages. Alongside the API directories live the configuration files for the GAPIC toolkit.

NOTE: The major version of an API is used to indicate breaking change to the API.

Generate gRPC Source Code

To generate gRPC source code for Google APIs in this repository, you first need to install both Protocol Buffers and gRPC on your local machine, then you can run make LANGUAGE=xxx all to generate the source code. You need to integrate the generated source code into your application build system.

NOTE: The Makefile is only intended to generate source code for the entire repository. It is not for generating linkable client library for a specific API. Please see other repositories under https://github.com/googleapis for generating linkable client libraries.

Go gRPC Source Code

It is difficult to generate Go gRPC source code from this repository, since Go has different directory structure. Please use this repository instead.

api-linter's Issues

api-linter fails golangci-lint

Are any of these important or are they all acceptable?

$ golangci-lint run

rules/descriptor/walk_test.go:76:7: Error return value of `Walk` is not checked (errcheck)
		Walk(test.descriptor, consumer)
		    ^
rules/descriptor/walk_test.go:89:6: Error return value of `Walk` is not checked (errcheck)
	Walk(f, consumer)
	    ^
lint/rule_info.go:29:2: `noPositional` is unused (structcheck)
	noPositional struct{} // Prevent positional composite literal instantiation
	^
lint/source_test.go:32:6: U1000: type `testDescriptorVisiting` is unused (unused)
type testDescriptorVisiting struct {
     ^
The command "golangci-lint run" exited with 1.

Stop using deprecated FindMessageByName

FindMessageByName has been deprecated and replaced with FindDescriptorByName.

Let's migrate to the new usage.

Create a guide for rule developers or contributors

This guide should explain how to create, test, and add a rule.

AIP-134: Standard methods (update) / Partial update

Old lint rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/partial-update.md

Integration test: needs to check final, marshaled text output.

We need to check the final, marshaled text output in JSON or YAML format.

Figure out whether column indices should be 1 or 0 based

Auto-update dependencies

The linter depends on the active-developing protobuf v2 and other packages. We need to find a way to auto update those dependencies so that we can avoid issue #96. For example, Renovate may be a good solution.

Error levels

As best as I can tell, lint.Problem lacks any concept of an error level.
Most linters support both warnings and errors, and we presumably should also (e.g. warn for "shoulds", error for "musts").

Most linters also have an "error on warnings" setting.

Can users disable specific problems rather than just rules?

Right now, there is a lot of boilerplate in rule-writing, which largely stems from duplicate setup.
For example, if writing rules for AIP-131, I have to ignore methods not starting with Get-, parse out the RPC, possibly traverse to the input and output, etc. All of these things are reasonable, but doing them 8-10 times is tedious.

Right now, the reason we have to do this is because users can only disable rules, so if I have multiple problems covered by the same rule, they are "all or nothing" for users.

Can we make it possible to disable (or ignore) specific problems, perhaps by giving them identifiers and namespacing under rules?

AIP-141: Quantities / forbidden types

Old linter rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/forbidden-types.md?cl=head

Update installation instructions after review

The instructions seem correct, I just want to make them a bit prettier.

AIP-203: Field behavior / optional format

Old lint rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/optional-format.md?cl=head

Inappropriate self-referenced import error

The following proto generates an error (com/example/v1alpha1/example.proto):

syntax = "proto3";

package com.example.v1alpha1;

message Foo {
}

message GetFooRequest {
    Foo foo = 1;
}

2019/06/23 13:54:44 proto: reference to type com.example.v1alpha1.Foo without import of com/example/v1alpha1/example.proto

I think this is coming from protodesc.NewFile or newDescrptorSource:

// NewProtoRequest creates a linting Request for a .proto file.
func NewProtoRequest(fd *descriptorpb.FileDescriptorProto, reg *protoregistry.Files) (Request, error) {
	f, err := protodesc.NewFile(fd, reg)
	if err != nil {
		return Request{}, err
	}
	s, err := newDescriptorSource(fd)
	return Request{
		fileDesc:   f,
		descSource: s,
	}, err
}

Any ideas?

Maybe version of go.googlesource.com/protobuf is about a month behind master?

Remove DescriptorSource

Remove DescriptorSource once protoreflect package supports source code info.

Test for every rule having a test case

I am struggling to understand TestEveryRuleHasATestCase.

I added three rules, and a test case for each (in #80).
CI still fails saying that not every rule has a test case, which appears to point to this code, which requires a yet-different test case for each rule?

General comment: We should expect O(10000) rules. I agree that they should all have test cases, but this piece here seems to expect something different, and alarmingly it wants it in a single file. That will not scale.

Example is missing the config file

Docoumentation refers the user to example.sh:

api-linter check --cfg ./example_config.json example_bad_naming.proto

However I could not find the example_config.json file. Did I just completely miss it? Maybe it was not checked in?

Output URI from RuleInfo in problem findings

We have a field URI on the RuleInfo struct, but we don't actually use it anywhere. Let's add it to the Problems that rules emit so that it gets added to the yaml out.

Add an example of config using a YAML file (now that we support that)

AIP-140: Field names / Naming format

Old linter rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/naming-format.md?cl=head

Make all internal rule pages/AIP public

We need to publish rule description pages (or AIP) so that external users can see.

Enable rule developers to easily return problems on package / syntax.

Developers should have an easy way to return a problem with the location of the package and syntax directives.

Do not check imported files.

When running api-linter checkproto a.proto b.proto, the linter should only check a.proto and b.proto, but not their imported files (dependencies). For example, if a.proto imports google/type/date.proto, it is pointless and wrong to lint date.proto.

Can someone help me read annotations?

Mostly hoping @noahdietz can help when he is back on Thursday.

I can not for the life of me read the google.api.http annotation properly. This is not really the fault of the linter, it is the fault of the fact that there are 2 protoreflect packages (and we use the one that has no search ranking), at least 4 separate official descriptor packages, a proto and protobuf package, and none of these actually work together in anything remotely approaching a sane way.

I committed the aip-131-annotations branch, which is in a non-working state. I spent all day trying to just read one proto extention and I am asking for help, because between the utter lack of documentation and the fact that we overloaded every term six ways from Sunday, I really have no idea how to do it.

AIP-140: Base64 fields

Old linter rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/base64-fields.md?cl=head

AIP-126: Enumerations

Old linter rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/enum-unspecified.md?cl=head

Unwritten cloud AIP: Non-project resources

We still need a cloud AIP for this one.

Old lint rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/non-project-resources.md?cl=head

AIP-124: Message names / naming format

Note: this AIP is not yet written.

Old linter rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/naming-format.md?cl=head

Ensure that api-linter: disabled works properly.

Requirements:

• The magic incantation must match the internal (Python) linter.
• The (-- and --) syntax must be optional.

AIP-203: Field behavior / input only format

Old linter rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/input-only-format.md?cl=head

How do we deal with versioning of the linter binary?

Let DescriptorSource include source code info in multiple files

Currently, DescriptorSource only contains the source code info in a single file, but linter may want to check descriptors across files, e.g., a service may include the request and response message in a different file.

We may let DescriptorSource use {filename, path} as the key to lookup the source code info (currently, we only use path as the key).

protoreflect analysis

@mingzhi Can you offer some history / context on why we chose to use Google protoreflect rather than the jhump one?

We had a discussion about it today and are considering migrating. In particular, I have the following concerns:

1. The library we are using is in a state of flux.
2. We have to run on an unreleased version (!) to read annotations.
3. The library is poorly documented.
4. The library does not provide a way to move from a proto object to a descriptor object, which makes certain things very hard.
5. The library is unintuitive to use (examples: .Name() functions that return an object rather than a string, or functions like .Methods() and .Fields() that return custom objects rather than arrays or slices).
6. The internal and external versions diverge significantly.

We want to ensure that there is not some critical reason we are missing before we make a decision on what to do here.

Common imports.

We need to be able to resolve imports for common protos (e.g. google/api/annotations.proto) in order to write many rules.

I assert that any protos defined in this repository should be able to be imported in test templates.

Automatically detect Location based on Descriptor for a problem

The documentation for Problem.Descriptor states that if Location is unset, we will use the location of the descriptor to set it automatically.

However, I don't believe we're currently doing this.

Add a contributing.md file

A contributing.md file is a set of guidelines for repository contributors about how to contribute. Let's create these guidelines to align internally on how we want to receive contributions and ensure that contributors know they are submitting work that is likely to be accepted.

Switch status:enabled/disabled (str) -> enabled:true/false (bool)

How do I use common imports?

@robcapo I tried to do this in a test...no dice...

Consider bundling a specific version of protoc with our binary

Use t.Fatalf instead of log.Fatalf in testutil

Testutil currently uses log.Fatalf when the MustX functions fail. Rather than failing a single test, this causes the program to terminate and prevents other tests from running. We should add a *testing.T parameter to testutil.MustX functions so that they can cause just the current test to fail.

AIP-203: Field behavior / output only format

Old lint rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/output-only-format.md

Shorten command-line arguments

@jgeewax and @lukesneeringer, I propose to shorten the command line to api-lint [command options] files... instead of api-linter checkproto [command options] files..., i.e., I will remove sub-commands (checkproto). There will be some amount of code changes, so I would like to have a nod from both of you first. Thanks!

How to disable rules that post problems on the package definition?

Currently, the only way to disable a rule that posts a Problem for a package definition is via file comments at the top of the file. Ideally, a user would be able to disable a rule like this:

// (-- api-linter: corp::package_name_must_include_version=disabled --)
package foo.bar;

This is a bit more complicated than it should be, because packages are not actually Descriptors, and in order to check if a rule is disabled, we check it against the comments of the descriptor that we're linting descriptor as well as the comments for the entire file.

AIP-140: Field names / abbreviations

Old linter rule: http://g3doc/google/api/tools/linter/g3doc/rules/abbreviations

Migrate lint rules into individual Go packages.

@robcapo Can you refactor the existing rules to have each of the files exist in a separate package, as we discussed? There are several things I am not sure how to do (e.g. not duplicate testdata throughout).

AIP-122: Resource names / HTTP url patterns

Old linter rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/http-url-pattern.md?cl=head

AIP-203: Field behavior / immutable format

Old linter rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/immutable-format.md?cl=head

Document that API Linter rules should only report errors on the same file in which they're being run

Rule: For GCP resources, use location-bound names (locations/* in the URI)

AIP-192: Documentation / leading & trailing comments

Old linter rule: https://g3doc.corp.google.com/google/api/tools/linter/g3doc/rules/leading-and-trailing-comment.md?cl=head

Move all existing buganizer bugs over