raystack / proton

This repository is home to the original protobuf interface definitions which are used throughout the Raystack ecosystem.

Proton

This repository contains the original interface definitions of Raystack APIs that support both REST and gRPC protocols. Reading the original interface definitions can provide a better understanding of Raystack APIs and help you to utilize them more efficiently. You can also use these definitions with open source tools to generate client libraries, documentation, and other artifacts.

Overview

Raystack APIs are typically deployed as API services that are hosted under different DNS names. One API service may implement multiple APIs and multiple versions of the same API.

Raystack APIs use Protocol Buffers version 3 (proto3) as their Interface Definition Language (IDL) to define the API interface and the structure of the payload messages. The same interface definition is used for both REST and RPC versions of the API, which can be accessed over different wire protocols.

There are several ways of accessing Raystack APIs:

1. JSON over HTTP: You can access all Raystack APIs directly using JSON over HTTP, using any API client libraries.
2. Protocol Buffers over gRPC: You can access Raystack APIs published in this repository through GRPC, which is a high-performance binary RPC protocol over HTTP/2. It offers many useful features, including request/response multiplex and full-duplex streaming.

Structure

This repository uses a directory hierarchy that reflects the Raystack API product structure. In general, every API has its own root directory, and each major version of the API has its own subdirectory.

The proto package names exactly match the directory: this makes it easy to locate the proto definitions and ensures that the generated client libraries have idiomatic namespaces in most programming languages.

Usage

Proton does not provide compiled language specific proto files or the descriptor sets for the respective protos. It is upto the users to pull these protos and use protoc or buf for language specific compiled files and have dependencies/imports in their code.

To generate gRPC source code for Protobuf APIs in this repository, you first need to install buf on your local machine. For next step, add buf.gen.yaml at the root of your project.

version: v1
plugins:
  - name: go
    out: api
    opt: paths=source_relative

Run below command to generate your proto to /api folder.

buf generate

Use below command if you just want to target specific package/folder

buf generate --path raystack/assets

Check out Compass implementation for reference.

Contribute

Prerequisites:

• Buf
• Git

You can run following command for linting protobuf files

buf lint

Run following command to format protobuf files

buf format -w

You can run following command for formatting protobuf files

$ buf format -w

You can add proto files when you need to introduce proto for Raystack projects. If you need to modify proto files, you need to ensure backward compatibility. To ensure the backward compatibility of your changes, you can run

buf breaking --against '.git#branch=master'

License

Proton is Apache 2.0 licensed.

proton's Issues

Update buf googleapis module

We recently made a change to our managed buf.build/googleapis/googleapis repository before certifying BSR as v1. You'll need to run buf mod update to update your buf.lock.

googleapis contains over 3800 files, mostly relating to Google's core APIs. This causes numerous issues for most users, such as timeouts when installing packages with huge swaths of unused code to pull in just a few files. Not only is this a lot of code over the network and on disk, but it can cause issues in editors that try to parse all of that code!

However, there are only ~30 files used by 99.999% of developers, and these files are the most common dependency in the Protobuf ecosystem. For example:

• google.type, which defines useful messages such as DateTime and Money.-
• google.rpc, for interacting with gRPC.
• google.api, for defining APIs, such as with grpc-gateway.

We considered these commonly used packages and included a subset of them into the new, slim googleapis repository.

We hate breaking anyone, even in an alpha, and our apologies that it's causing any difficulties here. The issues with googleapis were serious enough that we felt that we had to do this in this extraordinary circumstance. We're coming out of beta extremely soon, and once we do, we take our responsibility not to break customers (free users or otherwise) very seriously, we appreciate you riding the rough road for the moment!

Use plural for compass API response

Is your feature request related to a problem? Please describe.
In compass API, we use data field as a field that wrap results of GET API. Although the idea was to make all GET API having the same scaffolding of API response (since it will also be translated to REST response with grpc-gateway) but this might not be that flexible for future use cases. Having plural names of the type of results would make it more future-proof.

repeated Asset assets = 1;

Describe the solution you'd like
Update the response fields to use plural names instead of data

Proposal for proton structure

Proton is the central repository for all protobuf definitions. Which includes APIs, entities, events used across the entire ODPF ecosystem. To make the distinctions more clear. The following structure can be used.

├── odpf
│   ├── entities
│   │   ├── README.md
│   │   ├── common
│   │   │   ├── event.proto
│   │   │   ├── resource.proto
│   │   │   └── timestamp.proto
│   │   ├── facets
│   │   │   ├── columns.proto
│   │   │   ├── lineage.proto
│   │   └── resources
│   │       ├── dashboard.proto
│   │       ├── group.proto
│   ├── apis
│   │   ├── guardian
│   │   │   ├── v1
│   │   │   |    ├── guardian.proto
│   │   ├── optimus
│   │   │   ├── v1
│   │   │   |    ├── guardian.proto
  

This will also allow package naming to be more consistent.
io.odpf.apis.guardian.v1
io.odpf.enttiies.resources.topic
io.odpf.events.metadata.optimusJob

Move to github actions provided by github.com/bufbuild

Use the buf github actions provided by bufbuild org -

https://github.com/bufbuild/buf-breaking-action
https://github.com/bufbuild/buf-push-action
https://github.com/bufbuild/buf-setup-action
https://github.com/bufbuild/buf-lint-action

Cleanup unused java build steps

As of now none of the projects are using the java artefacts and generated FileDescriptorSet binary, hence we can remove the gradle setup from the repo.

Push Protobuf schemas to buf.build

Push repo to BSR so it can be used for dependency in other Protobuf repos and for remote generation.

Also provide a way to access the generated BSR sequence ID so it can be used to generate the synthetic version used for importing remote generated packages.

feat(assets): add email field in ownership proto

Is your feature request related to a problem? Please describe.
We observed that sometimes the ownership information contains e-mail. We need to add a new email field in assets/facets/ownership.proto to support that.

Describe the solution you'd like
Add a new email field with string datatype in assets/facets/ownership.proto.

Cleanup Secret Message in Project Specification, it is unused

Problem with buf-breaking-action CI

Describe the bug
buf-breaking-action in our CI is always failing (in main or in branch).

To Reproduce

• Run github action Validate / Publish

Expected behavior

• buf-breaking-action is working fine and all stages in CI passed

Screenshots

Run bufbuild/[email protected]
buf: downloading buf.build/grpc-ecosystem/grpc-gateway:de24a84aed3d4d84806[7](https://github.com/odpf/proton/runs/6948423099?check_suite_focus=true#step:5:8)40946b55d947
Failure: module had invalid dependency: empty digest.
Error: buf: downloading buf.build/grpc-ecosystem/grpc-gateway:de24a84aed3d4d84806740946b55d947
Failure: module had invalid dependency: empty digest.

Additional context
Add any other context about the problem here.

Stricter breaking change detection rules

As of now the configured category of rules is WIRE, which is the most permissive one. We should look into using one of the more stricter categories -

FILE > PACKAGE > WIRE_JSON > WIRE

One reason for us to move to WIRE_JSON would be for detecting changes that would break pipelines that use Beast.

fix(lint): versioning assets with v1beta1

Is your feature request related to a problem? Please describe.
There are lint errors in odpf/assets that are being used by meteor and columbus. We need to fix the lint error by adding version v1beta1 to support release later.

Describe the solution you'd like
Fix lint and add version v1beta1.

Use buf to manage external proto dependencies

Is your feature request related to a problem? Please describe.
As of now files from external sources are cloned and placed in odpf/third_party dir to be used in imports.

Describe the solution you'd like
Define the dependencies in buf.yaml and let buf handle it at time of generation.

Refactor siren grpc interface

Is your feature request related to a problem? Please describe.
There are a lot of lint error due to the siren grpc api that were defined are very specific and might be complicated in the future. We use buf lint to do linting. Some suggestion in designing grpc api can be seen here and with buf lint we can implicitly enforcing the design.

Describe the solution you'd like
Here are some changes needed.

RPCs should not sharing request & response, each rpc should have its own request & response ref

rpc CreateProvider(CreateProviderRequest) returns (Provider) 
rpc GetProvider(GetProviderRequest) returns (Provider)

We have above rpc, since they are sharing a message response, in the near future it is harder for us to modify only for a specific RPC. We need to do this instead.

rpc CreateProvider(CreateProviderRequest) returns (CreateProviderRequest) 
rpc GetProvider(GetProviderRequest) returns (GetProviderResponse)

With this approach each message should have a data field as a placeholder of the response. In current implementation we are using domain name
Change from

message GetProviderResponse {
    Provider provider = 1;
}

to

message GetProviderResponse {
    Provider data = 1;
}

We do this to all of our API response so it is consistent and predictable. Response's body always inside the data field.

Better to have a dedicated empty request & response rather than using google.protobuf.Empty ref

rpc ListNamespaces(google.protobuf.Empty) returns (ListNamespacesResponse)

Do this instead

rpc ListNamespaces(ListNamespacesRequest) returns (ListNamespacesResponse)
message ListNamespacesRequest {}

Make this NotifyReceiver rpc more generic and handle the conversion in the service ref

Current implementation is coupling gRPC API with slack contract, we can make this more abstract and do convert the notification message to slack contract in our app.

Define more generic alert history hook API (if possible) ref