bufbuild / protobuf-es

Protocol Buffers for ECMAScript. The only JavaScript Protobuf library that is fully-compliant with Protobuf conformance tests.

Protobuf-ES

A complete implementation of Protocol Buffers in TypeScript, suitable for web browsers and Node.js, created by Buf.

Protobuf-ES is the only fully-compliant JavaScript Protobuf library that passes the Protobuf conformance tests. Read more on our blog.

Protobuf-ES's companion RPC library is Connect-ES, which supports the Connect, gRPC, and gRPC-Web protocols.

What are Protocol Buffers?

In a nutshell, Protocol Buffers have two main functions:

• They are a language for writing schemas for your data.
• They define a binary format for serializing your data.

These two independent traits functions work together to allow your project and everyone who interacts with it to define messages, fields, and service APIs in the exact same way. In a practical sense as it relates to Protobuf-ES, this means no more disparate JSON types all over the place. Instead, you define a common schema in a Protobuf file, such as:

message User {
  string first_name = 1;
  string last_name = 2;
  bool active = 3;
  User manager = 4;
  repeated string locations = 5;
  map<string, string> projects = 6;
}

And it is compiled to an ECMAScript class that can be used like this:

let user = new User({
  firstName: "Homer",
  lastName: "Simpson",
  active: true,
  locations: ["Springfield"],
  projects: { SPP: "Springfield Power Plant" },
  manager: {
    firstName: "Montgomery",
    lastName: "Burns",
  },
});

const bytes = user.toBinary();
user = User.fromBinary(bytes);
user = User.fromJsonString('{"firstName": "Homer", "lastName": "Simpson"}');

The benefits can extend to any application that interacts with yours as well. This is because the Protobuf file above can be used to generate types in many languages. The added bonus is that no one has to write any boilerplate code to make this happen. Code generators handle all of this for you.

Protocol Buffers also allow you to serialize this structured data. So, your application running in the browser can send a User object to a backend running an entirely different language, but using the exact same definition. Using an RPC framework like Connect-ES, your data is serialized into bytes on the wire and then deserialized at its destination using the defined schema.

Quickstart

1. Install the code generator, the runtime library, and the Buf CLI:

   npm install @bufbuild/protobuf @bufbuild/protoc-gen-es @bufbuild/buf

2. Create a buf.gen.yaml file that looks like this:

   # Learn more: https://docs.buf.build/configuration/v1/buf-gen-yaml
   version: v1
   plugins:
      - plugin: es
        opt: target=ts
        out: src/gen

3. Download the example.proto into a /proto directory:

   mkdir proto
   curl https://raw.githubusercontent.com/bufbuild/protobuf-es/main/packages/protobuf-test/extra/example.proto > proto/example.proto

4. Generate your code:

   npx buf generate proto

   ** Note you can also use protoc if desired.

You should now see a generated file at src/gen/example_pb.ts that contains a class named User. From here, you can begin to work with your schema.

Packages

• @bufbuild/protobuf: Provides the runtime library, containing base types, generated well-known types, and core functionality. (source code).
• @bufbuild/protoc-gen-es: Provides the code generator plugin protoc-gen-es. The code it generates depends on @bufbuild/protobuf. (source code).
• @bufbuild/protoplugin: Helps to create your own code generator plugin. The code it generates depends on @bufbuild/protobuf. (source code).

Documentation

• Code example - Example code that uses protocol buffers to manage an address book.
• Generated Code - How to generate, and what code precisely is generated for any given protobuf definition.
• Runtime API - A detailed overview of the features provided by the library @bufbuild/protobuf.
• FAQ - Frequently asked Questions.
• Migrating to Protobuf-ES - Shows the changes you'll need to switch your existing code base.
• Writing Plugins - An overview of the process of writing a plugin using @bufbuild/protoplugin.

Ecosystem

• connect-es: Type-safe APIs with Protobuf and TypeScript.
• connect-es Examples: Examples for using Connect with various TypeScript web frameworks and tooling
• protobuf-conformance: A repository running the Protobuf conformance tests against various libraries.
• Buf Studio: Web UI for ad-hoc RPCs

TypeScript

The generated code is compatible with TypeScript v4.1.2 or later, with the default compiler settings.

Copyright

The code to encode and decode varint is Copyright 2008 Google Inc., licensed under BSD-3-Clause. All other files are licensed under Apache-2.0, see LICENSE.

protobuf-es's Issues

Should the name of oneof groups be lowerCamelCase?

Hi Team,

Thanks for the work!

I tried generating Typescript from our Protobuf, and I wondered if it was on purpose to not have lowerCamelCase applied to the name of the oneof groups?

I did some digging in the code, but couldn't locate where the logic is applied.

For instance, the message:

message ResourceRequest {
  string name        = 1;
  oneof  ProjectIdentifier {
    string project_id      = 2;
  }
}

would end up like this:

export class ResourceRequest extends Message<ResourceRequest> {
  name = "";
  ProjectIdentifier: {
    value: string;
    case: "projectId";
  } | { case: undefined; value?: undefined } = { case: undefined };
}

So it results with ProjectIdentifier, while I would expect to have projectIdentifier.

Thanks :-)

Ping @remyleone

Proto messages doesn't always produce canonical JSONs

The generated code for some types (e.g. Timestamp or enum) contain toJson and toJsonString methods. These seems to work when passing a Protobuf message object to JSON.stringify. But, when passing an object which contain such object as a property, this doesn't work as intended.

In our case Timestamp is serialized as an object of seconds and nanos and enums will serialize into their numbers.

We solved it by passing a replacer to JSON.stringify like:

body: JSON.stringify(
    body, (_, value) => value?.toJson ? value.toJson() : value
),

According to MDN's docs JSON.stringify looks for the method toJSON and not toJson or toJsonString. Am I missing something?

Proposal: Inherit constructor and static methods from base Message class

I was wondering why we are generating static methods instead of simply inheriting them from the base class. Is there a reason for that?

With these messages also also used in the frontend as part of e.g. connect-web, I think we have to be extra careful not to add too much weight especially for projects with a lot of messages.

I was thinking of doing something like this with the base Message class:

export class Message<T extends Message<T> = AnyMessage> {
  static readonly runtime: ProtoRuntime;
  static readonly typeName: string;
  static readonly fields: FieldList;

  static fromBinary<T extends Message<T>>(
    this: MessageType<T>,
    bytes: Uint8Array,
    options?: Partial<BinaryReadOptions>
  ): T {
    return new this().fromBinary(bytes, options);
  }

  static fromJson<T extends Message<T>>(
    this: MessageType<T>,
    jsonValue: JsonValue,
    options?: Partial<JsonReadOptions>
  ): T {
    return new this().fromJson(jsonValue, options);
  }

  static fromJsonString<T extends Message<T>>(
    this: MessageType<T>,
    jsonString: string,
    options?: Partial<JsonReadOptions>
  ): T {
    return new this().fromJsonString(jsonString, options);
  }

  static equals<T extends Message<T>>(
    this: MessageType<T>,
    a: T | PlainMessage<T> | undefined | null,
    b: T | PlainMessage<T> | undefined | null
  ): boolean {
    return this.runtime.util.equals(this, a, b);
  }

  constructor(data?: PartialMessage<T>) {
    const self = <typeof Message<T>>this.constructor;
    self.runtime.util.initPartial(data, this as any);
  }

  // ...
}

Are there any reasons why this wouldn't work at runtime?

can we enable discussions?

Proposal : Can you provide a conversion option to lower_snake_case?

hello.

The field name is only converted to lowerCamelCase.

Can you provide conversion options to lower_snake_case or preserve_case?

Is `moduleResolution=Node` a requirement?

I'm currently migrating a project to ESM and noticed that all my buf generate commands fail when I use "moduleResolution": "NodeNext" in my tsconfig.json files. Is the use of "moduleResolution": "Node" an absolute requirement or is there some way to work around this so that NodeNext can be used as resolution mechanism?

source/index_pb.ts:42:38 - error TS2576: Property 'fromBinary' does not exist on type 'TransactionChecksum'. Did you mean to access the static member 'TransactionChecksum.fromBinary' instead?

42     return new TransactionChecksum().fromBinary(bytes, options);
                                        ~~~~~~~~~~

source/index_pb.ts:46:38 - error TS2576: Property 'fromJson' does not exist on type 'TransactionChecksum'. Did you mean to access the static member 'TransactionChecksum.fromJson' instead?

46     return new TransactionChecksum().fromJson(jsonValue, options);
                                        ~~~~~~~~

source/index_pb.ts:50:38 - error TS2576: Property 'fromJsonString' does not exist on type 'TransactionChecksum'. Did you mean to access the static member 'TransactionChecksum.fromJsonString' instead?

50     return new TransactionChecksum().fromJsonString(jsonString, options);

Wrong import when using `--es_opt=target=ts`

👋 Hello,

I've noticed a problem when I import a type between my protos while using --es_opt=target=ts.

in a.proto:

enum MyEnum {
  SOME_VAL = 0;
}

in b.proto:

import 'a.proto';

message MyMessage {
  MyEnum type = 1;
}

and when using

protoc -I proto/*.proto --es_opt=target=ts --plugin=protoc-gen-es=./node_modules/.bin/protoc-gen-es ...

the generated b_pb.ts contains

...
import {MyEnum} from "./a_pb.js";
...

which is invalid since the file name is of course a_pb.ts.

I guess it should either generate the import with the .ts suffix OR just leave it off since that's valid, too?

%1 is not a valid Win32 application.

protoc -I . --es_out="D:\*\proto\/../packages/grpc/src" --es_opt=target=ts --plugin=protoc-gen-es="D:\*\proto\/../node_modules/.bin/protoc-gen-es" ./*.proto --es_out: protoc-gen-es: %1 is not a valid Win32 application.

Environment:

version: 0.0.7
package manager: pnpm

Wrong float values via `findCustomMessageOption`

I ran into a float decoding issue with the output of findCustomMessageOption() when generating code for float validation rules as defined here:

https://github.com/bufbuild/protoc-gen-validate/blob/main/tests/harness/cases/numbers.proto#L8

The value 1.23 in that float.const option is turned into 1.2300000190734863 once it reaches my protoplugin code generator here.

As seen here (in the output): https://github.com/fubhy/protobuf-zod/blob/main/packages/protoc-gen-validate-zod/tests/harness/cases/numbers_zod.ts#L47-L51

Is that a known issue or am I doing sth. wrong or is there a bug with that? I can dig deeper myself if nothing comes to mind right away but I thought I'd ask first ...

Too strict type checking during serialization

I started migrating my project to protobuf-es from protobuf-javascript.
I received an error message while calling Message.toBinary()

cannot unwrap field value, package.Message does not define a field wrapper

It happens because here. It uses an instanceof check to ensure that the object in the property is really an instance of the filed descriptor type.

However my situation is the following:

I have the proto definitions in a moduleA
I have moduleB that is compiled independently
I have moduleC that depends on moduleB and it receives proto objects from it the wraps them into a proto object instanciated locally, then tries to send them on the wire.

The code is something like this:

import {Wrapper} from "moduleA/wrapper_pb";
import {Service} from "moduleB/server";

function wrap(s: Service): Uint8Array {
   const wrap = new Wrapper();
   const msg = s.getMsg(); // this returns {Msg} from "moduleA/msg_pb"
   wrap.msg = msg;
  return wrap.toBinary(); // this is where I get the error
}

The reason for the error is that Msg the msg class has two definitions: one in moduleB and one in moduleC and however they are equivalent the instanceof operator will return false since the prototypes are different.

It probably should be enough to check if the object's type's name is equal to the field descriptor's type name or something.
This use case works fine with the protobuf-javascript.

Edit: replaced protobuf-ts with protobuf-javascript

Inherit top-level type (e.g. PlainMessage)

I have some user-facing functions that take PlainMessage<T> as input. The problem is that PlainMessage doesn't seem to trickle down to properties that themselves are messages. Take the below snippets as an example:

Protobuf

syntax = "proto3";

option optimize_for = SPEED;

message Op {
  string denomination = 1;
}

message Msg {
  repeated Op ops = 1;
}

// Type
type x = PlainMessage<Msg>

// Expected
type x = {
    ops: PlainMessage<Op>[];
}

// Actual
type x = {
    ops: Op[];
}

When using PlainMessage<T> I would expect all properties of T that are also messages to be also referenced as PlainMessage rather than expecting the (full) Message. Is this an intentional design choice and can this be worked around without a custom type outside the library or is this an oversight? Thanks in advanced!

How do I specify a .proto files path?

Hi, this may sound trivial, but I didn't find any documentation on how to specify a path to .proto files. My .proto files live in upper directory than my react project directory and I want to generate a client code for the connect-web. Could someone provide an example?

https://connect.build/docs/web/generating-code#local-generation uses this config

# buf.gen.yaml defines a local generation template.
# For details, see https://docs.buf.build/configuration/v1/buf-gen-yaml
version: v1
plugins:
  - name: es
    out: gen
    # With target=ts, we generate TypeScript files.
    # Use target=js+dts to generate JavaScript and TypeScript declaration files
    # like remote generation does.
    opt: target=ts
  - name: connect-web
    out: gen
    # With target=ts, we generate TypeScript files.
    opt: target=ts

with this script for package.json

"scripts": {
    "buf:generate": "buf generate buf.build/bufbuild/eliza"
},

Bazel rule

A Bazel rule for protobuf-es would be very enticing for me as I've found the bazel/typescript/grpc world to be frustrating and trying to get non-rules_proto_grpc rules to work with Bazel was painful (and eventually gave up). I see that rules_proto_grpc already has some buf rules for linting/etc.

map equality is incorrect

The current implemention of equals for maps does not detect new keys: https://github.com/bufbuild/protobuf-es/blob/main/packages/protobuf/src/private/util-common.ts#L166

          case "map":
            const keys = Object.keys(va);
            if (keys.some((k) => vb[k] === undefined)) {
              return false;
            }
            switch (m.V.kind) {
              case "message":
                const messageType = m.V.T;
                return keys.every((k) => messageType.equals(va[k], vb[k]));
              case "enum":
                return keys.every((k) =>
                  scalarEquals(ScalarType.INT32, va[k], vb[k])
                );
              case "scalar":
                const scalarType = m.V.T;
                return keys.every((k) =>
                  scalarEquals(scalarType, va[k], vb[k])
                );
            }
            break;

These are considered equivalent:

  console.log(
    new MyMessage({ myMap: { a: 'b' } }).equals(
      new MyMessage({ myMap: { a: 'b', c: 'd' } })
    )
  );

This is because only the keys from va are tested, whereas all the keys from both va and vb need to be tested:

union(Object.keys(va), Object.keys(vb))

Feature request: mergePartial

Hi team, I really love the work you're doing bringing protobufs to the browser!

Feature request:

One use case we have is using a Message<T> as the value of our internal state for some of the UI. This works really nicely because it's typed and can be marshalled/unmarshalled easily. However, we often want to deep merge another PartialMessage<T> into that state. Something like proto3.util.initPartial(), except instead of replacing nested values it merges them left to right.

Other things we have considered

• Using deepMerge from lodash. Works in v0.2.x so long as you don't have any 64bit ints in your schema. Broken in v0.3.x
• Attempted porting initPartial to do a mergePartial, but it was fairly complex. I might give it another go and open a PR.

Deterministic Serialization Guarantees

I use this library in an application that needs deterministic serialization guarantees, which protobuf doesn't offer out of the box based on something like https://gist.github.com/kchristidis/39c8b310fd9da43d515c4394c3cd9510.

So my question is if there are cases with the implementation in this repository where the same message with the same input over time could result in a different (binary) output and if so, what types would cause this and would avoiding them resolve this? Thanks!

Document constructors and intrinsic default values, explain robust bigint handling

Context: #49 (comment)

Performance is not ideal

We currently use protobufjs, and would love to switch over to protobuf-es for the typescript ergonomics and es6/tree shaking that it offers. Parts of our app deal with large messages, so I did a quick performance test, and unfortunately protobuf-es is rather behind at the moment. The test I used was decoding a 6MB binary message, it containing repeated fields with about 150k entries. Protobufjs parses it in about 100ms; protobuf-es takes about 350ms. For reference, the same message as JSON is about 27MB, and JSON.parse() parses it in about 100ms as well. If providing the message/proto would be helpful, please let me know.

Support google.type.Money

There is already native helper support for google.protobuf.Timestamp, and while google.type.* aren't WKTs, they're pretty close. It should be pretty simple to generate similar helpers for money.

// Create our number formatter.
var formatter = new Intl.NumberFormat('en-US', {
  style: 'currency',
  currency: 'USD',

  // These options are needed to round to whole numbers if that's what you want.
  //minimumFractionDigits: 0, // (this suffices for whole numbers, but will print 2500.10 as $2,500.1)
  //maximumFractionDigits: 0, // (causes 2500.99 to be printed as $2,501)
});

formatter.format(2500); /* $2,500.00 */

code from: https://stackoverflow.com/a/16233919/1691542

cc @tannerlinsley

Make `protoplugin` support asynchronous code generators

There are probably other use-cases for this too but I'm struggling with readline atm. which doesn't have a synchronous interface to conveniently read files line-by-line. I need that for ingesting a text file that contains a list of encoded proto messages to feed into test cases.

PR: #277

Cleanup

Just a tracking task from review.

Important:

• Conform to Go Style Guide :-) comments, splitting up of files, we should have a Golang eng do a pass with Timo
• DOCS - we should have extensive examples, justification about why this is better than alternatives
• Have someone who is not Timo who is also an expert in JS do a deep dive, probably Paul, optimally get some external feedback
• Update LICENSE to to 2021-2022 ✅

Random comments:

• Evaluate moving private/protoplugin to new repository protoplugin-es? Probably not to start but worth a conversation.
• Move cmd/internal to internal.
• See comments on #35, tldr get rid of options if possible ✅
• Update go.mod to at least 1.17 (not really needed but) ✅
• Rename GOOG to GOOGLE_ in the Makfile :-) ✅

Provide an option to use immutable default values for repeated & map fields

Our protobuf object model includes several types that have several repeated properties. These properties are often set to the default (empty array) value because the properties are only used for certain states. Our browser-based app downloads thousands of these objects using connect-web, and this creates hundreds of thousands of empty repeated values.

I propose that an option be added to BinaryReadOptions and JsonReadOptions:
defaultsImmutable: boolean;

When this option is on, default repeated and map objects are constructed using a single (frozen) Array or Object instance that is shared across all protobuf Message instances. The value is immutable, and thus there is no need to have multiple instances of an empty array or empty object.

This options provides our app with a major reduction in the amount of memory consumed. Over one million array objects were optimized (see screenshot from the Chrome memory view):

I added a single unit test for this, but more pervasive testing would be good. I'm happy to add that if you approve of this idea.

Issue

The TypeScript declaration files use the inline member initialization syntax, so there was no way for me to implement this feature for TypeScript files. Fixing this will require more extensive changes, but I didn't want to implement those changes unless you approved of this feature.

Document supported options

What are the supported options for opt and their defaults?

version: v1
plugins:
  - name: es
    out: proto/gen
    opt: target=ts
  - name: connect-web
    out: proto/gen
    opt: target=ts

Improve support for `oneof` fields

After using protobuf-es-generated code a bit more, I'm going to expand on my comment in another issue and make it an issue of its own.

Currently, accessing oneof fields in code generated by protobuf-es is a bit more cumbersome than in other protobuf code generators. In my protobuf definition I've got a oneof that might be a post, comment or profile. Both protoc-gen-ts in TypeScript and rust-protobuf provide ways to quickly access or set a .post. So in TypeScript I can do something like:

let post = item.post
if (post) { 
   // ... 
}

or

item.post = new Post(...)

But in code generated by protobuf-es, I have to do:

let post: Post|undefined = undefined
let it = item.itemType
if (it.case == "post") { post = it.value }
if (post) {
   // ...
}

which ends up in my code so often that I made a helper function for myself.

// Helper function for getting inner Item types.
export function getInner(item: pb.Item, field: "post"): pb.Post | undefined;
export function getInner(item: pb.Item, field: "profile"): pb.Profile | undefined;
export function getInner(item: pb.Item, field: "comment"): pb.Comment | undefined;
export function getInner(item: pb.Item, field: "post"|"profile"|"comment"): pb.Post | pb.Profile | pb.Comment | undefined {
    let it = item.itemType
    if (it.case == field) {
        return it.value
    }
}

And I just discovered that when I set a field, I have to do:

// Nope: item.comment = comment
item.itemType = {case: "comment", value: comment}

It would be nice if generated code would just expose oneof fields as top-level fields (or properties) like other protobuf generators.

All that said, really enjoying protobuf-es so far. Native ESMsupport works so much more nicely than trying to get Google's protobuf implementation to compile to all my targets. 😄

Expose well known types utils

The matchWkt function from protoc-gen-es is rather useful also outside of protoc-gen-es. Can we move that to @bufbuild/protoplugin or even @bufbuild/protobuf and expose it there?

Set max repeated length for DOS protection

Rationale

protobuf is very versatile format and can exacerbate DOS issues with amplification attacks. In Javascript 1 byte of protobuf can expand to ~80 bytes using lists.

In many applications it's unreasonable to allow lists of lengths of hundred thousands, and a sane default is known in advance.

Proposal

Add an option to limit repeated length at decode time. If more items that said length are found, ignore them or throw an error. proto2 or proto3 format could accommodate the feature with:

• With custom options

message Sample {
  repeated string str = 1 [max_length_ignore = 100] ;
}

• Prepending comments to the field

message Sample {
  repeated string str = 1; // __max_length_ignore=100
}

Remove the rewrite_imports preamble

I think we should consider removing the rewrite_imports preamble from the generated code. It's more of an implementation detail (and feature) of the BSR. By removing this we get:

1. cleaner generated code.
2. ability to improve the underlying glob patterns upstream without affecting the imports. We likely don't want to do this, but if we do then it'll be seamless to the user without diff changes.
3. this is helpful for debugging, but an issue with this chunk of code will still result in the wrong import path. So that's the thing the end-user should be focusing on, and less on this option.

// @generated by protoc-gen-connect-web v0.3.3 with parameter "rewrite_imports=./pet/v1/**/*_pb.js:@buf/acme_petapis.bufbuild_es,rewrite_imports=./payment/v1alpha1/**/*_pb.js:@buf/acme_paymentapis.bufbuild_es,rewrite_imports=./payment/v1alpha1/**/*_connectweb.js:@buf/acme_paymentapis.bufbuild_connect-web,rewrite_imports=./google/type/**/*_pb.js:@buf/googleapis_googleapis.bufbuild_es,rewrite_imports=./google/type/**/*_connectweb.js:@buf/googleapis_googleapis.bufbuild_connect-web"
// @generated from file pet/v1/pet.proto (package pet.v1, syntax proto3)
/* eslint-disable */
// @ts-nocheck

When target is "ts" (TypeScirpt) the ".js" extension should not be added into import

  - name: es
    out: src
    opt:
      - target=ts

Expected

import {GetAuditRequest, GetAuditResponse, ListAuditsRequest, ListAuditsResponse} from "./audit_pb";
import {MethodKind} from "@bufbuild/protobuf";

Got: .js at end of import line.

import {GetAuditRequest, GetAuditResponse, ListAuditsRequest, ListAuditsResponse} from "./audit_pb.js";
import {MethodKind} from "@bufbuild/protobuf";

Looked at making these changes, however, the deriveImportPath is used to determine imports. Making changes there affects all imports.

Was able to get the desired affect by changing the elToContent function. Passing in bool and then replace ".js" with "".

Example:

function elToContent(el: El[], importerPath: string, isTypeScript: boolean): string {
  const c: string[] = [];
  const symbolToIdentifier = processImports(
    el,
    importerPath,
    (typeOnly, from, names) => {
      if (isTypeScript) {
        from = from.replace(/.js/, "");
      }
      const p = names.map(({ name, alias }) =>
        alias == undefined ? name : `${name} as ${alias}`
      );
      const what = `{${p.join(", ")}}`;
      if (typeOnly) {
        c.push(`import type ${what} from ${literalString(from)};\n`);
      } else {
        c.push(`import ${what} from ${literalString(from)};\n`);
      }
    }
  );
  if (c.length > 0) {
    c.push("\n");
  }
  for (const e of el) {
    if (typeof e == "string") {
      c.push(e);
      continue;
    }
    const ident = symbolToIdentifier.get(e.id);
    if (ident != undefined) {
      c.push(ident);
    }
  }
  return c.join("");
}

`map<k,v>` doesn't support bigint and boolean keys

The map<k,v> field type currently uses a simple object as a representation. However, that doesn't work for bigint keys.

Same for boolean keys. That's also theoretically supported and there's even a possible pitfall here (stringified serializing and deserializing booleans that is). So I'd personally prefer to use Map<k, v>() here unless something speaks against that?

Support google.type.(date|datetime|timeofday)

There is already native helper support for google.protobuf.Timestamp, and while google.type.* aren't WKTs, they're pretty close. Since it will all be coerced to a JS Date, similar helpers could be provided for these three types.

cc @tannerlinsley

Request: Global Registry of generated types

I see that there is createRegistry that works with a FileDescriptorSet.

Since I'm already Generating the code, is there a way that it could be registered at load time like how the google protobufs attatch to window.proto?

I'm trying to do some stuff with runtime reflection, right now, I'm hand coding a Record<string, MessageType> so as to not have to figure out how to load a filedescriptorset.bin with create-react-app's webpack configuration.

Thanks again for the library! Love where it's going!

Outputting `service` for @grpc/grpc-js

Hey, is there currently any way of outputting service definitions as code? Basically --ts_proto_opt=outputServices=grpc-js from https://github.com/stephenh/ts-proto but for this package.

Currently I'm using a mix of protobuf-es and ts-proto where ts-proto is used for our gRPC server. Would it be possible with some extensions or transpiler if it isn't possible out of the box?

Don't expose options `eslint_disable` and `ts_nocheck`

They really only exist so we can sanity-check inside of this repository. It should be possible to make them private and undocumented.

Go - JSON - TS casing difference

Assuming one adheres to buf lint's naming convention (snake_case for casing), using protoc-gen-go generates snake_cased json tags for each message's go-struct representation. The TS types are camelCased as per TS/ES standards. Thus, one cannot marshal go-structs to JSON and load it in your TS code.

Describe the solution you'd like
I'd love to have an option to either have the go-struct json tags be camelCased or, even though it's weird in terms of TS style guides, have the TS types be snake_cased so that marshalling and unmarshalling / loading works as expected.

Describe alternatives you've considered
Manually fixing the casing / using something like retag

Document PlainMessage, PartialMessage, use with spread operator

We should document these types in the runtime API documentation.

Context: #49 (comment)

need support for delimited wire format

We use the delimited wire format in the project: each message is prepended with varuint32 indicating its length. This corresponds to parseDelimitedFrom in Google protobuf, or decodeDelimited() in protobufjs.

Similar functionality is needed in protouf-es, both for reading and writing.

Unable to import @bufbuild/protobuf from CommonJS

Hi there!

I'm currently trying to import the @bufbuild/protobuf package from CommonJS, and am encountering an error as Node thinks it's requiring an ESModule, apparently due to the type field being set to module in package.json.

Error output:

const protobuf = require("@bufbuild/protobuf");
                 ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/zach/code/bufbuild-protobuf-repro/cjs/node_modules/@bufbuild/protobuf/dist/cjs/index.js from /Users/zach/code/bufbuild-protobuf-repro/cjs/index.js not supported.
/Users/zach/code/bufbuild-protobuf-repro/cjs/node_modules/@bufbuild/protobuf/dist/cjs/index.js is treated as an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which declares all .js files in that package scope as ES modules.
Instead rename /Users/zach/code/bufbuild-protobuf-repro/cjs/node_modules/@bufbuild/protobuf/dist/cjs/index.js to end in .cjs, change the requiring code to use dynamic import() which is available in all CommonJS modules, or change "type": "module" to "type": "commonjs" in /Users/zach/code/bufbuild-protobuf-repro/cjs/node_modules/@bufbuild/protobuf/package.json to treat all .js files as CommonJS (using .mjs for all ES modules instead).

    at Object.<anonymous> (/Users/zach/code/bufbuild-protobuf-repro/cjs/index.js:1:18) {
  code: 'ERR_REQUIRE_ESM'
}

Based on the fact that Node is correctly determining it should require the dist/cjs/index.js file and the presence of the exports field in package.json, I'm assuming that CommonJS support is intended, but please correct me if I'm wrong!

I don't fully know what an ideal fix is here (it might be as simple as renaming the dist/cjs/index.js to dist/cjs/index.cjs?), but if desired I'm more than willing to contribute a fix.

I'm running Node 16.15.1 on macOS 12.4 and have a very minimal reproduction of the problem in this repo: https://github.com/zacharygraziano/bufbuild-protobuf-repro/tree/main/cjs.

Thanks!

Add release documentation

Add RELEASE.md or something equivalent that explains the release process.

Avoid circular imports in the well-known types

When generating the well-known types, we generate imports like ../../index.js.

It looks like rollup might produce warnings in that case, and we should try to avoid it.

Support [jstype = JS_NUMBER] for int64

protobuf-ts has an option to use Number to represent int64 fields which is often very convenient where int32 2^32 size is not enough to represent a value, but Javascript's MAX_SAFE_INTEGER (2^53) certainly is.

In my case, this happens when dealing with accounting and money.

Unneccesary memory allocations

Every call to fromBinary generates memory allocations

  static fromBinary(bytes: Uint8Array, options?: Partial<BinaryReadOptions>): I64Property {
    return new I64Property().fromBinary(bytes, options);
  }

The value already exists statically and the allocation isn't necessary

  static fromBinary(bytes: Uint8Array, options?: Partial<BinaryReadOptions>): I64Property {
    return I64Property.fromBinary(bytes, options);
  }

Extensions

The docs say:

We implement all proto2 features, except for extensions and the text format.

I don't think we can ignore extensions - what does the core js do?

`T` is not assignable to `PartialMessage<T>` for messages with `Oneof` fields

For messages with Oneof, the generated message type T is not assignable to PartialMessage<T> because the generated message type includes a case { case: undefined, value?: undefined } that is not added by PartialMessage.

I have a patch for this which seems to fix it for our use-case (both of the additions are necessary):
id3as@c8994e4

Use extensionless imports in gereated files

When I use the command:

protoc -I . --es_out build/js --es_opt target=ts a.proto b.proto

and assume that a.proto imports b.proto the generated file (a_pb.ts) will have an import something like this:

import {SomeProto} from "./b_pb.js"

This usually works fine, but I encountered an issue running jest tests where I got an error that it cannot find module "./b_pb.js"

As a workaround I added an extra step to the build that removes the extension from the imports:

find ./build/js -name '*.ts' | xargs -I{} sed -ibak `s/_pb\.js/_pb/g' {}
find ./build/js -name '*tsbak' | xargs -I{} rm -rf {}

After this everything worked ok.

Evaluate splitting proto-gen-es into three plugins

Explicitly omitting `undefined` values

Hey, I currently have an issue with the generated classes because they have default values and I can't seem to force certain values to be undefined for when I need them to be. So the code that is being generated is based on proto files that I don't control and I call them like new ResponseInfo({ ... }) to get an instance of that response which is then passed onto another entity I don't control that calls toBinary() on it. The result then includes default values that should be omitted.

ResponseInfo {
    data: '0.0.0',
    version: '0.0.0',
    appVersion: 0n,
    lastBlockHeight: 0n,
    lastBlockAppHash: Uint8Array(0) []
}

The issue is that I need to be able to omit lastBlockHeight and lastBlockAppHash from the response because otherwise the application that receives the response will panic and terminate. Is there any way for me to omit those properties if their value is set to undefined? Currently they are being set to 0 and an empty array rather than being excluded.

Is protobuf-es compatible with Deno?

I came across protobuf-es announcement and it looks great. The analysis was excellent as well. Since you have native ES support, have you tried it Deno? Thank you, keep up the great work.

Add `google.protobuf.Any.unpack` function

Currently, in order to unpack into a message instance, you have to do:

  const any = ... // Some arbitrary `google.protobuf.Any` any.
  const message = registry.findMessage(typeName);
  const instance = new message!();
  any.unpackTo(instance);

It would be great if google.protobuf.Any was generated with a utility for that accepting a typeRegistry as input.

`google.protobuf.Any.fromJson()` error when unmarshalling JSON produced by `google.golang.org/protobuf/encoding/protojson`

When unmarshalling a proto with the schema (shortened)

syntax = "proto3";

package tests.harness;

import "google/protobuf/any.proto";

message TestCase {
    google.protobuf.Any message = 1;
}

and a json payload of:

{"message":{"@type":"type.googleapis.com/tests.harness.cases.AnyNone","val":{}}}

I get the error:

cannot decode message google.protobuf.Any from JSON: "@type" is empty

Now... Why that's happening is crystal clear. But I wonder if the fault here is with @bufbuild/protobuf attempting to unmarshal an empty object (instead of just discarding it) or whether google.golang.org/protobuf/encoding/protojson should not produce that as output in the first place.

Minor details that ran out of sync between documentation and code

I noticed a couple of minor details that ran out of sync between documentation and code:

1. The doc for Any should illustrate Any.unpack()
2. The doc for generated files should mention the plugin option import_extension
3. Since registries are seeing more use with error details and JSON serialization options in connect-web, we might want to document createRegistry() as the most direct way to create a registry from generated types (with createRegistryFromDescriptors() being the alternative without generated code).
4. Since we've started using the Desc* types in @bufbuild/protoplugin, we might want explain them in the Reflection docs