Comments (12)
@timostamm Thanks for the reply! Our use case:
We define EventBridge message schemas with Protobuf. Each message is sent from a single producer
microservice to one or many consumer
microservices. Each microservice is in a different GitHub repository.
The ideal flow is as follows:
- In each
producer
repo, define Protobuf schemas and publish them to BSR - In each
consumer
repo, pull in BSR-generated assets for eachproducer
you handle messages from and handle the messages
Since we can't use the BSR-generated assets, however, the flow is:
- Same as above
- In each
consumer
repo, runbuf generate buf.build/[PRODUCER_REPO]:[SOME_REF] --include-imports
to generate theproducer
's proto files and handle messages
This works fine if a consumer
is handling messages from a single producer
. With more than one, however, we run into dependency management issues. Say we have two producer
s, P1 and P2. Both of their schemas depend on different versions of a dependency.
Assuming we're generating these files into the same location, say src/gen
, we need to ensure all of P1 and P2's dependencies are using the same versions or, at the very least, there aren't breaking changes between dependency versions. An alternative is to create a buf.gen.yaml
file for each producer
with different out
directories, complicating the configuration and likely leading to custom tooling.
This solution also means every consumer
is regenerating these files on each CI build with possibly differing versions of @bufbuild/protoc-gen-es
and @bufbuild/buf
, opening the door to hard to diagnose bugs that work fine in some repos but not others.
The above seems like a high price to pay just to keep using BSR as it provides minimal benefit for this use case. Instead, it seems less error prone and produces less cognitive load for microservice developers to move Protobuf schemas to a monorepo that generates the NPM packages. This gets us pretty close to the "ideal flow" above with the following downsides:
- Custom tooling we have to build and maintain (though, as described above, we'd have some custom tooling anyway unless BSR adds CommonJS support)
- The Protobuf schemas are no longer colocated with
producer
code, making iteration slower, which we could solve with yet more custom tooling
from protobuf-es.
Thanks for sharing the details on the use case, @therockstorm.
You can re-use buf.gen.yaml
files with the --template
and --output
flags. But I'd be the first to agree that remote packages are a better fit 🙂
We have a couple of other features in the pipeline, but this will be implemented. It will most likely land before the end of the year.
from protobuf-es.
@krapie, protoc-gen-connect-es wasn't updated to support the option yet. But if you update to the just released v1.2.0, it should generate CommonJS exports too:
version: v1
plugins:
# You'll need @bufbuild/protoc-gen-es v1.6.0 or later,
# and @bufbuild/protoc-gen-connect-es v1.2.0 or later
- plugin: es
out: src/gen
opt: js_import_style=legacy_commonjs
- plugin: connect-es
out: src/gen
opt: js_import_style=legacy_commonjs
We don't have integration tests for this plugin option, but it should do the job. Let us know if you encounter any problems 🙂
from protobuf-es.
Hi @krapie we just published v1.6.0 to npm: https://www.npmjs.com/package/@bufbuild/protobuf
from protobuf-es.
From this Slack thread, without BSR support for CommonJS (and, by extension, NestJS), the "best" workaround is to bypass BSR completely and instead create a monorepo with CI builds that generate CommonJS NPM packages.
Is this true? If so, how likely is Buf to support CommonJS and would we be talking months or years? If this is farther away than a couple of months, we'll have to move ahead with doing all of this ourselves and removing BSR from our workflow. Sadly, NestJS has no plans to update to ESM.
from protobuf-es.
Hey Rocky, I am not sure that bypassing the BSR completely is the best workaround. NestJS does not support ESM-only npm packages, but it does support TypeScript files with ESM imports.
For example, you might have an import like the following in your app:
// app.module.ts
import { AppController } from './app.controller';
This works in Nest, because all source files are transpiled to CJS.
If you generate code locally with protoc-gen-es, you can set the option target=ts
to get the same result - TypeScript files with ESM imports. If you import them in your app, they will be transpiled to CJS like any other source file.
It isn't as neat as remote packages, but you can generate local code from remote protobuf files on the BSR.
You need one configuration file:
# buf.gen.yaml
version: v1
plugins:
- plugin: buf.build/bufbuild/es
out: src/gen
opt: target=ts
- plugin: buf.build/connectrpc/es
out: src/gen
opt: target=ts
Then you can run buf generate
with any BSR module. For example:
$ buf generate buf.build/connectrpc/eliza
$ tree src/gen
src/gen
└── connectrpc
└── eliza
└── v1
├── eliza_connect.ts
└── eliza_pb.ts
3 directories, 2 files
Your use case might vary, but right now, this would be my recommended approach with the BSR in case the framework does not support ESM.
from protobuf-es.
Lastly, since we'll want this to be a "registry-compatible" plugin, we'll need a commonjs implementation for rewriting import paths.
This should not be necessary. Import statements are generated by @bufbuild/protoplugin
here:
protobuf-es/packages/protoplugin/src/ecmascript/generated-file.ts
Lines 206 to 209 in fcf1a1e
This could easily be extended to support CommonJS require()
calls with a plugin option. The logic to rewrite import paths would continue to work as is.
The more challenging part are export statements. They are currently handwritten by plugins. In order to support CommonJS with a plugin option, we should provide an API to generate export statements, which automatically produces a ECMAScript export statement or a CommentJS export assignment, based on the plugin option.
from protobuf-es.
That's excellent news, @timostamm! I was using https://github.com/timostamm/protobuf-ts prior to https://github.com/bufbuild/protobuf-es and they, along with the rest of the Buf tooling, are excellent to use. Thanks to the team for their efforts and for your helpful, timely responses on this issue.
from protobuf-es.
Wow, I really needed this option! Thanks!
I hope updated protobuf-es to be released ASAP :)
from protobuf-es.
@smaye81 Thank you for the release! But I'm still wondering why my XXX_connect.js
is still using export
statement, which causes ts-node SyntaxError: Unexpected token 'export'
error.
from protobuf-es.
@timostamm Ohh, I thought this release updates all plugins to provide new option (I was able to use js_import_style=legacy_commonjs
option on connect-es
) 😄
It is now working perfectly. Thank you for your fast feedback and release!
from protobuf-es.
Just to provide context:
All plugins based on the plugin framework @bufbuild/protoplugin
get the new option. protoc-gen-es, protoc-gen-connect-es, and protoc-gen-connect-query are all based on it.
But the plugin framework can only modify export statements for CommonJS when it knows what the export statements are. For example, protoc-gen-connect-es was using this call to generate a service:
f.print("export const ", localName(service), " = {");
"export const"
is just an opaque string to the plugin framework, and it would only be possible to modify it safely with a full parser, which is a big perf hit for large protobuf modules. So the latest release of the plugin framework adds a new method to generate export statements, and the latest release of protoc-gen-connect-es uses it:
f.print(f.exportDecl("const", localName(service)), " = {");
Similar to the already existing f.import()
, this method generates a statement conditionally. Plugins must use f.import()
and f.exportDecl()
to fully support the new plugin option. More information is available in the PR, and in the documentation.
from protobuf-es.
Related Issues (20)
- Comparing messages (Diff) HOT 2
- JSON serialization options HOT 2
- Get T property of FieldInfo HOT 3
- Expose message class's canonical name HOT 4
- Support option retention in @bufbuild/protoplugin HOT 2
- Regression: enum strings are broken in constructor of message types in 1.8.0 HOT 3
- Using message as a value in Map seems to break deserialisation HOT 2
- Values missing if set to 0 and after converting toJson() HOT 2
- Upgrade TypeScript to v5 in `@bufbuild/protoplugin` to reduce install size for users HOT 3
- Support for `google/api/field_behavior` ? HOT 1
- Tree-Shaking for `fields` HOT 3
- Incorrect typing of Struct.toJson() HOT 3
- v2 Access HOT 2
- getEnumType doesn't work when SSR HOT 3
- Fix Node10 subpath fallbacks
- Remove internal fields HOT 14
- The import is not working HOT 2
- Better terminal logs HOT 4
- Cannot encode field to JSON HOT 5
- @bufbuild/protobuf proto3 serialization regression: throw error when serializing oneof property with `undefined` value HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from protobuf-es.