Comments (2)
Did some research on various protoc plugins for generating js+ts code from protodef. We need something which makes metadata accessible (esp. field numbers) in order to implement the representation described in #2455. Here are my findings:
protobufjs ✅
The most popular lib I found (16M weekly downloads), originally designed by Google. It can generate js and ts, but to get what we want you need to call it three times - once to generate the js, once to generate to ts bindings, and once to generate JSON with pb metadata.
npm i -D protobufjs protobufjs-cli
npx pbjs -t static-module ../proto/survey.proto -o dist/survey.js
npx pbts dist/survey.js -o dist/survey.d.ts
npx pbjs -t json ../proto/survey.proto -o dist/survey.proto.json
We could load the JSON at runtime to pick out field numbers we need.
Alternatives considered
@protobuf-ts/plugin
146k downloads/week. Generates pure ts code in a single command, and includes field numbers in the message metadata. But given this is a less "standard", less used library, prefer using protobufjs
.
ts-proto
OTOO 300k weekly downloads, and doesn't encode the field numbers anywhere we can use at runtime, so we can exclude this one for now.
protoc-gen-ts
It seems the protoc
packaged with the one didn't support my M1 Mac.
npm i -D @protobuf-ts/plugin
npx protoc --ts_out dist --proto_path ../proto survey.proto loi.proto submission.proto
It doesn't produce or consume JSON, but it's fairly easy to go from ts object<>JSON anyway, plus we'll prob want more control over the formats.
grpc_tools_node_protoc_ts
Only 100k downloads per week, but uses the standard google grpc-tools. Doesn't seem to embed the set of valid field numbers, though individual fields can be fetched by field number:
npm i -D grpc-tools grpc_tools_node_protoc_ts
npx grpc_tools_node_protoc --js_out=import_style=commonjs,binary:./dist --grpc_out=grpc_js:./dist --plugin=protoc-gen-grpc=./node_modules/.bin/grpc_tools_node_protoc_plugin -I ../proto survey.proto
npx protoc --plugin=protoc-gen-ts=./node_modules/.bin/protoc-gen-ts --ts_out=grpc_js:./dist -I ../proto survey.proto
from ground-platform.
Also, carrying forward proposal from google/ground-android#2455:
- Create lightweight converters in Kotlin and TypeScript which take a
Map<Long, Any>
and produce protobufs and vice-versa, where keys in Firestore correspond to a protobuf field number, and values primitive types supported by proto.
Example survey data before:
{
"title": "My survey",
"description": "An example survey",
//...
}
Example survey data after:
{
1: "My survey",
2: "An example survey",
...
}
This has several advantages:
- Generated code ensures a consistent schema across web, Android, and Cloud Functions.
- Using field IDs instead of names allows field names can be changes without breaking clients.
- Storaging values as Firestore values rather than binary allows Firestore query filters to work without duplicating values in "index" fields, and it allows data to be viewed and debugged using standard tools.
- The implementation is simple, easy to write and debug.
- Using field numbers instead of names allows this sol'n to work with Android "lite" protobufs in which name bindings aren't available at runtime.
- Writing directly to generated code
This has a few disadvantages as well:
- Maintaining backwards compatibility with existing test surveys is very difficult. We would probably want to zap the dbs and start from scratch.
- Indirection in key names in Firestore; would need to check protobuf def to understand which field is which.
from ground-platform.
Related Issues (20)
- [Migration: protobuf for Firestore] Update Cloud Functions to trigger on new metadata
- [Migration: protobuf for Firestore] Update web app to read LOIs and submissions from new structure
- [Migration: protobuf for Firestore] Update export functions HOT 2
- Update Firestore and Storage rules to read from new schema
- Delete code which uses old schema (Web) HOT 1
- Emails in survey document are visible to all users w/access to survey HOT 1
- Stop exporting user emails and names HOT 3
- [Cloud Functions] Passlist not respected by Cloud Functions
- [Code health] Set up `lerna` to allow dependencies from `functions` and `web` on `lib`
- [Sharing] Allow survey organizers to share by link
- Use consistent language for "data collectors" and "data collection sites"
- Reverse order of "
- [Survey creation flow] Reverse order of sites upload and tasks definition HOT 1
- [LOI editor] Show "free data collection" before "import", which is a more advanced feature
- [LOI editor] Enable "free form" data collection by default
- Data collector can see other's collector's record HOT 5
- Predefine default name for a given LOI HOT 3
- [Submission detail] "No data collected" message unclear
- Missing duplicated jobs in Android
- Handle protobuf conversion edge cases
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ground-platform.