isabella232 / proton-1

cli protobuf to json converter

proton

CLI protobuf to json converter.

Installation

Execute:

$ go install github.com/beatlabs/proton/v2@latest

Or download from Releases

Or using Homebrew 🍺

brew tap beatlabs/proton https://github.com/beatlabs/proton
brew install proton

Usage

Usage:
  proton json [flags]

Flags:
  -m, --end-of-message-marker string   Marker for end of message used when piping data
  -f, --file string                    Proto file path or url
  -h, --help                           help for json
      --indent                         Indent output json
  -p, --package string                 Proto package
                                       Defaults to the package found in the Proton file if not specified
  -t, --type string                    Proto message type
                                       Defaults to the first message type in the Proton file if not specified

Examples

Proto file from URL with input message as argument

proton json -f https://raw.githubusercontent.com/protocolbuffers/protobuf/master/examples/addressbook.proto testdata/out.bin

Proto file from local with input message as argument

proton json -f ./testdata/addressbook.proto testdata/out.bin

Proto file from URL with input message piped

cat testdata/out.bin | proton json -f https://raw.githubusercontent.com/protocolbuffers/protobuf/master/examples/addressbook.proto

Proto file from local with input message piped

cat testdata/out.bin | proton json -f ./testdata/addressbook.proto

Multiple proto files from a producer with input messages piped

./testdata/producer.sh '--END--' | proton json -f ./testdata/addressbook.proto -m '--END--'

Usage with Kafka consumers

Because Proto bytes can contain newlines (\n) and often do, we need to use a different marker to delimit the end of a message byte-stream and the beginning of the next. Proton expects an end of message marker, or will read to the end of the stream if not provided.

You can add markers at the end of each messae with tools like kafkacat, like so:

kcat -b my-broker:9092 -t my-topic -f '%s--END--'

You can consume messages and parse them with Proton by doing the following:

kcat -b my-broker:9092 -t my-topic -f '%s--END--' -o beginning | proton json -f ./my-schema.proto -m '--END--'

Don't see messages?

If you execute the above command, but you don't see messages until you stop the consumer, you might have to adjust your buffer settings: You can do this with the stdbuf command.

stdbuf -o0 kcat -b my-broker:9092 -t my-topic -f '%s--END--' -o beginning | proton json -f ./my-schema.proto -m '--END--'

If you don't have stdbuf, you can install it via brew install coreutils.