doytsujin / quickbuffers

Protobuf 2 library suitable for real-time enviroments

QuickBuffers - Fast Protocol Buffers without Allocations

QuickBuffers is a Java implementation of Google's Protocol Buffers v2 that has been developed for low latency and high throughput use cases. It can be used in zero-allocation environments and supports off-heap memory.

The main differences to Protobuf-Java are

• All parts of the API are mutable and reusable
• A roughly 2x performance improvement in both encoding and decoding speed[1]
• A JSON serializer that matches the Proto3 JSON Mapping
• A serialization order that was optimized for sequential memory access
• Significantly smaller code size than the java and javalite options
• No reflection API or any use of reflections (no ProGuard config needed)

Unsupported Features

• Maps can be used with a workaround
• Extensions and Services are currently not supported
• Unknown fields are retained as raw bytes and cannot be accessed as fields

[1] The performance benefits depend heavily on the use case and message format, but most common use cases should see a roughly 2x performance improvement in both encoding and decoding speed over Protobuf-Java 3.9.1. For more information see the benchmarks section.

Runtime Library

You can find the latest release on Maven Central at the coordinates below. The runtime is compatible with Java 6 and higher.

<dependency>
  <groupId>us.hebi.quickbuf</groupId>
  <artifactId>quickbuf-runtime</artifactId>
  <version>1.0-alpha3</version>
</dependency>

Be aware that this library is currently still in a pre-release state, and that the public API should be considered a work-in-progress that may be subject to (likely minor) changes.

Building from Source

The project can be built with mvn package using JDK8 through JDK11.

Note that protoc plugins get started by the protoc executable and exchange information via protobuf messages on std::in and std::out. While this makes it fairly simple to get the schema information, it makes it quite difficult to setup unit tests and debug plugins during development. To work around this, the parser module contains a tiny protoc-plugin that stores the raw request from std::in inside a file that can be loaded in unit tests during development of the actual generator plugin.

For this reason the generator modules requires the packaged output of the parser module, so you always need to run the package goal. mvn clean test will not work.

Generating Messages

The code generator is setup as a protoc plugin that gets called by the official protobuf compiler. You can either generate the message sources manually, or use build system plugins to generate the sources automatically each time.

Manual Generation

• Download an appropriate protoc.exe and add the directory to the $PATH (tested with protoc-3.7.0 through protoc-3.9.1)
• Download protoc-gen-quickbuf-1.0-alpha3 and extract the files into the same directory or somewhere else on the $PATH.
  • Running the plugin requires Java8 or higher to be installed
  • Protoc does have an option to define a plugin path, but it does not seem to work with the wrapper scripts
• Call protoc with --quickbuf_out=<options>:./path/to/generate

Maven Configuration

The configuration below downloads the QuickBuffers generator plugin, puts it on the correct path, and executes protoc using the protoc-jar-maven-plugin. The default settings assume that the proto files are located in src/main/protobuf.

<build>
    <plugins>

        <!-- Downloads QuickBuffers generator plugin -->
        <plugin>
            <artifactId>maven-antrun-plugin</artifactId>
            <version>1.8</version>
            <executions>
                <execution>
                    <id>download-quickbuf-plugin</id>
                    <phase>generate-sources</phase>
                    <configuration>
                        <tasks>
                            <get src="https://github.com/HebiRobotics/QuickBuffers/releases/download/1.0-alpha3/protoc-gen-quickbuf-1.0-alpha3.zip"
                                 dest="../protoc-gen-quickbuf.zip" skipexisting="true" verbose="on"/>
                            <unzip src="../protoc-gen-quickbuf.zip" dest=".." overwrite="false"/>
                        </tasks>
                    </configuration>
                    <goals>
                        <goal>run</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

        <!-- Calls protoc.exe and generate messages -->
        <plugin>
            <groupId>com.github.os72</groupId>
            <artifactId>protoc-jar-maven-plugin</artifactId>
            <version>3.8.0</version>
            <executions>
                <execution>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>run</goal>
                    </goals>
                    <configuration>
                        <protocVersion>3.9.1</protocVersion>

                        <!-- plugin configuration, options, etc. -->
                        <outputTargets>
                            <outputTarget>
                                <type>quickbuf</type>
                                <outputOptions>store_unknown_fields=false</outputOptions>
                            </outputTarget>
                        </outputTargets>

                    </configuration>
                </execution>
            </executions>
        </plugin>

    </plugins>
</build>

Currently available options are

• indent sets the indentation in generated files
• replace_package allows replacing the Java package of the generated messages to avoid name collisions with messages generated by --java_out
• input_order enables an optimization that improves decoding performance when parsing messages that were serialized in a known order
  • quickbuf expects fields to arrive sorted by type and their ascending number (default)
  • number expects fields to arrive sorted by only the ascending number (official implementations)
  • none disables this optimization (not recommended)
• store_unknown_fields stores unknown fields that it encounter during parsing. This allows messages to be passed on without losing information even if the schema is not fully known
  • unknown fields are stored in binary form, so individual fields cannot be accessed directly
  • unknown fields are ignored when comparing with equals
• enforce_has_checks throws an exception when accessing fields that were not set
• json_use_proto_name changes the serialized json field names to match the original proto definition (my_field) instead of the default lowerCamelCase (myField) or json_name override option. Compatible parsers should be able to parse both cases.

Option	Value
indent	2, 4, 8, tab
replace_package	(pattern)|replacement
input_order	quickbuf, number, none
store_unknown_fields	false, true
enforce_has_checks	false, true
json_use_proto_name	false, true

For example,

protoc --quickbuf_out= \
    indent=4, \
    input_order=quickbuf, \
    replace_package=my.namespace.protobuf|my.namespace.quickbuf: \
    ./path/to/generate`.

Basic Usage

Overall, we tried to keep the public API as close to Google's Protobuf-Java as possible, so most use cases should require very few changes. The main difference is that there are no builders, and that all message contents are mutable.

All nested object types such as message or repeated fields have getField() and getMutableField() accessors. Both return the same internal storage object, but getField() should be considered read-only. Once a field is cleared, it should also no longer be modified.

Primitive Fields

All primitive values generate the same accessors and behavior as Protobuf-Java's Builder classes

// .proto
message SimpleMessage {
    optional int32 primitive_value = 1;
}

// simplified generated code
public final class SimpleMessage {
    public SimpleMessage setPrimitiveValue(int value);
    public SimpleMessage clearPrimitiveValue();
    public boolean hasPrimitiveValue();
    public int getPrimitiveValue();

    private int primitiveValue;
}

Message Fields

Nested message types are final and allocated during construction time. Setting the field copies the internal data, but does not change the reference, so the best way to set nested message content is by directly accessing the internal store with getMutableNestedMessage().

// .proto
message NestedMessage {
    optional int32 primitive_value = 1;
}
message RootMessage {
    optional NestedMessage nested_message = 1;
}

// simplified generated code
public final class RootMessage {
    public RootMessage setNestedMessage(NestedMessage value); // copies contents to internal message
    public RootMessage clearNestedMessage(); // clears has bit as well as the backing object
    public boolean hasNestedMessage();
    public NestedMessage getNestedMessage(); // internal message -> treat as read-only
    public NestedMessage getMutableNestedMessage(); // internal message -> may be modified until has state is cleared

    private final NestedMessage nestedMessage = NestedMessage.newInstance();
}

// (1) setting nested values via 'set' (does a data copy!)
msg.setNestedMessage(NestedMessage().newInstance().setPrimitiveValue(0));

// (2) modify the internal store directly (recommended)
RootMessage msg = RootMessage.newInstance();
msg.getMutableNestedMessage().setPrimitiveValue(0);

String Fields

Java String objects are immutable, so the API differs from Protobuf-Java in that accessors accept CharSequence arguments and return StringBuilder objects instead. StringBuilder can be converted via toString(), but you may want to use a StringInterner to share references if you receive many identical strings.

// .proto
message SimpleMessage {
    optional string optional_string = 2;
}

// simplified generated code
public final class SimpleMessage {
    public SimpleMessage setOptionalString(CharSequence value); // copies data
    public SimpleMessage clearOptionalString(); // sets length = 0
    public boolean hasOptionalString();
    public StringBuilder getOptionalString(); // internal store -> treat as read-only
    public StringBuilder getMutableOptionalString(); // internal store -> may be modified 

    private final StringBuilder optionalString = new StringBuilder(0);
}

// Set and append to a string field
SimpleMessage msg = SimpleMessage.newInstance();
msg.setOptionalString("my-");
msg.getMutableOptionalString()
    .append("text"); // field is now 'my-text'

Repeated Fields

Repeated scalar fields work mostly the same as String fields, but the internal array() can be accessed directly if needed. Repeated messages and object types provide a next() method that adds one element and provides a mutable reference to it.

// .proto
message SimpleMessage {
    repeated double repeated_double   = 42;
}

// simplified generated code
public final class SimpleMessage {
    public SimpleMessage addRepeatedDouble(double value); // adds one value
    public SimpleMessage addAllRepeatedDouble(double... values); // adds N values
    public SimpleMessage clearRepeatedDouble(); // sets length = 0
    public boolean hasRepeatedDouble();
    public RepeatedDouble getRepeatedDouble(); // internal store -> treat as read-only
    public RepeatedDouble getMutableRepeatedDouble(); // internal store -> may be modified 

    private final RepeatedDouble repeatedDouble = RepeatedDouble.newEmptyInstance();
}

Note that repeated stores can currently only expand, but we may add something similar to StringBuilder::trimToSize to get rid of unneeded memory (TODO).

Serialization

Messages can be read from a ProtoSource and written to a ProtoSink. At the moment we only support contiguous blocks of memory, i.e., byte[].

// Create data
RootMessage msg = RootMessage.newInstance()
    .setPrimitiveValue(2);

// Serialize into existing byte array
byte[] buffer = new byte[msg.getSerializedSize()];
ProtoSink sink = ProtoSink.newInstance(buffer);
msg.writeTo(sink);

// Serialize to byte array using helper method
assertArrayEquals(msg.toByteArray(), buffer);

// Read from byte array into an existing message
ProtoSource source = ProtoSource.newInstance(buffer);
assertEquals(msg, RootMessage().newInstance.mergeFrom(source));

Note that ProtoMessage::getSerializedSize sets an internally cached size, so it should always be called before serialization.

Off-Heap Addressing

Depending on platform support, the implementation may make use of sun.misc.Unsafe. If you are familiar with Unsafe, you may also request an UnsafeSource instance that will allow you to use off-heap addresses. Use with caution!

long address = /* DirectBuffer::address */;
ProtoSource source = ProtoSource.newUnsafeInstance();
source.setInput(null, address, length)