GithubHelp home page GithubHelp logo

hebirobotics / quickbuffers Goto Github PK

View Code? Open in Web Editor NEW
95.0 95.0 10.0 2.07 MB

Java Protobuf implementation suitable for real-time enviroments

License: Apache License 2.0

Java 99.97% Batchfile 0.01% Shell 0.01%
java protobuf protocol-buffers

quickbuffers's People

Contributors

dependabot[bot] avatar ennerf avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

doytsujin krystalics michaelandrepearce josepowera icodein

quickbuffers's Issues

Add API for streaming disk access

It's cumbersome to work with streaming file I/O due to requiring a contiguous block of memory and native ByteBuffers only being supported via Unsafe.

We should add some way for buffered I/O and maybe better support for direct byte buffers where unsafe is not available. Maybe to start off with we could just have an adapter for Protobuf-Java's Coded-Streams.

API Notes

  • ProtoSink::reset should return this
  • Maybe add defines for FIELD_XYZ_NUMBER

java.lang.StackOverflowError

Hi,

I am trying to use quickbuf with my Google Adx protobuf file but getting java.lang.StackOverflowError.
https://developers.google.com/authorized-buyers/rtb/downloads/openrtb-proto
https://developers.google.com/authorized-buyers/rtb/downloads/openrtb-adx-proto

command used:

/Users/test/protoc-21.12-osx-aarch_64/bin/protoc --quickbuf_out=indent=4,java8_optional=true,input_order=quickbuf:.  *.proto

Error:

Exception in thread "main" java.lang.StackOverflowError
	at com.oracle.svm.core.graal.snippets.StackOverflowCheckImpl.newStackOverflowError0(StackOverflowCheckImpl.java:328)
	at com.oracle.svm.core.graal.snippets.StackOverflowCheckImpl.newStackOverflowError(StackOverflowCheckImpl.java:324)
	at com.oracle.svm.core.graal.snippets.StackOverflowCheckImpl.throwNewStackOverflowError(StackOverflowCheckImpl.java:304)
	at com.oracle.svm.core.graal.jdk.SubstrateArraycopySnippets.boundsCheck(SubstrateArraycopySnippets.java:117)
	at com.oracle.svm.core.graal.jdk.SubstrateArraycopySnippets.doArraycopy(SubstrateArraycopySnippets.java:94)
	at java.lang.String.getBytes(String.java:4477)
	at java.lang.AbstractStringBuilder.putStringAt(AbstractStringBuilder.java:1720)
	at java.lang.AbstractStringBuilder.putStringAt(AbstractStringBuilder.java:1724)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:583)
	at java.lang.StringBuilder.append(StringBuilder.java:179)
	at us.hebi.quickbuf.generator.TypeRegistry.resolveMessageType(TypeRegistry.java:87)
	at us.hebi.quickbuf.generator.TypeRegistry.resolveJavaTypeFromProto(TypeRegistry.java:77)
	at us.hebi.quickbuf.generator.RequestInfo$FieldInfo.getTypeName(RequestInfo.java:445)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	... <SAME LINE>
	... <SAME LINE>
	... <SAME LINE>
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
	at us.hebi.quickbuf.generator.TypeRegistry.hasRequiredFieldsInHierarchy(TypeRegistry.java:130)
--quickbuf_out: protoc-gen-quickbuf: Plugin failed with status code 1.

Use optimized protobuf parsing/writing

I took a look at this project and am interested in trying it out. Before that, I also looked at your benchmarks, and while I agree that their code is the most idiomatic one, it's definitely not the fastest one possible. For example, there's a trick for faster protobuf parsing: https://stackoverflow.com/questions/21854687/improving-performance-of-protocol-buffers/21870564#21870564

Instead of using readDelimitedFrom(), construct a single CodedInputStream and use it to read multiple messages like so:

// Do this once:
CodedInputStream cis = CodedInputStream.newInstance(input);

// Then read each message like so:
int limit = cis.pushLimit(cis.readRawVarint32());
builder.mergeFrom(cis);
cis.popLimit(limit);
cis.resetSizeCounter();

Similarly, when throughput is actually critical, we're not writing protobufs one-by-one, creating a CodedOutputStream for every object. We create a single CodedOutputStream and write as many objects into it as possible in a loop / until a buffer is full:

codedOutputStream.writeUInt32NoTag(message.getSerializedSize());
message.writeTo(codedOutputStream);

Would you be interested in accepting a PR adding benchmarks with these? Do you think we can improve the current benchmarks? Or do you perhaps think such benchmarks would not be representative?

No such file or directory(base)

hi,
im manually running quickbuf with protoc 3.8 on ubuntu20, and got this error:

protoc-3.8.0-linux-x86_64.exe --quickbuf_out=java8_optional=false: market_events.proto
: No such file or directory(base)

i followed the instruction and put the quickbuf code in the same dir as protoc-3.8.0-linux-x86_64.exe and added it to path.
could you give a bit more detailed instructions on how to manually run it? thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.