Streaming RPC about go

[ugorji]

Why do you believe a streaming RPC is necessary? What use case would it solve? And how would you expect it to look like?

[tejorupan]

A typical use-case is retrieving 100s or 1000s of records of data from a database using SQL, in a loop. We may not know how many rows are going to be retrieved nor we would like to store all the records in memory and do the marshalling of the whole data, which may not be memory efficient.

It would be desirable to retrieve each row and marshall it out immediately in a loop, to emit data as a continuous stream

While data can be streamed out in a loop, provision to emit a header at the beginning and a footer at the end of stream is desirable.

Example:
var data = [ // header
{"a"=1, "b"=2}, // row 1
......
{"a"=3, "b"=4}, // row n
]; // footer

[ugorji]

This was why I asked.

You can already do this.

So I'd have to see a real use-case that shows that this isn't possible.

In some C++ libraries, they have different requirements, because they try to read the data, store it into a memory container (ie byte array, etc), and they decode that. So they need a separate streaming API, and separate ASYNC api, etc. However, Go library allows you decode directly from any inputstream (anything implementing io.Reader), and async is implemented under-the-hood and exposed via a sync API.

In theory, you can have a stream with 200 entities encoded in sequence, and create an encoder, and keep decoding from the stream until the stream is finished.

That's one reason why none of the Go encoding libraries (JSON, Gob, Msgpack, etc) have a separate streaming API, or net/rpc has a streaming API.

If you look at the msgpack rpc spec, there's no streaming API built into the spec. I don't see a way to even build it. I don't see a clear API that is needed to enable a use-case.

If you still think this is necessary, please provide a use-case which is not currently supported by the current API, and what API you believe would be necessary to support it. That will help me see what you are alluding to.

[tejorupan]

Another use-case is to receive a request over RPC to transmit data (of file(s), which could be 1 or 2 GiB size). Obviously the data can not be sent in one response packet. So, we end up opening the network connection directly and pump out the contents. While doing so, RPC package is expected not to interfere, which is not perfect. Or we end up avoiding RPC in such scenarios.

For RPC, the receiver methods can possibly take io.Reader/Writer or chan as input argument

[ugorji]

Hi @tejorupan it seems your comment was cut and didn't transmit completely. Can you retype it? I would love to see where you were going with it.

Thanks.

[tejorupan]

Another use-case is to receive a request over RPC to transmit data (of file(s), which could be 1 or 2 GiB size). Obviously the data can not be sent in one response packet. So, we end up opening the network connection directly and pump out the contents. While doing so, RPC package is expected not to interfere, which is not perfect. Or we end up avoiding RPC in such scenarios.

For RPC, the receiver methods can possibly take io.Reader/Writer or chan as input argument, to deal with the streaming. The exact naming of API we will get into later, once you are convinced.

[ugorji]

A "suggested" naming of the API and how it would be used to solve a use-case more elegantly than the currently provided constructs will help make your case.

My RPC support just provides a codec binding for binc/msgpack that integrates with net/rpc. There is no way to do anything that is not natively supported by net/rpc.

Beyond that, the use-case you provided involves bundling RPC communication and a file for download on the same stream. I don't think that would be a good architecture for solving this problem.

Most folks that request streaming APIs for RPC, etc typically want the ability to stream an array of results, processing them one at a time instead of waiting for the whole array to be populated first. The RPC functionality currently doesn't allow that. And other codecs with non-dependent RPC support (e.g. avro, thrift) have traditionally had a very hard time justifying the need, designing an API, and implementing support, for streaming RPC.

See:
http://highperformancesystem.wordpress.com/2012/11/28/xml-json-protocol-buffer-thrift-and-avro/
https://issues.apache.org/jira/browse/AVRO-406

[ugorji]

I did some more research to find out about any thought put into streaming RPC in the net/rpc library.

See:
https://groups.google.com/d/topic/golang-dev/q-xC7SnHYkY/discussion
https://groups.google.com/d/topic/golang-nuts/zQdDCkPfyuU/discussion

The summary of both is that streaming support for net/rpc was considered, but Rob thought it is very tricky to get right and hasn't seen it done right before. No support has been added.

I think you should consider opening up an issue on the Go issue tracker, because that is where streaming support should be. My codec library just creates a codec for binc/msgpack that integrates with net/rpc. The full usage model is controlled from the net/rpc package.

[ugorji]

@tejorupan

The latest commit 4f88b73 contains some code to alleviate the problem. You can build a custom streaming
solution that allows you perform both RPC and non-RPC functions on the same connection. We did this by
providing access to the buffered reader and writer.

It's not a full RPC streaming solution, but allows you construct a custom solution for your specific use-case.

[tejorupan]

You may find the following two interesting from the streaming RPC perspective
https://github.com/flynn/rpcplus‎ which is forked from https://github.com/youtube/vitess/tree/master/go/rpcplus‎
They use chan.

[ugorji]

@tejorupan

I looked at the vitess code. They forked net/rpc and added streaming support to it.

That's not the job of the codec library. Its the job of the layer that calls the codec library.

You may want to file an issue with the go net/RPC library to push them to add streaming support.

[tejorupan]

Thanks for your feedback.
I will file an enhancement feature request with go net/RPC.

[tejorupan]

I opened a new issue: 6569 ( https://code.google.com/p/go/issues/detail?id=6569 ) for this.
Kindly provide your support to steer it forward.
Thank you.