Server can't handle too big files because of grpc limits: StatusCode.RESOURCE_EXHAUSTED, grpc: trying to send message larger than max about bblfshd

[juanjux]

We should be able to change it with:

https://godoc.org/google.golang.org/grpc#MaxRecvMsgSize

We also should should test the performance implications of changing the size. Maybe we could dynamically change the size of the message as part of the protocol?

[juanjux]

From @EgorBu on slack:

size of file to reproduce error:
root@science-3:~/tmp# ls -sh icu4j/main/tests/core/src/com/ibm/icu/dev/test/bigdec/DiagBigDecimalTest.java
424K icu4j/main/tests/core/src/com/ibm/icu/dev/test/bigdec/DiagBigDecimalTest.java

[abeaumont]

@juanjux I don't think that can be part of the protocol since it's a server configuration parameter, which has to be set before receiving protocol messages.
I'll add a CLI option to be able to override that value.

[juanjux]

It could be a previous message type, a control message, this could be interesting if increasing the max size destroys performance for the average request for example, but it the performance is unaffected or only slightly worse i wouldn't care with the hassle of doing this. Also, maybe the server can't change these settings dynamically once its running so that would make this a moot point too.

[bzz]

FYI - for some reason, I was not hit by this issue using a sequential, single thread Golang client over the same dataset.
AFAIK @EgorBu uses Python one in a heavily parallel workflow.

[abeaumont]

@EgorBu can you make that Java file accessible somewhere or provide a link to it?

[abeaumont]

Should be fixed with #58.