Sending large number of messages via async method causes loss of data about graphyte

[benhoyt]

Interesting. I presume you're using a large (>10,000) queue_size? Or are you seeing a lot of queue full when sending error logs? If you are able to provide a small case that reproduces the problem, that'd be helpful.

[jclendenan]

Queue_size was set to 100,000 so we weren't getting queue full when sending errors. (I saw that before)

I think we just need to break the message stream up into batches of ~ 1000 or so in the send_socket method. I suspect there is a upper limit on how many single line records the carbon relay can receive in a single shot.

I'll see if I can do up a test case later on for this.

[jclendenan]

ok, so it's a bit of a sloppy commit with some bad tabs, but we got there in the end. works in my production environment.

Please review the tests etc for coverage etc.