Trigger batch send off of max byte count about serilog-sinks-http

[michaeltdaniels]

Worth noting that datalust/serilog-sinks-seq#12 sounds like the same ask but for a different sink. Unfortunately, it's been open since 2016 with no update.

[FantasticFiasco]

It's been added to my todo-list, and I will investigate it as soon as I've complete something else I'm currently working on. Can't give you a due date, but I will get back to you in this thread when the investigation is complete.

[FantasticFiasco]

A pre-release called 7.3.0-af8c758 now exists on NuGet. A new optional parameter called batchSizeLimitBytes has been added to the extension methods DurableHttpUsingFileSizeRolledBuffers and DurableHttpUsingTimeRolledBuffers. It can be set either using a long or with the helper class ByteSize.

Please let me know if the feature is working as expected.

Just a final heads up. There's one bug in the implementation that I'm aware of. The sink will stop sending log events if any single serialized log event exceeds the threshold of batchSizeLimitBytes. Hopefully this won't be a problem for you during the initial tests.

[FantasticFiasco]

I'm finishing up an application where we can configure the rate of written log events, as well as modifying the size of the messages. Alongside it I've written a simple log server capable of receiving the log events being sent over the network. I think with these two used in conjunction, we can get to the bottom of whether there is a bug here or just a misconfigurarion.

Here is a sneak peak of the "UI". First we have the application writing log events using Serilog.

And then we have the log server.

[github-actions]

Hi there and welcome to this repository!

A maintainer will be with you shortly, but first and foremost I would like to thank you for taking the time to report this issue. Quality is of the highest priority for us, and we would never release anything with known defects. We aim to do our best but unfortunately you are here because you encountered something we didn't expect. Lets see if we can figure out what went wrong and provide a remedy for it.

[FantasticFiasco]

Hi @michaeltdaniels!

You are correct, there is currently no way to configure the sink to send the request when the payload reaches a certain size. The parameters you can tweak are batchPostingLimit (The maximum number of events to post in a single batch. Default value is 1000.) and period (The time to wait between checking for event batches. Default value is 2 seconds.). eventBodyLimitBytes is targeting the size of a single log event and has less to say about the total payload size of the request.

I think it's possible to implement the functionality, but perhaps not as you've requested. I don't think it's possible to send the request when the payload reaches a certain threshold, period and its backoff schedule would still determine when a request would be sent, but what I would be able to expose is a parameter where you would specify a max payload size. It wouldn't be exact, I can only calculate the total size of the events, you'd have to accommodate for the extra bytes of data created by the batch formatter.

However, even though I think this is possible, it would be a breaking change and it wouldn't be easy to test.

I think we need more traction on this feature to get it implemented. I'll leave the issue open, lets see if other developers are experiencing the same problem and wish for its implementation.

I would like to thank you for bringing this issue to my attention, it's now one of the requirements I have to abide by in the future development of this sink.

[dusse1dorf]

Hey @FantasticFiasco,

I'd like to drop in and give a +1 for this issue! As Michael mentioned above, it feels a bit like guesswork trying to ensure that the existing batchPostingLimit doesn't cause us to go over our eventBodyLimitBytes. Instead having a batchPostingLimit specified in bytes that otherwise works the same as the existing setting would prevent silently dropped log messages. To avoid it being a breaking change, could you have it default to being null, where leaving it unspecified maintains current functionality?

[FantasticFiasco]

Sooooo, it's been a long time with no action from my end. Sorry about that. But this is a new year, and with the new year comes new priorities. So this one got promoted. Yay!

I've gotten pretty far with the implementation. I can't solve it according to the requirements of the original author. Instead I've opted to solve it as previously described. With this in mind, is there anybody willing to try a pre-release and provide feedback whether this feature is useful or not?

[dusse1dorf]

No worries on the delay! I tried my hand at implementing the feature at one point and ran into a similar issue =)

As described in that reply, I think that implementation will work for my use case. I can definitely try out a pre release version in my dev system whenever it's available!

[FantasticFiasco]

I was hoping to release a beta to you today, but I found a new edge case and don't want to give you something I haven't tested myself. I'll continue with solving the edge case and doing some initial testing tomorrow, and will get back to you then.

[dusse1dorf]

Got a chance to mess around with this a bit today, seems to be working as expected!

[FantasticFiasco]

Nice! I'm currently looking at another issue that would result in a breaking change. I'll have to investigate if we can release these two issues at the same time.

[vaibhavepatel]

I also tested this and I believe there might be another issue where the Sink stops pushing logs. Please take a look at the below configuration to see if this is the same issue that you were referring to.

Test scenario:

• 1,000 transactions with a log size of 100KB
• 1,000 transactions with a log size of 1MB
• 1,000 transactions with a log size of 5MB
• Transactions were executed sequentially in the following order 100KB, 1MB, 5MB......

Issue:

• Memory went from 135MB (which is ~average) to 2.1GB and doesn't come back down.
• Out of the expected 3,000 logs, I was only able to see 36.

Configuration

• A custom formatter and custom batch formatter is used.
• The batch formatter accepts a max size of 16MB.

        "Name": "Http",
        "Args": {
          "requestUri": "****",
          "bufferBaseFileName": "Sink.Buffer",
          "bufferFileSizeLimitBytes": 52428800,
          "bufferFileShared": false,
          "retainedBufferFileCountLimit": 10,
          "batchSizeLimitBytes": 16777216,
          "batchPostingLimit": 500,
          "period": "00:00:00.001",
          "textFormatter": "**, **",
          "batchFormatter": "**, **"
        }

[vaibhavepatel]

I also executed the same test I specified above using 'DurableHttpUsingFileSizeRolledBuffers' instead of 'Http'. The sink only seemed to have only pushed 2,154 logs out of the 3,000 logs.

[FantasticFiasco]

This is great! I would love to do the same benchmark to see the issue. Are you able to share the code using a repo here on GitHub?

And just for clarification, in your configuration you are defining the Http sink, but that sink only stores log events in memory and has no configuration for bufferBaseFileName nor batchSizeLimitBytes. Is the configuration correct?

As a side note, I haven't implemented batchSizeLimitBytes for Http, but I guess I ought to for consistency between the sink types.

I'll also make sure to release a new pre-release where the known issue mentioned in previous post has been fixed.

[vaibhavepatel]

You're correct, I think my configuration may not be correct for the test I ran for Http. I think you can ignore my results for that test. When I ran for 'DurableHttpUsingFileSizeRolledBuffers', the configuration and the test scenario was the same as described.

In regards to the code used to run that benchmark, there's some things that I am unable to push, but I did upload a simpler version and uploaded it. I am getting the same behavior within this service as well. Link is here

Let me know if you see any issues with my configuration or anything I may have missed.

Additionally, I did find the following when running the tests again:

• When logs are generated faster than the HTTP sink can push them, it is possible for logs to be overwritten depending on the buffer capacity. If the Http Sink is unable to push the logs fast enough, then the older logs will be dropped to make room for the new ones. It is possible that this might be what is occurring within this scenario, but I can't be 100% sure if there's anything else. In this type of scenario, is there a way to push requests in parallel? (I think this would be another enhancement/issue, but I think it would be beneficial for those that would like it)

[FantasticFiasco]

Thanks! I'll take a look at the sample tomorrow.

[FantasticFiasco]

Two questions.

1. What do you use to drive the request towards your web server? Do you have any scripts you can share, or give me some timings and the rate of your requests? You said you did an iteration of sending a request that triggered a 100KB log event to be written, and then a request that triggered a 1MB log event to be written, and finally a request that triggered a 5MB log event to be written. Am I getting this right?
2. What are you using as a log event receiver on https://localhost:8888/? Is that also a ASP.NET application you've built?

[vaibhavepatel]

1. I am using Postman to drive the requests to the api. Postman has a test runner tool that will execute the transactions sequentially when you save the requests as part of a collection. More details can be found here.

I am not doing anything special in terms of throttling and am just executing the next transaction as the one is completed. I predefined the requests with the different payloads and then set the iteration on the test runner to 1,000 and delay between the two as 0.

2. So, I have an ALB setup with a Fluentd agent running behind it. This agent receives the logs over http and processes them to push to Elasticsearch. I just replaced the ALB endpoint with the localhost.
   I believe in your ReadMe, there's a sample application with Fluentd and you might be able to use that same Fluentd, with an updated endpoint (Here). I'll add something similar to my repo tomorrow when I get the chance.

[FantasticFiasco]

1. Can you give me an estimate on the time it takes for Postman to run the 3000 requests?
2. I think if we can simplify the setup, it would be better. I will create a console application to generate the log events, and then have a ASP.NET application as the receiver of the logs. If you can verify the issue on that setup we are good to go.

[vaibhavepatel]

Yeah, a simplified setup would probably be better. I did multiple tests, here are the two that I found, each transaction took ~100ms:

• Test 1: Total time of 348789 ms
• Test 2: Total time of 362814 ms

[FantasticFiasco]

I've created serilog-sinks-http-diagnostics to diagnose this issue, but I think the repo can be useful in other situations as well. Every reference from now on will be to the applications in that repo.

Now to my analysis. I started the console application with 10 tasks with a rate of 100 log events/s/task, resulting in 1000 log events per second. I also configured the max message size to be 2560 KB (2.5 MB) which will produce log events with a max size of ~5 MB (message template + rendered message = ~double the size).

What I noticed was that the sink wrote more log events to the buffer files than it was able to send over the network, and with a batchSizeLimitBytes of 10 MB many of these these batches didn't contain many log events. The math is simple. With these incredibly large log events, the rate of sending them over the network is lower than the rate of writing them to buffer files. Given that this is the average rate and not a burst, there will be no time for the sink to catch up. Given you have unlimited storage you can save the log events in buffer files, and the log events will eventually be sent over the network, but the time from when the log event was written to a buffer file to when it will be received by the log server will increase as long as the application is running. To me, this is a state we cannot imagine to support.

Seeing that the number of buffer files increased, I paused writing new log events, and sure enough, within a few minutes the sink caught up and all log events where sent to the log server. Please note that I paused the console application before overflowing the buffer files.

Can you redo your tests using the applications found in the diagnostics repo? The question I would like to find an answer for is whether this new functionality of being able to restrict payload sizes has a bug in it, as you hinted to earlier.

[vaibhavepatel]

Awesome, thanks for looking into this issue! You were able to reproduce the behavior that I reported. When log files are large and logs generated is high, the sink is not able to push the logs over the network fast enough. This is exactly what was happening to me. It is also one of the reasons why suggested adding #166. This can enable us to add more logs into a single payload and potentially decrease the latency for logs sent over network because payload is smaller.

I believe the other problem I was referring to was how the buffer gets overwritten when it's we keep up with the throughput. The sink is not able to keep up and eventually (depending on the buffer file and size of buffer files) the older buffer files will get overwritten. If these overwritten files contain any logs that the sink is not able to send out before getting overwritten then they are lost. I may have caused some confusion by relating it to this feature as this would happen regardless of having the max byte count configured or not. Is this what you wanted me to execute the test for?

[FantasticFiasco]

I think we need some summarisation and bring the issue back to batches (payload) of a specific size. I'll open a new discussion where we can continue with what is happening when an application is generating more log events than it can send over the network.

I haven't found any bug with the implementation. Specifying a batch size which accommodates for the extra bytes added by the batch formatter seems to work. This is a huge improvements for those of us sending our log events to a log server which rejects payloads over a specific size.

I will continue with implementing the same behaviour for the non-durable sink, i.e. the extension Http.

I may have caused some confusion by relating it to this feature as this would happen regardless of having the max byte count configured or not. Is this what you wanted me to execute the test for?

I understand. This is correct and is happening indifferently of whether this new configuration is set. I didn't understand this from the start, and thought you had found a bug in the implementation of the batch size limitation, but as you say your issue was related to buffer overflows and not the batch size limitation.

[FantasticFiasco]

This feature has been released in v8.0.0-beta.1. It will eventually end up in v8.0.0 but not until another couple of features on the roadmap have been implemented. This will take some time, but please use the pre-release until then.