ElasticsearchLogShipper has the wrong success criteria for Bulk operations about serilog-sinks-elasticsearch

[megakid]

Due to our monotonic bookmarking in the durable shipper, as noted in the linked issue, it is difficult to know what to do when a single document in the middle of a bulk index fails.

We can use the _id field to stop us inserting duplicates here. If we generate a unique _id when we save the files to the durable file, we can use this in each of our index commands to allow ElasticSearch to either: reject duplicate _ids that are already indexed using the create operation or "upsert" (updating any existing document with the same _id or create if nonexistent) using the index operation. Only if the entire batch passes do we bookmark and continue. Perhaps we could retry a batch a certain number of times before bookmarking it anyway?

[Lanoli]

What's the status on this issue ?
@konste stated on serilog/serilog#599 that a PR was made with NEST 2.0, but I can't seem te find it.
@mivano was working on a feature branch https://github.com/serilog/serilog-sinks-elasticsearch/tree/handlefailures , but this was never finished or merged to dev.

It would be nice to just parse the bulk response and log any failures. Retries are less important to me, as most failures will most likely be the result of malformed requests.

Are there still any plans to continute this work, or will it be discarded?

[konste]

This is custom modification of the official Serilog ES Sink, done for two reasons:

1. To be able to apply modified ES Json Formatter;
2. To modify EmitBatch method so that it records possible server side failures to internal log;

ElasticsearchSinkMod.cs.txt

[Lanoli]

Thanks @konste . I'm using the DurableElasticsearchSink though, the parsing of the response happens in the ElastsearchLogShipper. Which means I'ld have to copy a lot more code. I'll first try fixing the problem in my fork. I see you are using Nest to parse the response to BulkResponse. Something I already tried and which wasn't 100% correct. So I'll look into that. Problem is the overlap with @mivano 's work, of which I wonder why it was discontinued.

[Lanoli]

PR : #110

[mivano]

It just lost interest but we had indeed some ideas how to act on failures. So at least check the result from ES by parsing the return data. If there is a failure in there (can even be one of the 50 in the batch) we allowed to either send it to selflog or another sink.
If interest is there, I can port it to the current version.

[mivano]

Since v5.5 there have been some improvements in this area.