Comments (16)
There are other projects similar to lumberjack that support redis:
I don't anticipate supporting redis. The lumberjack protocol has requirements that redis doesn't support (compression, encryption, message acknowledgement). In fact, lumberjack probably wouldn't exist if the redis developers were amenable to patches that provided compression, encryption, etc, but they refuse.
from logstash-forwarder.
(closing, but still willing to discuss options!)
from logstash-forwarder.
Would you consider patches that add Redis support (without compression, encryption nor message acknowledgment)? Lumberjack is not interesting only because of its protocol but by being fast and small. Other solutions with Redis support are written in a slower language.
from logstash-forwarder.
I'm still open to discussion, but I'm confident that the lumberjack protocol gives us more benefit than using redis, so I'll throw a bunch of questions at you -
What is a "slower language"? Given it's easy to write slow or buggy code in any language, why are we blaming the language and not a bug in the shipper code or design?
There are so many shipping options available out there. Beaver, logstash, rsyslog, and many many others.
What shipper(s) have you tried and what was too slow about them? Were they slow on purpose? If not, can we fix those bugs?
What benefits are added by adding redis support? What about the lumberjack protocol doesn't do what you need? Can we fix the protocol?
from logstash-forwarder.
As for Redis, I see two advantages to have a broker:
- I can restart logstash without losing messages.
- I can add more logstash without reconfiguring every client.
I am unsure if logstash-forwarder could fix the first point. Is it the spool-size
?
I have tried Beaver but it is quite slow at reading very big files (which is problematic for my use case, it is reading files slower that they are being written). It is also known to suffer from a memory leak. I was told to try the Perl equivalent. This could be a solution. I didn't consider rsyslog. As far as I know, I cannot easily attach tags and things like that to preprocess the logs. But maybe I am wrong.
from logstash-forwarder.
- the lumberjack protocol uses reliable messaging to ensure messages are not lost.
- you can achieve this today by using dns to list downstream logstash servers (logstash-forwarder resolves dns and picks an address from that list). Alternately, you can use haproxy.
Regarding 'fixing the first point' - the lumberjack protocol has application-level message acknowledgements to prevent loss of messages. If you see message loss, it's either a bug or something else going wrong.
Regarding beaver, have you raised a bug on the project? These certainly sound like bugs if it's slow at reading files and additionally leaks memory.
Regarding rsyslog, you would be using it in this case for the same purpose as beaver or logstash-forwarder. Simply ship logs from files to logstash just like anything else.
from logstash-forwarder.
For 1, what happens if you get a "connection refused" or something like that? Is the message dropped? If it is buffered, how can I control how much buffer to use?
For 2, you are right, haproxy would work just fine.
For beaver, the memory leak is a known problem: python-beaver/python-beaver#186. For rsyslog, the inability to add types and tags is limiting compared to beaver and logstash-forwarder.
from logstash-forwarder.
On connection refused, logstash-forwarder will simply retry another server (possibly the same one). No messages are lost. Messages are only forgotten once they are acknowledged by a downstream server. There's little or no buffering in a typical sense because the files you are reading with lsf are the buffer/queue.
from logstash-forwarder.
OK, I thought that there were internal buffering but instead, the whole chain will just wait if we cannot send messages to logstash. This is all fine by me then.
from logstash-forwarder.
I also prefer redis support.
from logstash-forwarder.
When I tried to use logstash-forwarder as simple forwarder to logstash parser, it used just one logstash parser (but I defined several). So performance became very low.
schematically it was:
logstash-forwarder -> logstash cluster with elasticsearch (actually just one logstash server from cluster)
When I tried to forward logs to local logstash (I don't need ssl for that, so it would be great if I could disable SSL) without any parser (just send logs to redis, the rest logstash parser cluster transfer logs from redis) - performance increased.
schematically it was:
logstash-forwarder -> logstash -> redis -> (logstash cluster with elasticsearch)
So I guess if logstash-forwarder could just send logs to redis, performance became much higher. And it will be no necessary to keep logstash instance for redis logs writing.
schematically:
logstash-forwarder -> redis -> (logstash cluster with elasticsearch)
from logstash-forwarder.
This project is intended to be light weight, simple, secure, reliable, and fast. Adding additional message transport options (redis, rabbitmq, etc) adds weight and complexity.
Most of the requests I see to have lsf output to other transports are based on ideas that seem to mistake bugs for intentional behaviors. lsf is supposed to be fast over the lumberjack protocol. If it isn't, either the downstream (logstash) is slow, or there's a bug. We can fix bugs without adding complexity (new transports).
from logstash-forwarder.
I apologize if this isn't the right forum for this, but this issue seemed to closely related to my interest. I'm wondering if the pure logstash fowarder + lumberjack model addresses the buffering / event spike concern and if not what the best practice for doing so is. I get the sense from @jordansissel's comments that a queue is unnecessary (but I could be misinterpreting his intent). Should I assume the lumberjack input has a built in mechanism for buffering events it can't process immediately? Thanks in advance.
from logstash-forwarder.
Hi @wjimenez5271 - forwarder waits for logstash to finish the spool of events before sending the next. Thus it will slow down as needed. Only issue at the moment is if logstash runs really slow due to huge bulks of events, forwarders can timeout and reconnect faster than logstash realises and cause it to crash. If you increase network timeout enough, and/or reduce spool size on forwarders enough - you can live completely without a redis queue. This only applies to instances where spikes cause the logstash processing time to be longer than the network timeout. For light loads everything is fine
My PR #180 made it so you don't need to reduce the spool size or increase network timeouts. I have since forked my changes into my own project, log-courier, so I've stopped maintaining that PR and closed it.
from logstash-forwarder.
Redis has a very useful role in large scale logstash environments; it can act as a buffer between the agents and the ES nodes; where the indexer can run against the data in Redis without issue.
Your LSF would be very handy if it could send unencrypted data to Redis, as then I could use it on a number of windows servers to send their log data to Redis for the above architecture. Without that feature, I can not use your tool in that architecture.
Please reconsider adding this feature to LSF.
from logstash-forwarder.
@rcaston Logstash already supports this (redis output). Recommend you use that.
from logstash-forwarder.
Related Issues (20)
- Logstash Forwarder Keeps Rotated Files Open HOT 5
- Beta4 -> RC1, send fail
- logstash-forwarder tcp input HOT 4
- rotate logstash forwarder's log HOT 4
- Logstash-forwarder cannot get data from the beginning. HOT 2
- way to override 24hr inactivity in a file harvester? HOT 2
- logstash forwarder HOT 1
- how to exclude files using pattern HOT 1
- tailing log file not registering events (logstash forwarder with -tail=true) HOT 1
- Make offset field in LSF optional HOT 1
- Filebeat - transport.go:125: SSL client failed to connect with: read tcp connection reset by peer HOT 1
- logstash multiline codec not working as expected
- how to set logstash-forward encoding HOT 1
- Received fatal alert: bad_certificate HOT 2
- Minor miss in lumberjack documentation HOT 2
- Logstash-forwarder exclude gzip files HOT 7
- c HOT 1
- Run on the ARM (RaspberryPi) HOT 1
- Update Lumberjack documentation to latest
- How to update logstash 8.9.1 to 8.12.02
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logstash-forwarder.