vklochan / python-logstash Goto Github PK
View Code? Open in Web Editor NEWPython logging handler for Logstash.
License: MIT License
Python logging handler for Logstash.
License: MIT License
This would be very helpful with topic exchanges
Got an error with logging inner json:
test_logger.info('python-logstash: test extra fields', extra=extra)
class Test:
def __init__(self):
self.a = 1
self.b = 1
def __repr__(self):
return "Test(a={}, b={})".format(self.a, self.b)
test_logger.info("test inner json", extra={"meta": {"test": Test()}})
Gives:
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/handlers.py", line 577, in emit
s = self.makePickle(record)
File "/home/stas/workspace/partners/python-logstash/logstash/handler_udp.py", line 17, in makePickle
return self.formatter.format(record)
File "/home/stas/workspace/partners/python-logstash/logstash/formatter.py", line 142, in format
return self.serialize(message)
File "/home/stas/workspace/partners/python-logstash/logstash/formatter.py", line 83, in serialize
return json.dumps(message)
File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python2.7/json/encoder.py", line 184, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: Test(a=1, b=1) is not JSON serializable
Just had to work this out for myself. If you're dependent on logstash for keeping records, then unhandled exceptions should really be shipped there too. I did this by calling the method below in my setup:
def config_global_exception_handling():
'''
Override the global exception hook to come here.
Here, log the exception which includes sending to logstash.
'''
def customexcepthook(exc_type, exc_value, exc_traceback):
sys.exc_info = lambda: (exc_type, exc_value, exc_traceback)
logging.exception('Unhandled exception')
sys.excepthook = customexcepthook
"Unhandled exception" becomes the message attribute, and the stack_trace attribute gets populated appropriately
python-logstash/logstash/formatter.py
Line 26 in a8be251
Because these are stripped down in the base class, it's somewhat problematic to re-extract them for use in debug and production. Things like lineno
are particularly useful in logging. I would love to see an easier means to bypass this removal, and include them for logstash.
Would be very handy if this was just a flag passed down through the LogstashHandlers.
Hi, the current version on pypi is 0.4.2. Following the installation instructions (pip install python-logstash
) will install that version.
It would be very nice if you could make a new release to pypi, so that it is possible to install the latest version from it.
Thanks!
I am using this logger for a Django project and it seems that whenever I try to log a nested JSON object, it gets silently ignored. The following works fine:
logger.debug('trace', extra={
# 'request': {
# 'path': request.get_full_path()
# },
'path': request.get_full_path(),
})
But this one gets silently ignored:
logger.debug('trace', extra={
'request': {
'path': request.get_full_path()
},
'path': request.get_full_path(),
})
Any ideas?
on pypi last version of package is 0.4.6
0.4.6 version have no ssl feature as I can see... :(
My Logstasah is listening TCP on 5000, and when trying you sample
import logging
import logstash
import sys
host = 'localhost'
test_logger = logging.getLogger('python-logstash-logger')
test_logger.setLevel(logging.INFO)
test_logger.addHandler(logstash.LogstashHandler(host, 5959, version=1))
# test_logger.addHandler(logstash.TCPLogstashHandler(host, 5959, version=1))
test_logger.error('python-logstash: test logstash error message.')
test_logger.info('python-logstash: test logstash info message.')
test_logger.warning('python-logstash: test logstash warning message.')
# add extra field to logstash message
extra = {
'test_string': 'python version: ' + repr(sys.version_info),
'test_boolean': True,
'test_dict': {'a': 1, 'b': 'c'},
'test_float': 1.23,
'test_integer': 123,
'test_list': [1, 2, '3'],
}
test_logger.info('python-logstash: test extra fields', extra=extra)
Nothing Happened , no error , nothing
Even if I changed the port to 500 still nothing.
but simple nc localhost 5000 < /var/log/test.log
it works.
This more me being a noob, but I am guessing that BaseLogstash Handler is the same as the UDP Handler defined? (I'm new to python :P)
https://github.com/vklochan/python-logstash/blob/master/logstash/handler_udp.py#L21
Sorry for any inconvenience :P
Formatter class has tags
argument, but handler class always sets tags to empty list.
Am i missing something or there is no way to set tags if you configure logging with dictConfig (like in Django)?
Hello,
Name: python-logstash
Version: 0.4.6
Summary: Python logging handler for Logstash.
Home-page: https://github.com/vklochan/python-logstash
Author: Volodymyr Klochan
Author-email: [email protected]
License: UNKNOWN
Location: /usr/local/lib/python2.7/dist-packages
Requires:
Using python-logstash through ansible logstash call back I get the following error:
TASK [java/jdk : Préparation répertoire destination Java] ***********************************************************************************************************************************************************************************
changed: [tnr-container]
TASK: java/jdk : Préparation répertoire destination Java
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/handlers.py", line 578, in emit
s = self.makePickle(record)
File "/usr/local/lib/python2.7/dist-packages/logstash/handler_tcp.py", line 25, in makePickle
return self.formatter.format(record) + b'\n'
File "/usr/local/lib/python2.7/dist-packages/logstash/formatter.py", line 136, in format
message.update(self.get_extra_fields(record))
File "/usr/local/lib/python2.7/dist-packages/logstash/formatter.py", line 45, in get_extra_fields
fields[key] = repr(value)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 19: ordinal not in range(128)
Logged from file logstash.py, line 131
The problem comes from the latin1 character present in the handled value.
What should be done from a python client when a Logstash server is restarted?
I'm running a Logstash v2.3.4 container on my local post via a docker image found on docker hub. https://hub.docker.com/_/logstash/
logstash.conf
input {
tcp {
port => 5000
codec => json
}
}
output {
stdout {
codec => rubydebug
}
}
I'm also running a Flask app on my localhost with python-logstash connecting via TCP.
When I restart the container that is running Logstash I no longer see logged messages being displayed to stdout until I restart my Flask server. If anyone can help illuminate what should be done here, or else point me in the right direction, then that would be appreciated.
hi
in the example in the README
extra = {
'test_string': 'python version: ' + repr(sys.version_info),
'test_boolean': True,
'test_dict': {'a': 1, 'b': 'c'},
'test_float': 1.23,
'test_integer': 123,
'test_list': [1, 2, '3'],
}
you mention a list 'test_list': [1, 2, '3'],
, where elements have different types ==> THIS IS NOT WORKING!
../../lib/logger/elkhandler.py:5: in
import logstash
lib/python3.4/site-packages/logstash/init.py:2: in
from event import Event
E ImportError: No module named 'event'
I have installed events and python-logstash using pip and both are satisfied.. I'm using python
python --version
Python 3.4.3
pip install events
Requirement already satisfied: events in /lib/python3.4/site-packages
pip install python-logstash
Requirement already satisfied: python-logstash in /lib/python3.4/site-packages
hi
if python-logstash support send to domain unix sock?
For python 3 LogstashFormatterBase.serialize method returns bytes instead of string. This incosistency results in no possibility in reusing this formatter with buildin handlers like pythons StreamHandlers (e.g. FileHandler), because it expects string object returned from formatter.
For python 2.X LogstashFormatterBase.serialize method behaves correct returning string.
Hi, thanks for this library, so I have some consecutive logs that have the same @timestamp
but looking in Kibana, they are not in the order that they are printed in the code
For example, the first and last logs are in the right order here, but the two in the middle are not:
Has this happened to anyone else and is there a possible fix for this?
Given that this is a utility library, it would make sense to follow semantic versioning.
In fact, the change in 0.4.6
(renaming field names) changed the external API. We mistakenly believed that python-logstash
would follow semantic versioning and pinned our dependencies using python-logstash >= 0.4.2, <1.0
so the 0.4.6
update was included in our updates.
This has now created a bunch of entries in our ELK stack with the wrong field names and our visualisations now show incorrect information. And this was only noticed by pure chance.
It would be nice if the versioning scheme used in python-logstash
would be documented so nobody else falsely assumes that this is semver.
Message i log is nested inside dictionary with key "message", together with other python-logstash
values. Here's raw content i get in kibana:
{
"_index": "logstash-2014.07.06",
"_type": "syslog",
"_id": "Vshw1docTteonciCQktJZA",
"_score": null,
"_source": {
"message": "{\"host\": \"HOSTNAME\", \"logger\": \"test.log\", \"type\": \"logstash\", \"tags\": [], \"path\": \"C:\\\\Users\\\\user\\\\file.py\", \"@timestamp\": \"2014-07-06T07:51:26.004000Z\", \"foo\": \"bar\", \"@version\": 1, \"message\": \"Hello world\", \"levelname\": \"INFO\"}",
"@version": "1",
"@timestamp": "2014-07-06T07:46:31.445Z",
"type": "syslog",
"tags": [
"syslog",
"_grokparsefailure"
],
"host": "10.100.1.1",
"priority": 13,
"severity": 5,
"facility": 1,
"facility_label": "user-level",
"severity_label": "Notice"
},
"sort": [
1404632791445,
1404632791445
]
}
I configured logger more or less like example in readme file.
What am i doing wrong?
I was running through how to set this up, most specifically the receiving part in logstash.
I noticed in your examples you set the handler's target host port to 5959, but in the example logstash configuration it is port 5000.
Additionally, it's mentioned we're sending messages to logstash using UDP, but the logstash configuration example is for a TCP input.
I added "try... except..." to example1.py of this repo as following:
import logging
import logstash
import sys
host = 'localhost'
try:
test_logger = logging.getLogger('python-logstash-logger')
test_logger.setLevel(logging.INFO)
# test_logger.addHandler(logstash.LogstashHandler(host, 5959, version=1))
test_logger.addHandler(logstash.TCPLogstashHandler(host, 5959, version=1))
test_logger.error('python-logstash: test logstash error message.')
test_logger.info('python-logstash: test logstash info message.')
test_logger.warning('python-logstash: test logstash warning message.')
# add extra field to logstash message
extra = {
'test_string': 'python version: ' + repr(sys.version_info),
'test_boolean': True,
'test_dict': {'a': 1, 'b': 'c'},
'test_float': 1.23,
'test_integer': 123,
'test_list': [1, 2, '3'],
}
test_logger.info('python-logstash: test extra fields', extra=extra)
print('Succeed to save event.')
except Exception as e:
print('Failed to save event: ',e)
Even though no Logstash on my localhost, it will say "Succeed to save event", I am wondering how to add error handling of writing to Logstash with TCPLogstashHandler. Thanks.
Currently, the emitted messages only contain level names (f.ex. ERROR
, CRITICAL
, ...). Those names correspond to level values. Those names can be converted to/from their numeric values using logging.getLevelName
. Additionally, the numeric values should be available on the log-record behind the attribute levelno
:
>>> from logging import LogRecord
>>> import logging
>>> r = LogRecord('foo', logging.DEBUG, 'path', 11, 'msg', {}, False)
>>> r.levelno
10
>>> r.levelname
'DEBUG'
Adding the numeric level value makes it much easier to filter on the severity by simply filtering on everything greater than 40
for example.
Hi,
In Elasticsearch the '@Version' field is a string, but because python-logstash sends it as an integer, it is being saved as integer in ES.
Suggested is to send the field's value as string.
Thanks,
Yarden
Since pika uses python logging framework internally, if the root logger is configured with AMQPHandler then a loop is created where pika logs are sent through the amqp handler thus causing more logs to be emitted.
We would like to use the amqp logging handler within tornado, it would be nice if python-logstash supported pika connection adapters.
Here are two other similar libraries, and they both maintain a similar list of blacklisted keys which would conflict with the python log record stuff:
https://github.com/marselester/json-log-formatter/blob/master/json_log_formatter/__init__.py
https://github.com/madzak/python-json-logger/blob/master/src/pythonjsonlogger/jsonlogger.py
The one notable field that those have which this library doesn't have is stack_info
, so I wanted to ask whether this is something that should be added.
It seems this was added to the log record in some python version
Its appearance there seems to be comparable to the other fields
The following error is thrown when configuring logstash for Gunicorn.
Traceback (most recent call last):
File "/usr/lib/python3.5/logging/handlers.py", line 621, in emit
s = self.makePickle(record)
File "~/Projects/env/lib/python3.5/site-packages/logstash/handler_tcp.py", line 25, in makePickle
return self.formatter.format(record) + b'\n'
TypeError: Can't convert 'bytes' object to str implicitly
Changing handler_tcp.py
to:
from logging.handlers import DatagramHandler, SocketHandler
from logstash import formatter
# Derive from object to force a new-style class and thus allow super() to work
# on Python 2.6
class TCPLogstashHandler(SocketHandler, object):
"""Python logging handler for Logstash. Sends events over TCP.
:param host: The host of the logstash server.
:param port: The port of the logstash server (default 5959).
:param message_type: The type of the message (default logstash).
:param fqdn; Indicates whether to show fully qualified domain name or not (default False).
:param version: version of logstash event schema (default is 0).
:param tags: list of tags for a logger (default is None).
"""
def __init__(self, host, port=5959, message_type='logstash', tags=None, fqdn=False, version=0):
super(TCPLogstashHandler, self).__init__(host, port)
if version == 1:
self.formatter = formatter.LogstashFormatterVersion1(message_type, tags, fqdn)
else:
self.formatter = formatter.LogstashFormatterVersion0(message_type, tags, fqdn)
def makePickle(self, record):
return str.encode(self.formatter.format(record)) + b'\n'
Specifically the last line:
return str.encode(self.formatter.format(record)) + b'\n'
Encoding the string to bytes allows the library to run.
Didn't want to submit a pull request in case you or someone else had a better solution.
tl;dr the extra
parameter in logger.info(mesg, extra)
is dropped from the sent json request. Why?
Given the following code
#!/usr/bin/env python3.4
import os
import logging
import logstash
# logging init
logger = logging.getLogger(os.path.basename(__file__))
logger.setLevel(logging.INFO)
# logging logstash
logger.addHandler(logstash.TCPLogstashHandler('localhost', 5000, version=1))
mesg = 'hello'
extra = {'test1': 1}
logger.info(mesg, extra)
Capturing the data sent over the wire using tcpdump
or wireshark
proves 'test1': 1
is not present.
{
"level": "INFO",
"stack_info": null,
"message": "hello",
"type": "logstash",
"path": "/tmp/logstash-examples/exampleIssue.py",
"tags": [],
"host": "myhost",
"logger_name": "exampleIssue.py",
"@timestamp": "2017-02-18T03:26:13.432Z",
"@version": "1"
}
Stepping through the code shows the extra
parameter is skipped for attaching to the record
(a dict
) within LogstashFormatterBase.get_extra_fields
. extra
becomes referenced by object attribute args
within (.../Lib/logging/__init__.py
)
def __init__(self, name, level, pathname, lineno,
msg, args, exc_info, func=None, sinfo=None, **kwargs):
# ...
self.args = args
And then args
skipped for adding within LogstashFormatterBase.get_extra_fields
because 'args'
is in skip_list
class LogstashFormatterBase(logging.Formatter):
# ...
def get_extra_fields(self, record):
# The list contains all the attributes listed in
# http://docs.python.org/library/logging.html#logrecord-attributes
skip_list = (
'args', 'asctime', 'created', 'exc_info', 'exc_text', 'filename',
'funcName', 'id', 'levelname', 'levelno', 'lineno', 'module',
'msecs', 'msecs', 'message', 'msg', 'name', 'pathname', 'process',
'processName', 'relativeCreated', 'thread', 'threadName', 'extra',
'auth_token', 'password')
# ...
fields = {}
for key, value in record.__dict__.items():
if key not in skip_list:
if isinstance(value, easy_types):
fields[key] = value
else:
fields[key] = repr(value)
From the call stack
get_extra_fields (.../Lib/site-packages/python_logstash-0.4.6-py3.4.egg!/logstash/formatter.py)
format (.../Lib/site-packages/python_logstash-0.4.6-py3.4.egg!/logstash/formatter.py)
makePickle (.../Lib/site-packages/python_logstash-0.4.6-py3.4.egg!/logstash/handler_tcp.py)
emit (.../Lib/logging/handlers.py)
handle (.../Lib/logging/__init__.py)
callHandlers (.../Lib/logging/__init__.py)
handle (.../Lib/logging/__init__.py)
_log (.../Lib/logging/__init__.py)
info (.../Lib/logging/__init__.py)
As a module user, the current behavior is unexpected, confusing, and prevents me from passing along required extra
information. I reviewed logging.html#logrecord-attributes. It was not clear why args
attribute (extra
parameter) should be effectively dropped. I suggest changing the current behavior to not drop extra
parameter information.
If the current behavior must be kept and this problem is to be worked around then moving skip_list
to be class instance variable would allow the user to remove 'args'
from skip_list
as needed.
Using python-logstash 0.4.7 (a8be251ad9fc102af668c38a5397515b0804da75
) on cython 3.4.1.
Because, I am running into I/O Error : too many files opened.
And when I do, lsof| grep pid, I see a lot of UDP connections opened and eventually I exhaust on the ulimit number. As soon as the ulimit number is exhausted, and more UDP sockets need to be opened for logging to log stash, I/O Error : too many files opened is encountered.
Raising ulimit doesn't completely fix this. Please help.
I'm anxiously waiting for a new version of python-logstash to be released with #41 included. I see that the version in setup.py
has already been bumped. Are you going to push it to PyPI soon?
https://www.python.org/dev/peps/pep-0632/#migration-advice
Using setuptools.setup might help here.
Line 1 in 7bd98c0
Hi there,
How can I set the name or an extra field fixed for all log messages? I'm trying to use a generic input/output having a variable for index, so I need to set the name or have a field on my python app.
output {
elasticsearch {
index => "%{name}"
hosts => "elasticsearch:9200"
}
}
Hi
I want to know if python-logstash has SSL support, because i cannot find any way to define custom certificate.
Thanks
CentOS 7
$ pip version
pip 9.0.1 from /usr/lib/python2.7/site-packages (python 2.7)
$ pip install python-logstash
Collecting python-logstash
Using cached python-logstash-0.4.6.tar.gz
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/usr/lib/python2.7/site-packages/pip/commands/install.py", line 324, in run
requirement_set.prepare_files(finder)
File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 634, in _prepare_file
abstract_dist.prep_for_dist()
File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 129, in prep_for_dist
self.req_to_install.run_egg_info()
File "/usr/lib/python2.7/site-packages/pip/req/req_install.py", line 412, in run_egg_info
self.setup_py, self.name,
File "/usr/lib/python2.7/site-packages/pip/req/req_install.py", line 387, in setup_py
import setuptools # noqa
File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 12, in <module>
import setuptools.version
File "/usr/lib/python2.7/site-packages/setuptools/version.py", line 1, in <module>
import pkg_resources
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 72, in <module>
import packaging.requirements
File "/usr/lib/python2.7/site-packages/packaging/requirements.py", line 59, in <module>
MARKER_EXPR = originalTextFor(MARKER_EXPR())("marker")
TypeError: __call__() takes exactly 2 arguments (1 given)
I'm not exactly sure what to do with a pip install error. I'm a gem/cpan guy.
An example Logstash config would be helpful to get started easier.
I will post my own as soon as I have a working one - but maybe the author also could post his/add it to the README? :-)
my Logstash is up and running in my system but when I wanted to test you sample code faced with
AttributeError: module 'logstash' has no attribute 'LogstashHandler'
while I've installed logstash package in my system.
python version : 3.6
python-logstash/logstash/formatter.py
Line 45 in a8be251
The value of extra fields are formatted via repr()
. In most cases it seems to me that str()
would be better. A typical example is Path
objects:
from pathlib import Path
path = Path("foo/bar")
print(repr(path))
print(str(path))
The repr-version is PosixPath('foo/bar')
, whereas the str-version is foo/bar
. In logs, I always want to have the latter version and, currently, have to manually enforce this via extra={"path": str(path)}
. This is also true for many other objects.
According to Python docs repr()
should return "the “official” string representation of an object" whereas str()
should return a "nicely printable string representation of an object". For logs, I think, the latter is almost always what one wants.
Therefore my question: would you be open to changing the formatting of extra fields from repr()
to str()
? Am I missing some reason why it is a bad idea? If you agree, I would submit a PR.
I imagine this would be as simple as creating a LogstashHandler equivalent that subclasses directly from logging.SocketHandler.
I thank message'field is field,not json string
from kibana
@timestamp 2015-03-30T05:09:42.151Z
@Version 1
_id AUxpFdtj8ZYGI1FQ4LPB
_index logstash-2015.03
_type logs
host 192.168.1.50
message {"nima": "ruifengyun", "message": {"@source": "logstash://bj-log-1/run.py", "source": "ruifengyun", "@source_host": "bj-log-1", "@message": "python-logstash: test extra fields", "@tags": [], "@fields": {"test_integer": 123, "nima": "ruifengyun", "test_dict": {"a": 1, "b": "c"}, "test_boolean": true, "test_list": [1, 2, "3"], "test_float": 1.23, "logger": "python-logstash-logger", "levelname": "INFO", "test_string": "ruifengyun(2, 6, 6, 'final', 0)"}, "@timestamp": "2015-03-30T05:09:42.150821Z", "@source_path": "run.py", "@type": "logstash"}}
I have an application which needs to send messages with different tags to logstash. The tags are used to classify messages and change over runtime. It seems wasteful to me to generate a new handler instance just to support tags.
For one execution I expect to be in the tens for tags (so not that many). But I can't say from the beginning which tags will be emitted so sending a message looks like this currently:
def send_message(tags, message):
handler = TCPLogstashHandler([...], tags=tags)
LOG.addHandler(handler)
LOG.info(message)
LOG.removeHandler(handler)
Having the tags exposed in the via the "extra" attribute would allow me to write this:
def send_message(tags, message):
LOG.info(message, extra={'tags': tags}
which would even make the send_message
function redundant.
I have a StackOverflow question, but it hasn't gotten any attention, so I thought I'd bring it up here.
Create a logger that can log the hostname
as a part of the formatter. The logger will have many handlers, one of which is a logstash handler.
import logging
import logstash
from socket import gethostname
class CustomLogger(logging.Logger):
def _log(self, level, msg, args, exc_info=None, extra=None):
if extra is None:
extra = { 'hostname' : gethostname() }
super(CustomLogger, self)._log(level, msg, args, exc_info, extra)
def setup_custom_logger(host, port):
# add hostname to the formatter.
logging.setLoggerClass(CustomLogger)
formatter = logging.Formatter(fmt='%(hostname)s - %(asctime)s - %(levelname)s - %(module)s - %(message)s')
logstash_handler = logstash.LogstashHandler(host, port, version=2)
logstash_handler.setLevel(logging.DEBUG)
logstash_handler.setFormatter(formatter)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.addHandler(logstash_handler)
return logger
Logging anything results in:
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/logging/handlers.py", line 620, in emit
self.send(s)
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/logging/handlers.py", line 678, in send
self.sock.sendto(s, self.address)
TypeError: a bytes-like object is required, not 'str'
I've tried logger.info('hello')
, logger.info(b'hello')
, logger.info(1)
-- they always result in a TypeError.
Does anyone have an idea why this isn't working? I'm using 0.4.6.
makePickle function is missing (+ b'\n') at the end.
Check out https://github.com/logstash/logstash-logback-encoder#standard-fields or https://github.com/logstash/log4j-jsonevent-layout#usage. Both are supported by logstash crew.
Mismatches are minor but having several names for the same field is no fun as well.
logger_name
vs logger
level
vs levelname
I find my self doing this:
class CustomUDPLogstashHandler(logstash.UDPLogstashHandler):
def __init__(self, *a, **kw):
self.global_extras = kw.pop('global_extras', {})
super().__init__(*a, **kw)
def emit(self, record):
for key, value in self.global_extras.items():
setattr(record, key, value)
super().emit(record)
LOGGING={
...
'logstash': {
'level': 'DEBUG',
'class': 'util.CustomUDPLogstashHandler',
'host': 'logstash.service',
'port': 5000,
'version': 1,
'tags': ['tag1', 'tag2'],
'global_extras': {
'product': "my_test_product",
'instance': "my_test_instance",
'site': 333
}
}
...
It would be cool if that functionality were included
When I want to format logs I face error[Appendix A] telling that concatenate of str and bytes is impossible, but it works when I remove parameter: formatter=text
in my log.ini file[Appendix B]
"Waiting for application startup."
--- Logging error ---
Traceback (most recent call last):
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/logging/handlers.py", line 673, in emit
s = self.makePickle(record)
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/site-packages/logstash/handler_tcp.py", line 25, in makePickle
return self.formatter.format(record) + b'\n'
TypeError: can only concatenate str (not "bytes") to str
Call stack:
File "<string>", line 1, in <module>
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
return self._bootstrap(parent_sentinel)
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/site-packages/uvicorn/subprocess.py", line 76, in subprocess_started
target(sockets=sockets)
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/site-packages/uvicorn/server.py", line 60, in run
return asyncio.run(self.serve(sockets=sockets))
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/asyncio/base_events.py", line 633, in run_until_complete
self.run_forever()
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/asyncio/base_events.py", line 600, in run_forever
self._run_once()
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/asyncio/base_events.py", line 1896, in _run_once
handle._run()
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/site-packages/uvicorn/server.py", line 77, in serve
await self.startup(sockets=sockets)
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/site-packages/uvicorn/server.py", line 88, in startup
await self.lifespan.startup()
File "/home/atmosphere/miniconda3/envs/research_elastic_search/lib/python3.10/site-packages/uvicorn/lifespan/on.py", line 45, in startup
self.logger.info("Waiting for application startup.")
log.ini
config file[handler_logstash]
class=logstash.TCPLogstashHandler
level=INFO
formatter=text
args=('127.0.0.1', '8080', '1')
[formatter_text]
format="%(message)s"
datefmt='%d-%m-%Y %H:%M:%S'
python-logstash has to be modified to work with the new JSON format in logstash 1.2.x. The change simply involves removing the "@" signs from all the dictionary keys.
When using the TCPLogstashHandler
and sending a message, I get the following warning:
ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.27.0.74', 36854), raddr=('172.27.0.63', 5001)>
I'm not certain if this comes from the underlying SocketHandler
from Python or from python-logstash
. If it's the former, feel free to close this issue.
Hey there, we had some old config sitting around pointing to a localhost logstash – which did not exist.
It turns out that this was causing 1000ms latency when calling logger.info
for the first time.
I don't understand the root cause, but I just thought I would note here that this seems like a silent failure mode which could be fixed.
Related django config
"logstash": {
"level": "INFO",
"class": "logstash.TCPLogstashHandler",
"host": os.getenv("LOGS_IP", "localhost"),
"port": 5000, # Default value: 5000
"version": 1,
"message_type": "django_logstash", # 'type' field in logstash message.
"fqdn": False, # Fully qualified domain name. Default value: false.
# 'tags': ['django.request'], # list of tags. Default: None.
},
As per gunicorn documentation it is accepting the standard Python logging module’s Configuration file format.
but how can I give logstash handler to gunicorn in configuration file , what is the syntax for that?
I gave like below:
[loggers]
keys=root, logstash
[handlers]
keys=console , logstash
[formatters]
keys=generic, access
[logger_root]
level=INFO
handlers=console
[logger_logstash]
level=DEBUG
handlers=logstash
propagate=1
qualname=logstash
[handler_console]
class=StreamHandler
formatter=generic
args=(sys.stdout, )
[handler_logstash]
class=logstash.TCPLogstashHandler
formatter=generic
args=('localhost',5959)
[formatter_generic]
format=%(asctime)s [%(process)d] [%(levelname)s] %(message)s
datefmt=%Y-%m-%d %H:%M:%S
class=logging.Formatter
[formatter_access]
format=%(message)s
class=logging.Formatter
not working
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.